DEV Community: Seth Wheeler

What a Language Needs Before It Can Compile Itself

Seth Wheeler — Sat, 05 Sep 2026 03:00:00 +0000

Code: Megapixel99/lambda-language

lm is a small low-level language I wrote: static types, explicit memory, no closures, no garbage collector, and four independent backends that emit C, WebAssembly, ARM64 and bytecode for a VM. Its compiler is about 4,400 lines of JavaScript. The obvious next question is whether the language can compile itself, and the obvious first step is the lexer, which is 129 lines.

In lm the same lexer is 355 lines. That ratio is the finding, because almost none of it is lm being a verbose language. Six specific absences account for nearly all of it, and writing them down was a planned milestone rather than an afterthought: the point of porting the lexer first was to find out what the language could not do while the port was still small enough to abandon.

The one that cost the most

src/lexer.js has a single advance(n) that moves pos, line and col together, called from 14 places. lm had no way to take the address of a scalar local, so a function could not mutate a caller's variable, and a function returning three values would need a struct allocated on every call. So advance does not exist. All 14 sites write pos += 1; col += 1; inline, and the newline case writes the three-line variant.

That is the single largest source of the size difference, and it also caused the only correctness bug in the port. Column counting inside a string literal has to skip UTF-8 continuation bytes, and because the logic is inlined rather than centralised there is no one place to fix it. The two comment scanners over-count a column in exactly the same way. They get away with it only because a comment always ends at a newline, which resets the column before anything reads it.

That is worth sitting with. A centralised advance would have been fixed once and been right in all three places. Instead the code is right in one place by correction and in two others by luck, and the luck is load-bearing: change what terminates a comment and two latent bugs become live ones. Duplication did not make the bug; duplication made the fix local when the bug was not.

The other five

No globals or constants. The keyword and operator tables get built inside main at run time, every run. The waste is small; the real cost is that anything needing them takes them as parameters, and the same goes for the one scratch buffer the output helpers share, which has to be threaded through six functions by hand. Without that threading each helper would call alloc itself, and with no free that leaks one cell per call: emitting the token stream for this file is around ten thousand such calls, or 80 KB of unreclaimable heap out of 1 MiB.

No character literals and no hex. Every byte constant is a decimal number. c == 35 for #, c == 92 for a backslash, c == 34 for a quote, + 48 to turn a digit into ASCII, c & 192 == 128 for a UTF-8 continuation byte. The JavaScript writes c === "#". The lm version is correct and close to unreadable, and a comment naming each constant is the only defence.

No string type, so no string comparison. Matching an identifier against fifteen keywords is strlen plus a byte loop per keyword. The JavaScript is KEYWORDS.has(text).

No early exit from a search. break cannot carry a value and there are no labelled loops, so the keyword and operator scans run to completion and guard every iteration with a flag:

for i in 0..34 {
  let o = ops[0][i];
  if (hit as i64) == 0 && matchesAt(src, pos, o) { hit = o; }
}

That is 34 comparisons where OPERATORS.find stops at the first hit. Correct, slower, and it reads worse than what it replaces.

No growable array. A token list cannot be built, because there is no realloc and every token would own a string, so tokens are streamed to stdout as they are recognised. For a lexer that is arguably the better design; for a parser it is not, and I wrote at the time that this was the item most likely to decide whether the next stage was feasible.

What did not come up

The list above is misleading without this part. Structs, fixed arrays, arrays of pointers, function pointers, for loops, compound assignment, the full integer type set and u64 arithmetic all did what was needed with no workaround at all. The array-of-strings table works exactly as you would write it:

let ops = new([*u8; 34]);
*ops = ["<<=", ">>=", "==", "!=", ...];

The language was not the obstacle. Six specific absences were, and that distinction is the whole value of doing this as a measurement rather than an impression. "The language felt awkward" would have produced a wishlist. Counting what each absence cost produced an ordered one.

The list was ordered, and two of them got built

The recommendation, in the order they pay off, was: address-of on a scalar local, then a growable allocation, then top-level const, then character and hex literals. The first two are now in the language, and the parser, checker and emitter stages that followed all used them.

& carries a cost the original note recorded and I want to repeat, because it is the sort of thing that is obvious only after you have it. An addressed local lives in a frame allocated from the flat region on entry, and there is still no free, so a function that takes the address of its own local and is called N times leaks N frames. That suits a lexer, where main holds the state and passes &pos down; it does not suit a recursive descent parser, where the functions taking addresses are the ones called thousands of times.

The classification was wrong once, and the next stage proved it

I filed "no globals" under merely annoying; in the parser it was a correctness problem.

With no globals, the operator precedence table is either threaded through every call or allocated where it is read, and the parser allocated it: a 54-entry allocation per precedence test, tens of thousands of them, against a heap with no free. One of the test programs simply never finished parsing. The tables live in the parser state now.

So the severity ranking in that document was right about which items cost the most lines and wrong about which one could break a program. Those turn out to be different questions, and the lexer could not distinguish them, because a lexer allocates once per token and a parser allocates inside its hot loop. A single-stage measurement produced a single-stage ranking, and I presented it as though it generalised.

What generalises

Self-hosting is usually described as a milestone, and it is more useful as an instrument. A language's missing features are invisible while you are writing programs that fit on a screen, because you route around each absence in a few lines and forget it. Porting a program you did not write, against a reference implementation you can diff, turns each of those routes into a measurable cost: 14 inlined sites, 34 comparisons instead of 1, 80 KB of unreclaimable heap, one bug that survives on the behaviour of comments.

The gate I set for this milestone was to stop if the list ran long, on the grounds that growing the language to fit is a different project from self-hosting it. Six items with two clear priorities is not long. The judgement that mattered was not "is lm good enough", which is unanswerable, but "which four absences are worth building, in what order", which the port answered on its own.

LIMITS.md is in the repository, with each entry beside the code it cost.

No Add-on Can Block Ads in This Browser

Seth Wheeler — Thu, 03 Sep 2026 15:45:05 +0000

Code: Megapixel99/kestrel

Kestrel is a macOS browser I have been building on WKWebView to test whether a browser can hold a memory budget the way a game engine holds a frame budget. It used to ship six built-in imitations of common extensions, including an ad blocker. When macOS 15.4 shipped WKWebExtension I replaced all six with the real thing: Kestrel unpacks an .xpi and hands it to WebKit, and all 25 add-ons from my Firefox profiles load.

Then I measured whether the real ad blockers actually block anything. They do not. Neither manifest version works, the failure is silent in both, and the mechanism I deleted is the only one I can demonstrate working.

The measurement

blocktest loads an add-on, opens five known tracker URLs, and reports which of them the network layer actually fetched.

cd kestrel && ./.build/debug/kestrel blocktest ubolite 45

what	how	result
MV2 blocking `webRequest`	test add-on returns `{cancel:true}`	request went through
MV3 `declarativeNetRequest`, uBO Lite, 6 enabled rulesets including EasyList	5 tracker probes	0 of 5 blocked
the same, after 180 s for rule compilation	same	0 of 5 blocked
the same, in a web view built from the controller's own configuration	same	loaded
`WKContentRuleList` compiled by Kestrel itself	same probe	blocked

The last row is the reason to believe the other four. A negative result from a probe is worthless unless you can show the probe detects the positive case; without that row "loaded" could just as easily mean the test never looked. The self-check runs every time, so "loaded" means the request really was not blocked.

What WebKit does here is worse than refusing. It grants webRequestBlocking and then ignores what the handler returns. It accepts declarativeNetRequest rulesets, and hasContentModificationRules reports true, without applying them to any web view I can reach through the public API. That includes one built from WKWebExtensionController.Configuration.webViewConfiguration, which is the configuration the extension controller itself hands you. An add-on has no way to detect any of this. It reports itself as working, its dashboard shows six enabled rulesets, and every request goes through.

I stated the opposite twice

Before running the test properly I wrote in this project's own notes that MV2 blocking webRequest was the problem and that an MV3 blocker would be the fix. I wrote it twice, with more confidence than the evidence supported, and it was wrong in a specific way worth naming: the manifest version is not the discriminator. Both fail; MV3 fails after accepting the rulesets and reporting them compiled, which is why it looked like the answer from the outside.

What made the claim survive is that I checked whether uBO Lite loaded, which it does, rather than whether it blocked, which it does not. Those are different questions and only one of them needed a probe.

What I have not ruled out

Two possibilities are still open, and neither is settled by anything above.

WebKit may honour only dynamic rules, added through declarativeNetRequest.updateDynamicRules, and not manifest-declared static rulesets. A test add-on that adds a single dynamic rule for a known URL would settle it in an afternoon, and it is the obvious next experiment. Note that this would not rescue uBO Lite, whose rulesets are static, but it would change the finding from "declarativeNetRequest does not work" to something narrower and more useful.

Alternatively, Safari may implement extension content blocking through its own content-blocker plumbing, which a host application embedding WebKit does not inherit. If that is the case, this is not fixable from outside the engine and should be recorded that way rather than left looking like a bug someone could fix.

I am stating both because the measurement supports "no add-on blocked anything through the public API" and does not support "WebKit cannot block." Those get conflated easily, and the second is a much larger claim than I have evidence for.

The thing that works is the thing I deleted

WKContentRuleList compiled by the browser blocks reliably. The self-check proves it on every run of blocktest, against the same probe the add-ons fail.

That is precisely the mechanism the deleted built-in ad blocker used: it converted Adblock Plus filter lists into a WKContentRuleList and handed it to the web view. I removed it, along with five other imitations, on the reasoning that real add-ons had made the imitations redundant. For five of the six that was true. For the ad blocker it was exactly backwards, and I did not find out until after the code was gone, because I checked that the replacements loaded rather than that they worked.

Restoring a native content blocker is the only demonstrated way to block ads in this browser. Which is an odd conclusion for a project whose whole point was to stop reimplementing things the platform provides.

A related failure, for contrast

Adblock Plus's options page is a separate problem with a clean answer, and it is worth putting beside the first one because it looks similar and is not.

The page came up blank. ABP's options page is a shell whose only content is an iframe of desktop-options.html, and WebKit refuses that subframe unless the add-on declared web_accessible_resources, which is stricter than Firefox or Chrome and which ABP does not declare. Kestrel now navigates the tab to the inner page directly when a lone iframe fails, so the content loads with nothing loosened: no manifest rewriting, and nothing of the add-on made readable by arbitrary web pages. The check has to be repeated rather than made once, because at didFinish the iframe carries only data-src; ABP's options.js is deferred and sets the real src a beat later.

The page now renders, and then refuses:

options page in a tab: …/desktop-options.html  nodes=8
  getBrowserInfo=undefined
  ua=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:141.0) Gecko/20100101 Firefox/141.0
  text=Your browser version is no longer supported. Please upgrade

It is not the user agent, which is a clean Firefox string with no AppleWebKit prefix; it is browser.runtime.getBrowserInfo, a Firefox-specific API that WebKit's runtime does not implement, so ABP has no version to compare against and defaults to unsupported. That one is shimmable by injecting a script into add-on pages, but shimming it means telling an add-on it is Firefox 141 and inviting every other Gecko-only path it might then take, which fail less visibly than this one does. uBO Lite is unaffected, since its dashboard is a page rather than a frame shell.

The contrast is the useful part. The ABP failure is a missing API that announces itself: undefined is checkable, the add-on notices, and it tells the user. The blocking failure announces nothing at all. An API that is present, accepts your input, reports success and does nothing is harder to find than one that is absent, and it is the reason this took a purpose-built probe with a control rather than an afternoon of clicking around.

What generalises

When you test whether a subsystem works, the cheap version of the test asks the subsystem. The expensive version asks something downstream that has no reason to cooperate. Here the cheap version was uBO Lite's dashboard reporting six enabled rulesets, and it was wrong for three minutes and then still wrong. The expensive version was five URLs and a check of what the network actually fetched.

The control row is what turns that from an anecdote into a result, and it costs one extra case: block the same URL by a mechanism you know works, and confirm the probe notices. Without it a negative finding is indistinguishable from a broken test, which is the position I was in when I twice wrote down the wrong cause.

The full measurement, including what has been ruled out and what has not, is in BROKEN.md. Kestrel is 10,746 lines of Swift across 45 files; measurements are on an M1 Max running macOS 15.5, Swift 6.1.2 and system WebKit.

Measuring the Wrong Process for Eight Months

Seth Wheeler — Thu, 03 Sep 2026 03:00:00 +0000

Code: Megapixel99/kestrel

In What a Hibernated Browser Tab Actually Costs I reported that Kestrel's memory ladder held 121.7 MB against an unmanaged browser's 319.9 MB on a set of real websites, and that it never went over its budget. Both numbers were wrong. Re-measured, the same run holds 345.9 MB against an unmanaged 255.3 MB, and it is over budget in 85% of samples. The ladder did not do less good than I published; on that workload it did harm, and the published figures reported it as the best result in the post.

The correction is worth more than the original finding, because what broke was not the design. It was the instrument.

The number that gave it away

Kestrel reported a Jira board as costing 52 MB. The process rendering that page held 511 MB. The 52 MB was real, and it belonged to a completely different application's WebContent process that had been running for eight days.

I did not catch this with a test. I caught it with a screen recording, watching the same figure sit unchanged before and after loading an entirely different page. That is a temporal observation, and no single assertion or screenshot contains one. You have to see two moments next to each other and notice they agree when they should not.

Three mistakes, stacked

Every memory figure in the earlier post was a per-tab attributed sum: one process id per tab, claimed by diffing ps output around tab creation, then added up. Three things were wrong with that, and each one hid the next.

ps lists every WebContent process on the machine. They are XPC services parented to launchd with identical command lines, so there is no parent and no client tag to filter on. Nothing filtered at all, which made any content process on the system a candidate, including ones belonging to Safari or to an Electron app someone left open. The single sound discriminator available is age: a process older than the browser cannot belong to it.

Tab creation picked .first on an unordered set. WebKit spawns several content processes at once (five, on a measured launch), and the diff-at-creation trick assumed it would see exactly one new pid. Whichever one it happened to grab, the tab reported for the rest of its life.

Nothing distinguished "cannot measure" from "costs nothing." Both rendered as 0 MB. A live page the browser had lost track of was therefore counted as free by the scheduler: never in the total, never a candidate for demotion.

Why it survived eight months

The number was always plausible. 52 MB for a page is unremarkable; so is 115, and so is 399. A wrong number inside a believable range is invisible in a way that a crash is not, and every test written against it passed, because the tests asked the browser what it thought rather than checking the browser against the machine.

That is the same failure this project had already recorded once. An earlier round of work improved what about:memory reported while returning nothing to the OS, and I wrote it down at the time. I then built a benchmark with the identical shape and did not recognise it for eight months. Writing a lesson down is not the same as having learned it.

The corrected numbers

The benchmark now records both the attributed sum and the browser's real footprint, which is every WebContent process younger than the run, summed. That second figure needs no attribution at all, so it cannot fail in this way. Heavy pages, 800 MB budget, 40 events:

policy	attributed	measured	tabs accounted for	over budget: attributed to measured
none	975.9 MB	974.9 MB	100%	82% to 82%
discard-LRU	547.1 MB	802.2 MB	68%	2% to 62%
Kestrel	587.9 MB	788.1 MB	75%	2% to 48%

The 100% row is the one that matters. With every tab live and every process owned by a tab, the two methods agree to within 1 MB, which means attribution became exact once the web view named its own process. So the 25 to 32% gap under the demoting policies is not measurement error. It is real memory sitting in processes that outlived the tabs they belonged to.

That splits a question the project could not previously answer. Of the gap, roughly 0% is mis-attribution and all of it is processes orphaned by demotion. Demoting a tab releases its web view; WebKit keeps the process; nobody owns that memory, and nobody was counting it.

Per event, the shape is unmistakable once the two columns sit side by side:

policy=none     event 12:  attributed 1049   measured 1089   procs 10
policy=kestrel  event 12:  attributed  428   measured  668   procs 10
policy=kestrel  event 36:  attributed  749   measured  989   procs 13

The error is not uniform, and it flatters exactly the policies the benchmark exists to evaluate. With no policy running, every process belongs to a live tab and attribution captures effectively all of it. Start demoting and the processes stop being attributed to anything, so the reported number falls while the memory does not. A measurement whose error scales with the treatment is not a noisy measurement; it is a measurement that manufactures the result.

Corrected against the unmanaged run, discard-LRU comes to 975 / 802 = 1.22x and Kestrel to 975 / 788 = 1.24x. Across three runs Kestrel measured 1.28x, 1.28x and 1.24x, and discard-LRU 1.21x, 1.20x and 1.22x. The "0% over budget" claim does not survive at all: measured, Kestrel is over budget in 48% of samples and discard-LRU in 62%. The scheduler was demoting until its own accounting said it was under budget, which turns out not to be the same thing as being under budget.

The light pages inverted

The published post's worst-looking table was the light-pages run, and I described it there as a loss. It was considerably worse than a loss. Eleven mixed-weight real sites, 150 MB budget, 40 events:

policy	published (attributed)	measured	vs unmanaged	over budget	tabs destroyed
none	319.9 MB	255.3 MB	n/a	85%	0
discard-LRU	104.6 MB	414.8 MB	1.62x worse	90%	19
Kestrel	121.7 MB	345.9 MB	1.35x worse	85%	15

Attribution accounted for 100% of the unmanaged run and only 25% and 36% of the managed ones. The policies were reporting roughly a quarter of what they actually held, and the error was largest precisely where the design looked best.

This is not a new failure mode. It is this design's own predicted one, finally measured. DESIGN.md states the feasibility rule as budget > live working set + (39 MB x parked tabs). Eleven tabs at the 39 MB COLD floor need 429 MB before a single page is displayed, and this run was handed 150 MB, which is 2.9x below its own floor. The scheduler says so in its logs: gave_up fires 5 times for discard-LRU and 7 times for Kestrel.

What the design predicted below the floor was degradation toward discard-LRU. What actually happens is degradation below doing nothing. A demotion leaves the old WebContent process alive, the restore spawns another, and 45 demotions with 18 restores over 40 events churn processes faster than WebKit reclaims them. Measured peak reached 530 MB for Kestrel and 650 MB for discard-LRU, against 309 MB for the run that managed nothing at all, on the workload whose entire premise was that the pages were light.

What survives

The ladder's own physics were measured correctly the whole time. Re-measured with the web view naming its own process, six tabs, same synthetic page:

LIVE    128.0 MB
WARM    106.0 MB   (83% of live)
COLD     39.0 MB   (30% of live)
restore COLD -> LIVE: 87 ms

Identical to the published figures, to the decimal. Two warnings the re-measurement was watching for did not fire: the pid diff was never ambiguous in that run (one tab at a time, 3.5 s apart, exactly one WebContent process each), and navigating to about:blank did not move the page to a different process, which was the way the 39 MB COLD figure could have been some abandoned process's footprint rather than the parked tab's. The feasibility floor built on 39 MB stands unchanged, which is why it was able to predict the inversion above.

The comparative case also came out stronger than the old numbers suggested. Kestrel holds less real memory than discard-LRU (788 MB against 802 MB) while destroying fewer tabs (6 against 8) and spending far less time over budget (48% against 62%). On the attributed figures it looked marginally worse on memory and better only on state loss.

So the honest summary is workload-dependent, and one side of it is negative:

workload	budget vs. floor	Kestrel vs unmanaged
heavy pages, 800 MB budget	above the floor	1.24x better
light pages, 150 MB budget	2.9x below the floor	1.35x worse

The ladder helps when the budget is reachable and hurts when it is not, because below the floor the process churn costs more than the parked pages save. The old numbers hid that completely. They reported 2.63x better on precisely the run where the design was 1.35x worse.

What I changed, and what I did not

WKWebView._webProcessIdentifier is private API, verified on macOS 15.5, and it is now asked at every point where a tab's process is established, with age-filtered ps diffing as a fallback. The whole-browser total is measured independently by summing the run's own processes, which needs no attribution and therefore cannot fail this way. Where the two disagree by more than 20%, the interface says so rather than picking one.

The scheduler itself is unfixed, and a tweak is the wrong response. A browser that cannot meet its budget should refuse the budget: surface the floor, name the number of tabs it can hold, and stop demoting, rather than thrash against a target it can prove is unreachable. That is recorded in BROKEN.md and not implemented.

The part that generalises

Both times this project measured itself wrong, the instrument reported on the thing it was part of, and both times it reported success. Tier-down improved about:memory while returning nothing to the operating system. Per-tab attribution improved the benchmark's total while the memory stayed in processes nobody counted. The fix was the same on both occasions: measure from outside the thing being measured, where the number cannot be produced by the component whose behaviour is in question.

The tell is worth naming, because it is available before you know anything is wrong. Ask what your measurement would report if the feature did nothing at all, and then ask what it would report if the feature worked perfectly. If the second answer can be produced by the component simply losing track of its own work, the instrument is not measuring the feature. Mine could, and for eight months it did.

The full account of the bug is in DEBUGGING.md, and the corrected tables with their per-event traces are at the end of RESULTS-ENGINE.md. Kestrel is 10,746 lines of Swift across 45 files on top of WKWebView; the measurements here are all on one machine, an M1 Max running macOS 15.5, Swift 6.1.2 and system WebKit.

When a SQL Engine Records Column Types but Never Reads Them

Seth Wheeler — Wed, 02 Sep 2026 03:00:00 +0000

Code: Megapixel99/sql-nodejs

sql-nodejs is an in-memory SQL database I wrote to understand how a parser turns a statement into stored rows. It takes SQL strings, creates tables, stores rows, and answers SELECT. Its CREATE TABLE accepts column types, and it records them.

It never reads them again.

Here is the whole thing, against the published package:

const SqlParser = require('sql-nodejs');       // 0.0.6
const db = new SqlParser();

db.Parse('CREATE DATABASE mydb;');
db.Parse('CREATE TABLE users (id INT, name VARCHAR, age INT);');
db.Parse('INSERT INTO users (id, name, age) VALUES (1, alice, 30);');
db.Parse('INSERT INTO users (id, name, age) VALUES (2, bob, 25);');

db.Parse('SELECT * FROM users;');
// [ [ '1', 'alice', '30' ], [ '2', 'bob', '25' ] ]

id was declared INT and comes back as '1'. Every value there is a string, including the two columns whose declared type says otherwise. The type survives parsing and reaches storage; nothing downstream consults it.

The query that finds nothing

Storing numbers as strings is a limitation the README already admits. On its own it is not very interesting. What makes it worth writing down is that WHERE appears to work anyway:

db.Parse('SELECT * FROM users WHERE age=25;');
// [ [ '2', 'bob', '25' ] ]

That is the right row, and it is exactly why the problem is easy to miss, because the obvious test passes and nothing suggests looking further. Now ask for the same row a different way:

db.Parse('SELECT * FROM users WHERE age=25.0;');
// []

25.0 and 25 are the same integer, bob is still 25 years old, and the result is empty.

Why the first query works

The comparison is a strict equality between two strings, in table.js:

_where[i].split("=")[1] === this.rows[j].data[
  this.getColmunNames().indexOf(_where[i].split("=")[0])
]

The left side is a fragment of the query text: everything after the = in age=25, which is the string '25'. The right side is what INSERT put in the row, which is also the string '25', because that is how the value arrived and nothing converted it. So '25' === '25' is true and the row matches.

Nothing in that expression knows the column is an INT. It matches because the two pieces of text are spelled identically, and 25.0 is spelled differently. The equality operator is doing string comparison and getting the right answer for integers by coincidence. That coincidence held for every example I wrote while building it, because I wrote the same integer on both sides every time.

The part that has not bitten yet

WHERE currently supports one column=value equality and nothing else, and that limit is what has been hiding the rest of this, because equality is the one operator where comparing strings usually agrees with comparing numbers. Ordering does not have that property:

'9' > '10'   // true

Adding > by reusing the existing comparison would produce a database in which nine is greater than ten, and it would pass any test whose numbers happen to have the same digit count. The bug would not be in the new operator, but in the much earlier decision to keep the parsed type and never look at it, and the new operator would only be the first thing to ask.

What I would fix, and what that costs

The fix is coercion at INSERT, using the type CREATE TABLE already captured: store 30 rather than '30', and parse the right-hand side of a WHERE before comparing it rather than slicing it out of the query string. That is where the recorded type starts doing work.

It is also a behaviour change for anyone reading results today, since SELECT would begin returning numbers where it now returns strings. That belongs in a version bump rather than a patch.

What generalises

Parsing a type and honouring a type are separate pieces of work, and it is possible to ship the first while believing you shipped both. CREATE TABLE users (id INT, ...) is accepted, stored, and displayed back, so every surface agrees the type is real. The only way to find out it is decorative is to ask a question where the string and the number disagree. age=25 is not that question; age=25.0 is, and it costs one line to ask.

Choosing Video Frames by Content Instead of by a Clock

Seth Wheeler — Tue, 01 Sep 2026 08:21:06 +0000

Code: Megapixel99/video-timeline

The usual way to hand a video to a language model is to pull a handful of screenshots and let it reason over them. The trouble isn't in reading any one frame; it's in reading the gaps between them. A screenshot doesn't say when it happened, how long after the last one, or what moved in between. So the model fills that in with a plausible story. VTL replaces those screenshots with a measured timeline. The first question worth asking is whether it actually sees more of a video than evenly spaced sampling does at the same cost.

It does, and the gap is large. Both methods get the same frame budget, run over five real videos: eighteen minutes of a narrated slide deck, two screen recordings, a scroll capture, and handheld phone footage. Then you count what each one misses. Across 28 shots, uniform sampling never looks at 9 of them. That is a third of the video, gone. It also spends 12 of its 40 frames re-photographing pictures that differ from the previous frame taken by under 7%. VTL misses no shot and wastes 2 frames. You can check this yourself. python3 tests/benchmark.py runs the comparison on generated fixtures, or on your own files if you pass them.

The mechanism is simple enough to disagree with. Uniform sampling puts a frame every T seconds, regardless of what is on screen. VTL puts frames where the picture changes, drops the near-duplicates, and enforces a floor. No shot goes without at least one frame, and no stretch longer than the coverage limit goes unobserved. The budget uniform sampling spends re-photographing a motionless slide, VTL spends on the shots uniform never reached.

Where it does not win

That is not the same as "smaller gaps everywhere," and the benchmark is built to catch me if I claim it is. On the size of the largest unobserved gap, the two methods come out level. On one video, the dashboard capture, uniform sampling is the better of the two: a worst gap of 0.07 against VTL's 0.14. That result is real, and it follows from the design. VTL concentrates its frames at the moments of change, and it deliberately declines to re-photograph a slide that is sitting still. Across the still part, that can leave a wider gap than even spacing would. The difference I will defend is not that the gaps are smaller. It is that VTL names its gaps in the timeline as frozen spans, where uniform sampling's gaps are simply unexplained.

One honest caveat on the coverage number. "Shots never seen" is counted against VTL's own shot boundaries, which are the thing under test, so it is not an independent oracle. It is fair for measuring coverage, because a shot boundary is a measured discontinuity whichever tool you ask. I would not push it further than that.

Why I trust the numbers

Most of these figures are checked against ground truth I can compute rather than eyeball. The camera-motion rates are asserted against a fixture whose every frame is rendered in Python, by moving a viewport along a closed-form path. The true pan, tilt and zoom rates are therefore known exactly. Pan and tilt land within about 10%. Zoom reads roughly 30% low: true +0.115, measured +0.080. That is not noise. Block displacement under a zoom is only a fraction of a proxy pixel near the frame centre, so the direction is reliable and the magnitude is a lower bound. Both facts are printed in the bundle, next to the measurement that produced them. On real footage there is no ground truth to assert against, so a second script measures the same clips with an independent algorithm: FFT phase correlation at 480 px, sharing no code with the converter's block matching on a 96 px proxy. On a pan across a grass field the two agree, at -0.088 and -0.090.

Every one of those checks began by finding the code wrong. A fade from black fabricated a confident zoom_in on a static title card, because the matching is brightness-sensitive and read the brightening as the frame growing toward the camera. Aliasing on the downscaled proxy turned a true +0.94 px/frame drift into a confident -3.94, wrong in both sign and magnitude, until one blur pass before matching fixed it. OCR read the tool's own timestamps, burned into each frame header, back out and reported them as text found in the video. None of these looked broken; they looked like measurements, which is exactly why the fixture has to carry an answer you already know.

On how it was built

The README is straight about this, and the post should be too. VTL was written with heavy AI assistance (Claude Code). The typing was assisted. The design judgement, and the "is this actually true?" loop that caught every bug above, were the work. Two of those bugs were found only because a fixture I had written was itself wrong. That is why the motion fixture renders each frame from a closed-form path instead of leaning on video filters. A fixture you have to debug is not a fixture.

The general point is the one the screenshots get wrong. When you sample a video on a fixed clock, the frames you get are an accident of the clock's phase against the content, and the gaps between them go unexamined. VTL measures where the content actually changes, then states the worst gap it left rather than hoping there wasn't one. That costs about 67 seconds for eighteen minutes of video. It turns those intervals from a guess into a number. The errors were never in the frames; they were always in the spaces between them.

Choosing an Unreleased API Over the One Already There

Seth Wheeler — Tue, 01 Sep 2026 03:00:00 +0000

express-openapi is Wesley Todd's library, not mine. It generates an OpenAPI document from an Express app by reading the routes the app actually registered, which beats maintaining a spec by hand and hoping it still matches the code.

I opened a pull request to add Express 5 support, and I had hoped it would be small. It came to +244/-259 across eight files. That was not the mistake. The mistake came later and took two lines.

How the routes are found

Express does not offer a list of registered routes. It has a router, the router has a stack of layers, and each layer holds a handler plus a compiled regular expression for the path it matches. The library walks that stack directly.

Walking is the easy half; the hard half is that a layer knows its path as a regular expression and an OpenAPI document needs it as a string. There is no inverse for "compile this path", so the library recovers the path by stringifying the regular expression and deleting the parts the router put in:

const match = thing
  .toString()
  .replace('\\/?', '')
  .replace('(?=\\/|$)', '$')
  // Added this line to catch the express v5 case after the v4 part is stripped off
  .replace('(?:\\/(?=$))?$', '$')
  .match(/^\/\^((?:\\[.*+?^${}()|[\]\\/]|[^.*+?^${}()|[\]\\/])*)\$\//)

That comment is not mine and it predates my patch, which tells you this had already caught someone. Express 5 compiles paths differently again, so the tests that pinned particular regular expressions to particular strings had nothing left to say. That is why test/_regexRoutes.js lost 117 lines instead of gaining a case, and why lib/generate-doc.js came to +87/-122.

The line that needed an API that did not exist

Rewriting the walk left one problem. When a sub-router is mounted, the document needs the prefix it was mounted at, and my rewrite asked the router for it:

const p = router.getRoutes()[0].path.split('/')[1]

getRoutes() is not part of the router. It is pillarjs/router#174, an open pull request by bjohansebas that adds a method for listing registered routes, which is exactly what a consumer needs so that nobody has to un-compile a regular expression again. It is the right feature and I wanted it.

So my package.json pointed at a branch:

"router": "github:bjohansebas/router#maproutes"

and I said in the thread, on the day I opened it, that this should not merge until getRoutes() was released. That was August 2025.

Four days later I did not need it

On 2 September I replaced that call with a property the layer was already carrying:

-      const p = router.getRoutes()[0].path.split('/')[1]
+      const p = routeLayer.pathPatterns

-      "router": "github:bjohansebas/router#maproutes",
+      "router": "^1.3.8",

Two lines, and the dependency on an unreleased feature was gone. The commit is titled "remove dependency on the yet to be released feature getRoutes()". It passed CI on Node 20, 22 and 24.

Twenty minutes after the last of those pushes, I force-pushed the branch back to the version that calls getRoutes(), and that is what the pull request contains today.

Why I put it back

Two reasons, and I still think the first one is right.

routeLayer.pathPatterns is another internal. It is an undocumented property on a layer object, subject to exactly the same drift as the regular expression I had just finished being burned by. getRoutes() is a public method whose entire purpose is to answer this question. Given a choice between reading one more private field and calling an API designed for the job, the API is the better code, and it is still the better code now.

The second reason was an estimate, and the estimate was wrong. I assumed getRoutes() was weeks away. It was opened in July 2025, it is +350/-3, it carries twenty-nine reviews, and it is still open. None of that is unusual or anyone's fault: adding a method to the router that sits under Express is exactly the kind of change that should collect twenty-nine reviews, and the people doing that review are doing it for free, around whatever else their lives contain. A careful API takes as long as it takes.

What I got wrong was treating someone else's review cycle as a schedule I could plan against. I did not decide to wait a year; I decided to wait a few weeks, and then never revisited it, because a pull request that is blocked does not remind you that it is blocked.

What generalises

Choosing the cleaner interface over the available one is not really a code decision, it is a scheduling bet on people who never agreed to your schedule. That bet is often worth making. What made it expensive here is that I placed it once and never priced it again, and nothing in my setup would ever have prompted me to.

The version without the dependency still exists. It is unreferenced now, reachable only because GitHub keeps orphaned commits after a force-push, and it is green. My patch has sat still for a year on top of a two-line change that was already written, already tested, and already passing.

Reimplementing Enough of Kubernetes to Fool kubectl

Seth Wheeler — Mon, 31 Aug 2026 14:00:00 +0000

Code: Megapixel99/nodejs-k8s

I have a repo that reimplements Kubernetes' core APIs in Node (pods, deployments, replica sets, services, jobs, namespaces, configmaps, the rest) backed by MongoDB instead of etcd, running "pods" as sibling Docker containers. It's the only project I've published that strangers actually looked at.

I don't think that's because the code is good. I think it's because of the claim in the README: point your real kubectl at it. Not a diagram of Kubernetes' architecture, not a from-scratch exercise, a thing you can falsify in one command. Every other repo I've written leads with how it was built, and none of them got read.

Making that claim true turned out to be almost entirely about protocol details that aren't in the resource schemas at all. Here are the ones I didn't see coming. Everything below is against kubectl v1.34.1, and you can reproduce the header captures yourself in about a minute.

kubectl's first move is discovery, and it will not proceed without it

Before kubectl get pods fetches a single pod, it asks the server what exists. Point kubectl at a server that 404s and watch what it sends:

GET /api?timeout=32s
Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,
        application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,
        application/json

GET /apis?timeout=32s
Accept: (same)

It asks for aggregated discovery first (a single document describing every group and version) and falls back to plain application/json if the server doesn't offer it. Then it retries. In my capture it hit /api and /apis five times each and gave up without ever requesting a pod.

This is the thing to implement first, and I didn't. You can have a flawless PodList handler and kubectl will never reach it, because the client refuses to guess at what the server supports. Two endpoints returning almost-empty JSON are the difference between "nothing works" and "everything works."

`kubectl get` asks the server to do the formatting

This is the one that genuinely surprised me. Once discovery succeeds, here's what kubectl get pods actually requests:

GET /api/v1/namespaces/default/pods
Accept: application/json;as=Table;v=v1;g=meta.k8s.io,
        application/json;as=Table;v=v1beta1;g=meta.k8s.io,
        application/json

as=Table. kubectl is not asking for pods and then formatting them. It's asking the server for a table (column definitions and rows of pre-rendered cells) and printing what comes back nearly verbatim. The NAME READY STATUS RESTARTS AGE header you've read ten thousand times is a string the API server chose.

Which means you can produce a convincing kubectl get pods without implementing Kubernetes at all. This is a complete server:

const http = require('http');
const J = (res, obj) => {
  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify(obj));
};

http.createServer((req, res) => {
  const path = req.url.split('?')[0];

  if (path === '/api')
    return J(res, { kind: 'APIVersions', versions: ['v1'] });

  if (path === '/apis')
    return J(res, { kind: 'APIGroupList', apiVersion: 'v1', groups: [] });

  if (path === '/api/v1')
    return J(res, { kind: 'APIResourceList', groupVersion: 'v1', resources: [
      { name: 'pods', singularName: 'pod', namespaced: true, kind: 'Pod',
        verbs: ['get', 'list', 'watch'] }] });

  if (path.includes('/pods'))
    return J(res, {
      kind: 'Table',
      apiVersion: 'meta.k8s.io/v1',
      columnDefinitions: [{ name: 'Name', type: 'string', format: 'name', priority: 0 }],
      rows: [{ cells: ['demo-pod'] }],
    });

  res.writeHead(404, { 'Content-Type': 'application/json' });
  res.end('{"kind":"Status","code":404}');
}).listen(8899);

Point a kubeconfig at http://127.0.0.1:8899 and run kubectl get pods:

NAME
demo-pod

That is real kubectl, fully satisfied, talking to sixty lines of Node that has no concept of a pod.

I find this genuinely good design once the surprise wears off. It's why kubectl get works on resource types your kubectl binary has never heard of, including CRDs that shipped after it was built: the server knows how to display them and the client doesn't have to. But it does mean "API-compatible" is a much bigger surface than the resource schemas suggest. Every one of the ~55 kinds I route needs its own table() returning its own columns. Pods return Name, Ready, Status, Restarts and Age; each column carries a type, a format, a priority and a description string.

There's a subtlety in the streaming case too. When kubectl watches a table, the column definitions should only appear on the first event: repeat them on every update and the client re-prints headers. So the watch path sends them once and then nulls the field:

Model.table([asJson]).then((table) => {
  if (eventType !== 'ADDED') table.columnDefinitions = null;
  eventStream.push(`${JSON.stringify({ type: eventType, object: table })}\n`);
});

The protobuf wire format is not protobuf

kubectl itself asked for JSON in every capture above. But client-go (which is what operators and controllers are built on) negotiates application/vnd.kubernetes.protobuf, and if you want those clients to work you have to speak it.

I assumed that meant "serialize the object with the .proto definitions." It's two layers more than that. Every message is:

A four-byte magic prefix: 0x6b 0x38 0x73 0x00, the ASCII bytes k8s followed by a null. Decoding starts by skipping it.
An Unknown envelope, a protobuf message with a typeMeta field (kind, apiVersion) and a raw field.
The actual object, encoded separately, stuffed into raw as bytes.

So you encode twice (the object into bytes, then those bytes into a wrapper) and prepend the magic:

let dataInfo = dataType.encode(prepareForProto(data)).finish();
let encoded = unknownType.encode({
  typeMeta: { kind: data.kind ?? '', apiVersion: data.apiVersion ?? '' },
  raw: dataInfo,
  contentEncoding: '',
  contentType: '',
}).finish();
return Buffer.concat([Buffer.from([107, 56, 115, 0]), encoded]);

The typeMeta duplication is the interesting part: kind and apiVersion appear in the envelope and inside the payload, so a client can route a message to the right decoder without decoding the payload first. Reasonable, and impossible to guess from the schemas.

Watch events add a third layer; each event is a WatchEvent message wrapping the already-wrapped object.

The failure mode is a wrong number, not an error

Kubernetes has scalar types that are structs on the wire and strings in JSON, and converting between them is where I lost the most time. Four of them:

JSON	protobuf
`"2026-08-13T10:00:00Z"`	`Time { seconds, nanos }`
`"100m"`, `"512Mi"`	`Quantity { string }`
`8080` or `"http"`	`IntOrString { type, intVal, strVal }`
large integers	`Long { low, high, unsigned }`

Quantity is the dangerous one. It looks like a string, so the obvious thing is to encode it as one. If you do, the Go client doesn't reject it: it decodes as zero. A pod whose CPU limit you carefully set to 100m arrives with a limit of nothing, and every layer reports success.

The fix is to know which keys hold quantities and wrap their values, which can't be inferred from the value's own shape:

const QUANTITY_MAP_KEYS = new Set([
  'limits', 'requests', 'min', 'max', 'default', 'defaultRequest',
  'maxLimitRequestRatio', 'capacity', 'allocatable', 'hard', 'used',
]);

I've now written this same paragraph about three different projects. A crawler whose rate limiter reported a correct delay while sending a hundred simultaneous requests. A browser whose memory instrumentation was the largest consumer of the resource it measured. And a serializer that turns a resource limit into zero and returns 200. In all three the code was locally correct, nothing threw, and the only way to see the bug was to look at what the other side received.

What I couldn't make work

One protobuf shape defeated me, and the comment in the code says so:

// Known-bad proto shape: our Event schema stores a populated
// series/deprecated* tree that protobufjs can't encode back into
// a wire-compatible EventSeries, which causes client-side decode
// failures. Force JSON for Event responses.

Events fall back to JSON. Clients that accept both are fine; a strict protobuf-only client would not be. I'd rather ship that with a comment than pretend the encoder is complete.

The larger honest list is in the README, and it's long: no CNI, so pods get a synthetic ClusterIP that routes nowhere. No CoreDNS. No CRDs, which rules out most real operators. No RBAC enforcement: every request is effectively cluster-admin. No server-side apply; apply-patch+yaml is accepted and quietly treated as a strategic merge. Conformance tests that touch any of that will fail no matter how much API surface I add, and I'd rather say which ones up front than let someone discover it after an afternoon.

The part worth generalizing

The technical lesson is that wire compatibility lives almost entirely outside the data model. Discovery, content negotiation, server-side printing, envelope framing, scalar coercion: none of it appears in a resource schema, and all of it is load-bearing. If I'd written the schemas first and the protocol second I'd have had a thing that looked complete and worked with nothing.

The other lesson is about how I describe work. "Point your real kubectl at it" is a claim a stranger can refute in one command, and that turns out to be the whole difference. My other projects are described in terms of the effort that went into them (written from scratch, no dependencies, built by hand) which asks the reader to take my word for something and gives them nothing to do. This one handed them a test. It's the only one anybody ran.

Checking a Cost Model Against a Stranger's Config File

Seth Wheeler — Sun, 30 Aug 2026 14:00:00 +0000

I wrote a cost model for streaming a mixture-of-experts model off an SSD, and the part I was proudest of was that it gated itself twice against artifacts I had nothing to do with. Gate 1 reproduced a third-party runtime's on-disk container byte-for-byte. Gate 2 predicted the size of a model conversion published by different people using a different tool, and came within 1.1% against a 2% tolerance.

The experiment behind that post has a limitations section I now want to revisit. It listed an ambiguity in the model's config.json, bounded the damage, and moved on:

dense_mlp_idx=2 is ambiguous: one dense layer, or layers 0 and 1. This takes one, which moves bytes/token by 2.4%. Immaterial to the conclusion, unresolved regardless.

The bound was right. The assumption was wrong. Correcting it took Gate 2 from 1.1% off to 3.4% off, which is outside the tolerance it had been passing, and the reason is more interesting than the arithmetic.

The header states the fact outright

A config file describes a model's architecture in the vocabulary of the framework that trained it; reading one therefore means inferring what a field name meant to its author. dense_mlp_idx=2 could reasonably be "the layer at index 2 is dense" or "layers below index 2 are dense", and those give 41 and 40 mixture-of-experts layers respectively.

I eventually downloaded the model in GGUF form (the container format llama.cpp uses) to run it. GGUF puts a block of key/value metadata at the front of the file, and that metadata is written by the conversion script after it has already resolved every question of this kind. So the answer was sitting in a file on my own disk:

inkling.block_count               = 42
inkling.dense_block_count         = 2
inkling.expert_count              = 256
inkling.expert_used_count         = 6
inkling.expert_feed_forward_length = 2048
inkling.expert_shared_count       = 2

dense_block_count = 2. Forty mixture-of-experts layers, not forty-one.

Reading it took about forty lines of Python. The parser was wrong the first time in a way worth repeating: GGUF stores arrays of strings length-prefixed, and I read only the first few elements without consuming the rest. Every subsequent offset was then shifted, and the parse failed later with an invalid-looking type code rather than at the array that caused it. If you write one of these, consume every element even when you intend to display four.

What the correction does

The expert size does not change, so the per-token cost moves purely through the layer count:

	assumed (1 dense)	header-confirmed (2 dense)
mixture-of-experts layers	41	40
routed experts on disk	148.58 GB	144.96 GB
streamed per token	3.482 GB	3.397 GB
Gate 2 against a published 153.5 GB	151.77 GB, 1.1% off	148.23 GB, 3.4% off

The model now under-predicts the published file by 5.27 GB, and the script does what I built it to do: it prints GATE 2: FAIL and refuses to emit any token-rate estimate at all. That refusal was written months earlier for exactly this case; it is the only reason I noticed, rather than quietly publishing a slightly different number.

Two errors, pointing opposite ways

Here's the mechanism, and it's the whole point of the post.

Gate 2 predicts a whole container: routed experts, shared experts, attention, embeddings, dense layers. The 1.1% agreement was the sum of two errors that happened to have opposite signs.

Error one, mine. An extra mixture-of-experts layer added 256 experts of 14,155,776 bytes each, inflating the prediction by 3.62 GB of routed experts; net of the dense layer it displaced, +3.54 GB.

Error two, structural and disclosed. The model does not attempt the vision and audio encoders, the multi-token-prediction module, the RMS norms, or the fact that the published conversion keeps non-expert tensors at 8 bits rather than 4. With the layer count correct, those account for -5.27 GB.

Add +3.54 to -5.27 and you land 1.73 GB low, which reads as 1.1% and looks like a model that understands the format. Remove the inflation and the deflation stands alone at 5.27 GB, or 3.4%. The residual was always that large. The agreement was partly luck, and a tolerance was never evidence that the two sides matched for the same reason.

I want to be precise about what this does and does not undermine, because "a gate failed" is easy to over-read.

Gate 1 is untouched and still byte-exact. It compares computed integers against a published layout with no tolerance at all: expert stride 1,769,472, per-layer blob 452,984,832, 256 experts, 40 layers. Equality has no room for two errors to meet inside it; that is the argument for preferring an exact gate to a tolerance wherever the domain offers one.

The load-bearing number got better, not worse. The whole model exists to produce one figure: bytes streamed per token. That figure depends on the expert stride (Gate 1 verifies it exactly), the number of experts routed per token, and the mixture-of-experts layer count. The header now confirms the last two outright, so 3.397 GB/token rests on three measured quantities where 3.482 rested on two measured and one inferred.

So a gate went from pass to fail while the number I actually care about became better grounded. That is only a contradiction if you think a container-size prediction and a per-token-cost prediction are the same claim. They share inputs and they fail differently, and this is the case that separates them.

Two independent checks agree with the corrected figure. A separate script that estimates per-token bytes from published file sizes rather than from architecture gives 3.405 GB against 3.397 computed directly, 0.2% apart. And when I finally ran the model, the kernel's page-in counters put actual disk traffic between 1.80 and 3.96 GB per generated token, a bracket that contains it.

What I'd take from this

The rule I had been operating under is that a gate passing within tolerance means the model is right. It doesn't. It means the sum of every error sits inside the tolerance, and a sum can be small because the terms are small, or because they cancel. Those two situations look identical from outside and behave completely differently when you fix something: one improves, the other exposes what the cancellation was hiding.

There's a cheaper habit buried in this, too. I spent real effort inferring what dense_mlp_idx meant, wrote a careful bound around my uncertainty, and shipped the inference. The artifact stated the fact outright in a header I could read in forty lines. When you find yourself bounding an inference about someone else's format, check first whether some downstream tool has already resolved it and written the answer down, because conversion scripts have to answer these questions to do their job at all.

And the disclosed limitation earns its keep again. The last post's failure was attributable because the caveat had been written in advance; this one was detectable because the tolerance and the refusal-to-proceed were written in advance. I keep finding that the value isn't in the estimate, which gets superseded. It's in having decided, before the measurement, what would count as being wrong.

The code for this one isn't public. It's a research repo with no release, and I'd rather link nothing than link something nobody can run.

Predicting the Speed of a 276B Model Streamed From an SSD

Seth Wheeler — Sat, 29 Aug 2026 14:00:00 +0000

A mixture-of-experts model only activates a few of its experts per token, which means (unlike a dense model, where every forward pass touches every weight) you can leave the weights on disk and read in just the ones each token routes to. That turns "does this model fit in RAM" into "how fast is your SSD," and it puts models far larger than your machine nominally supports within reach, at some token rate.

I wanted to know what that rate actually is. The question that started it was concrete: can Inkling-Small (276B total parameters, 12B active) run on a 24 GB Mac mini?

The answer wanted to be arithmetic rather than a guess. This is what happened when the arithmetic met the machine.

Act 1: a prediction is worthless unless it can be wrong

I wrote a ~250-line cost model with no dependencies that downloads no weights. Every input is a config.json or manifest.json; every output is a byte count derived from it. It models MLX affine group quantization the way it's actually stored on disk: per quantized projection, the weight bytes plus one bf16 scale and one bf16 bias per group of 64 values along the input dimension.

The obvious failure mode is a model tuned until it agrees with the one thing you checked it against. So it's gated twice, against artifacts I had nothing to do with.

Gate 1: reproduce a real container byte-for-byte. Swiftlet is a third-party Swift/Metal runtime that stores MoE weights in its own on-disk format. From a model's config.json alone, predict the layout it publishes. These are exact integers, so the gate demands equality, not closeness:

quantity	computed	published
expert stride	1,769,472	1,769,472
experts per layer	256	256
per-layer blob bytes	452,984,832	452,984,832
layer count	40	40

All four exact. The full container total came within 0.137%, the residual being the tokenizer and chat template, which the model doesn't attempt.

Gate 2: predict a stranger's conversion. Gate 1 could still pass by being fitted to the one container it was checked against. So: predict the on-disk size of a different model, converted by different people using a different tool. Predicted 151.8 GB against a published 153.5 GB, 1.1% off, against a 2% tolerance. The residual is unmodelled and deliberately not fudged: vision and audio encoders, RMS norms, 8-bit router gates, safetensors headers.

Correction, 2026-08-14. This gate no longer passes, and the 1.1% was two errors cancelling. The model assumed one dense-MLP layer where the published model's own header states two, so it counted 41 mixture-of-experts layers instead of 40. Correcting that drops the prediction to 148.23 GB, which is 3.4% off and outside the 2% tolerance. The extra layer had been inflating the prediction by 3.54 GB while the genuinely unmodelled components listed above deflated it by 5.27 GB, so the two nearly cancelled and the residual was always larger than 1.1% suggested. Gate 1 above is unaffected and still exact. The follow-up post works through it.

Both gates run on every invocation, and the script refuses to print any token-rate estimate if either one fails. That mattered later.

The interesting structural result was that total parameter count is nearly irrelevant. What sets the per-token cost is experts_per_token × layers × expert_size. And the useful finding was a cliff rather than a gradient: the fraction of each model's expert set that fits in 24 GB of RAM:

model	cacheable in 24 GB	streamed per token
Qwen3.6-35B-A3B	93.7%	0.566 GB
Inkling-Small (276B)	10.0%	3.397 GB

Correction, 2026-08-14. That last figure read 3.482 GB when this post went up, for the layer-count reason described above. The corrected 3.397 GB raises the predicted token rate quoted later in this post by a factor of 1.025, which is far too small to affect the 23× conclusion. The Qwen row is unaffected.

That 93.7% explains something I'd otherwise have taken at face value. Swiftlet's README reports its decode loop is "dispatch bound, not IO bound," which sounds like a claim about kernel quality. On a 24 GB machine, after warmup, almost all of the 35B's experts are resident; it is barely streaming at all. The streaming machinery is what lets it start in a couple of gigabytes, not what it lives in. Past the cliff, no amount of kernel work helps.

Now I just needed a GB/s number to turn bytes per token into tokens per second.

Act 2: the benchmark was measuring RAM

An earlier experiment of mine had priced disk reads at "a nominal 100 µs" and said so in its own limitations; the latency was an assumption, not an observation. So before trusting any rate, I measured the SSD.

The first version reported 11–24 GB/s on a device that does about 7.

The cause is a specific and easy misreading of a macOS API. fcntl(fd, F_NOCACHE, 1) stops the kernel from caching future reads on that descriptor. It does not evict pages that are already resident. I was reading a file I had just written, so it was entirely in the unified buffer cache. The flag was set, it did exactly what it says, and the benchmark measured memory bandwidth.

What makes this the interesting half is that no flag fixes it. The fix has to be structural: write a file 2.5× larger than physical RAM (160 GiB on this machine) and sample only from its first quarter, whose pages the later 120 GiB of writes necessarily evicted.

I also stopped trusting myself and added two gates that run every time:

Content gate: every timed read must contain a magic header written at that offset. This catches short reads, misalignment, and filesystem tricks returning zeros at implausible speed. It immediately caught a page-alignment bug: 8 of 384 reads verified.
Cache gate, read the same offsets twice. With caching genuinely off, the two passes must agree. If pass 2 is much faster, the cache is serving reads and the run is void. It came out at 1.10×.

The real numbers, in the access pattern a router actually produces (top-k random experts inside one layer's blob, then the next layer) were 3.54 GB/s issued serially and 7.82 GB/s with six reads in flight.

That 2.2× is the single biggest lever in the whole exercise. Issuing the top-k expert reads one at a time leaves half the device idle. For Inkling-Small it is the difference between roughly 1.1 and 2.5 tok/s: between unusable and marginal.

A separate measurement at 4 KB found something worth designing against: the mean read is 154.1 µs, but p99 is 757.6 µs, 5.8× the median. A mean-based estimate is structurally blind to that, and any paging design needs a deadline policy that answers from whatever is resident rather than waiting on the tail.

The generalizable version of this act: when a mechanism is supposed to disable an optimization, verify the optimization is actually off rather than trusting the switch. I had a flag that did what it promised, and a benchmark that was wrong by 3×.

Act 3: running it, and missing by 23×

Prediction, with the measured SSD: 1.46–3.22 tok/s.

Getting to a real number was mostly an exercise in discovering that published weights are not evidence that anything can load them. Four plausible runtimes:

path	status
Swiftlet	Different architecture family. A port, not a config change.
MLX / mlx-lm	`mlx-lm` 0.29.1 ships 100 model modules and this architecture isn't one n/a despite a converted 4-bit build existing on HuggingFace.
llama.cpp upstream	Release b10288's `libllama.dylib` contains zero occurrences of the architecture name. The GGUFs declare it.
llama.cpp PR #25731	The only path. Still open, 9 commits, 63 files.

Three of those four have published quantized weights and no working loader. The GGUF ladder exists because its author built it against their own unmerged branch.

It runs. On an M1 Max, CPU-only, streaming 127.4 GB from SSD via mmap:

	measured	predicted
decode rate	0.064 tok/s (15,678 ms/token)	1.46–3.22 tok/s
paged in from disk	59.5 GB over 33 tokens	n/a
→ per generated token	1.80–3.96 GB	2.926 GB ✓

The bytes side held. 14,516,912 page-ins over 18 prompt plus 15 generated tokens brackets the predicted 2.926 GB/token: 3.96 GB if every page-in is charged to the generated tokens, 1.80 GB if spread across all 33. The batched prompt routes to far more than six experts per layer, so the true figure sits inside that bracket. Config arithmetic and kernel page-in counters are entirely independent methods, and they agree. That's the thing the experiment existed to get.

The rate side missed by 23×. And here is the part I actually care about.

Act 1's write-up carried this in its limitations, written months before any of this ran: compute is not modelled at all; every rate here is an IO ceiling, the real rate is the minimum of IO and compute, and this experiment cannot tell you which binds.

That is precisely what happened. The only configuration that completes a forward pass is CPU-only: every Metal configuration dies with an out-of-memory error, because ~125 GB of experts leak onto a GPU with a 51.5 GB working set, and the flag that's supposed to keep them off it doesn't work for this architecture. So 12B active parameters get multiplied on 8 CPU threads with no GPU, and that sets 15.7 seconds per token. The SSD was never the constraint.

The IO ceiling was never wrong. It was just not the binding term in the only configuration available, and the run that works is therefore also the run that cannot test the ceiling. The 24 GB mini's predicted rate remains unmeasured, and I'd rather say that than quote 0.064 tok/s as though it answered the question.

The part that was worth the whole exercise

There's an operational footnote I'll keep short: a process streaming an oversized mmap'd model can become genuinely unkillable, sitting in uninterruptible-exiting state for 40 minutes after SIGKILL, because it cannot finish unwinding 127 GB of mapped memory while blocked in the page-fault path. pkill reports success. Only ps -o stat= shows otherwise. On unified memory this starves everything else on the machine, which is a good reason not to do this on a box something else depends on.

But the thing I'd take away is about the caveat.

The estimate was not the valuable output. Estimates get superseded. What made this worth doing is that the failure was attributable: the prediction missed for a reason that had been written down, in advance, as the thing the model could not account for. I didn't have to reconstruct an explanation after the fact, and I didn't get to choose a flattering one.

That is the difference between a model that taught me something and a number that happened to be wrong. A caveat written before the measurement is a prediction about your own blind spot; a caveat written after it is an excuse. They read identically in the final write-up, which is exactly why the order matters.

I've now had this pattern three times in a row. A crawler whose rate limiter enforced a perfect delay while sending a hundred simultaneous requests. A browser whose hibernated tabs cost three orders of magnitude more than I'd modelled. And a cost model that was right about every byte and wrong about the clock. In all three the arithmetic was fine and the boundary of the arithmetic was where the truth was hiding, so the useful discipline isn't being more careful inside the model, it's being explicit about where the model stops.

What a Hibernated Browser Tab Actually Costs

Seth Wheeler — Fri, 28 Aug 2026 14:00:00 +0000

I've been building a macOS browser called Kestrel (about 7,900 lines of Swift on top of WKWebView) to test one idea: that a browser should hold a memory budget you give it, the way a game engine holds a frame budget. You say 800 MB, and the browser demotes background tabs down a ladder of progressively cheaper states until it fits, rather than either swelling without limit or destroying tabs outright.

Before writing any of it I simulated the policy against real tab-usage distributions. The simulation said the scheduler would use 7.9–11.5× less memory than an unmanaged browser, with zero state-losing reloads.

Built against an actual engine, it delivers 1.4–2.6×, and state loss is reduced by 18–42% rather than eliminated.

The whole gap traces to a single number. The simulation priced a hibernated tab at 32 KB: a session descriptor and a thumbnail. WebKit charges 39 MB, because the renderer process survives and cannot be terminated on request. Three orders of magnitude, in the one parameter every downstream claim rested on.

Kestrel isn't public yet, so this is a description rather than an invitation to read the code. The measurements below are all on the same machine: M1 Max, macOS 15.5, Swift 6.1.2, system WebKit.

The ladder, and what each rung actually costs

The design has four states per tab. LIVE is a normal loaded page. WARM is frozen but resident. COLD keeps a serialised session image and throws the page away. STUB destroys the web view entirely and remembers only a URL.

The premise is that these get monotonically cheaper, so a scheduler can walk a tab down the ladder until the budget is satisfied. Driving one tab through every rung, measured with phys_footprint:

rung	measured	% of LIVE	recovered	design target
LIVE	128 MB	100%	n/a	< 25 MB
WARM	106 MB	83%	17%	< 2 MB
COLD	39 MB	30%	70%	~2 KB
STUB	59 MB	46%	n/a	~2 KB

Two things in that table are wrong in ways I didn't anticipate.

WARM recovers 17%, against a target of under 2 MB. WARM is everything a host application is permitted to do to a page short of destroying it: detach it from the view hierarchy, setAllMediaPlaybackSuspended(true), clear every timer and interval. That buys 17%, and the reason is structural: a host app cannot compact WebKit's heap. The design's WARM assumed a full compacting GC followed by madvise, and compaction is the step that actually returns pages to the OS. There is no API for it. WARM as specified is an engine-internal operation being attempted from outside, and it doesn't work.

I had swept this parameter in simulation from 0.02 to 0.50. Reality is 0.83: worse than the most pessimistic case I tested.

STUB costs more than COLD. 59 MB against 39 MB. Destroying the web view is more expensive than keeping it and navigating it to about:blank. So the ladder is not monotonic on this engine, which means a scheduler cannot assume an ordering; it has to measure each rung's cost and sort by the result.

Why COLD can't get below a whole process

COLD captures WKWebView.interactionState and navigates the page away. That recovers 70%, and leaves 39 MB of WebContent process behind that nothing I can reach will reclaim.

I tried three separate things to kill that process, because I did not believe it at first:

Release the last reference to the WKWebView. Process still alive after 110 seconds.
Give each tab its own WKProcessPool, so the pool dies with the view. No effect.
Navigate to about:blank and release the view. 59 MB, still alive: worse than navigating away and keeping the view, which is why the COLD path now does the latter.

This is WebKit's WebProcessCache doing exactly its job. It keeps renderer processes warm so the next navigation is fast. It's a reasonable trade that happens to be precisely the wrong one for a memory scheduler, and there's no supported way to opt out. Private SPI (_WKProcessPoolConfiguration.usesWebProcessCache) would likely make teardown deterministic, at the cost of leaving supported API.

One caveat I want to be honest about: the process cache is bounded and evicts under memory pressure, so the floor at scale is probably not 39 MB × N tabs. I haven't tested that. The naive extrapolation says 80 tabs would floor at 3.1 GB, and I don't believe that number; I just haven't disproved it.

A null result that read like a finding

The first end-to-end comparison of three policies on real sites returned this:

none        mean 197.8 MB  peak 320.0 MB  over budget 0%  demotions 0
discardlru  mean 182.3 MB  peak 278.0 MB  over budget 0%  demotions 0
kestrel     mean 181.0 MB  peak 274.0 MB  over budget 0%  demotions 0

Three nearly identical rows look like "the policies are equivalent." They actually meant the experiment never ran. The budget was 400 MB and the peak was 320 MB, so no policy ever had anything to do.

There was a second methodological bug underneath. My revisit trace was a Zipf distribution over an LRU stack (the same model the simulation used) which concentrates so hard on recently-used tabs that with 10 tabs and 40 events, only 9 were ever loaded. In simulation this was masked, because tabs were created by the trace as it ran. Replaying the same model against a fixed tab set is not the same experiment.

The fix was to open each tab once before revisiting, the way a user filling a window does, and to lower the budget until it binds. I'm recording it because the failure mode is easy to miss in the direction that flatters you: when every arm of an experiment agrees, check that the independent variable actually varied before believing the result.

Where the advantage went

With the budget binding, two workloads. Light pages are ten real sites (Wikipedia, MDN, Hacker News, go.dev) at a 150 MB budget:

policy	mean	peak	over budget	tabs destroyed
unmanaged	319.9 MB	368.6 MB	88%	0
discard-LRU	104.6 MB	140 MB	0%	17
Kestrel	121.7 MB	144 MB	0%	14

That's a loss. More memory than discard-LRU for a rounding error's worth of saved tabs.

The warm-up ramp explains it. Per-tab LIVE cost on those ten sites was 37, 50, 32, 24, 5, 17, 111, 31, 22 and 40 MB: a median of 31 MB. The COLD floor is 39 MB. On the light set, hibernating a tab costs more than leaving it running. Seven of ten tabs are cheaper live than cold; the compression ratio is 0.95×.

So the ladder had nowhere to put anything. Kestrel parked zero tabs at COLD in that run (the rung does not appear once in the trace) and fell through to STUB, which is what discard-LRU already does, only with more bookkeeping.

Heavy pages are a synthetic 20,000-node DOM with a retained JS heap, at a 500 MB budget. Compression ratio 3.28×:

policy	mean	peak	over budget	tabs destroyed
unmanaged	983.7 MB	1111 MB	90%	0
discard-LRU	420.0 MB	452 MB	0%	12
Kestrel	411.5 MB	482 MB	0%	7

Same memory as LRU, 42% fewer destroyed tabs. The advantage tracks the compression ratio, which is the one piece of evidence here that I understand the mechanism rather than merely observing it.

The rule the simulation never surfaced

Falling out of the 39 MB floor is a feasibility condition:

budget must exceed  (live working set) + (COLD floor × parked tabs)

For 10 heavy tabs at a 500 MB budget: roughly 384 MB of protected live set plus 273 MB of parked tabs is 657 MB of demand against 500 MB of budget. Which is exactly why 7 tabs still had to be destroyed: below that floor, no policy can do better.

That's a prediction, so I tested it. Four protected live tabs at 128 MB plus six parked at 39 MB is about 746 MB, so 800 MB should be enough for the ladder to reach zero state loss:

policy	mean	peak	over budget	tabs destroyed
unmanaged	968.2 MB	1111 MB	82%	0
discard-LRU	668.5 MB	765 MB	0%	5
Kestrel	678.8 MB	787 MB	0%	0

Kestrel destroys nothing where discard-LRU destroys 5, for 1.5% more memory. State loss across all three runs tracks the rule exactly:

run	budget	headroom vs COLD floor	LRU destroys	Kestrel destroys
light pages	150 MB	far below	17	14
heavy pages	500 MB	below	12	7
heavy pages	800 MB	above	5	0

Note what the 800 MB run does not show: a big memory reduction. It's 1.43× below unmanaged, against 2.4× at the tighter budget. That isn't a regression; the budget is the control input, and a looser budget buys less reduction by construction. What the ladder buys isn't a multiplier. It's the ability to hit whatever budget you name without shredding tabs: 0% of samples over budget against unmanaged's 82%, with zero tabs destroyed.

Two things did survive intact. Restore from COLD to a live page takes 82 ms, timed to didFinish rather than to the API returning, against a design target of 100. And p95 restore latency was lower for Kestrel than for discard-LRU on both workloads (397 ms vs 400 ms, and 3541 ms vs 3546 ms) because deserialising a session image beats refetching the page.

The instrumentation became the largest cost in the system

This is my favourite bug in the project, because of what it is a bug in.

The complaint was "scrolling seems clunky." Not a hang, not a crash, just slightly wrong. A screen recording put a number on it: during a ten-second scroll, 25% of frames were pixel-identical to their predecessor. One frame in four, dropped.

Tab.currentBytes returned a real, current measurement of a tab's memory. It was correct by construction. It got that measurement by shelling out to /usr/bin/footprint, which costs 226 ms per call, and it did so on every read. The scheduler's budget loop read it. The status bar read it. Every row of the tab strip read it. About eight times per 1.5-second UI tick: roughly 1.8 seconds of main-thread blocking per 1.5 seconds of wall clock.

It didn't present as a hang because WebKit composites pages in their own process. The page kept scrolling in a process that wasn't blocked, just badly, while the process responsible for deciding how much memory to use spent all of its time finding out.

The fix is unglamorous: sample on a background queue, let the UI read a cache. 1000 reads now take 0.13 ms, with a regression test asserting that reads stay free.

But I'd already made this mistake once, in a different place. An earlier tier-down implementation optimised the number reported by about:memory while returning nothing at all to the OS: the wrong number, measured well. This is its mirror image. A memory manager that stalls the UI to find out how much memory it's using has spent more than it can possibly save. Anything you sample per frame or per tick has to be cheap enough to be free, or it becomes the problem it was added to observe.

What I actually learned

Three things the real engine confirmed, and they're the three the design needs to exist at all. Per-tab memory is genuinely attributable; each WKWebView gets its own WebContent process, and four identical tabs measured 130.0 MB each, identical to the decimal, which is what you want from a control. (WKWebView exposes no pid, so I attribute processes by set-diffing ps output as tabs are created one at a time.) The session image I thought I'd have to invent already ships as interactionState: 138 bytes for my probe page. And restore is fast enough that a user doesn't perceive it.

What broke is cleanly separated from what held, and the split is not where I expected. Everything in the browser's policy layer (the budget, the scoring, the floors, the fail-safe that refuses to demote a tab for a trivial gain) ported to a real engine without modification. Everything in the engine layer (heap compaction, cheap frozen tabs, process teardown) cannot be built on top of an engine that doesn't offer it. No amount of additional simulation would have told me which of my components were which, because the simulation was where I'd encoded the assumption.

The specific correction I'd give my earlier self: I modelled a hibernated tab as a data structure, when on this platform it's a process. Data structures are free to throw away. Processes are only free to throw away if something will let you kill them.

It also produces an argument I didn't expect for a fork-server design, from a direction I wasn't looking: everyone talks about cheap process startup. Cheap process teardown turns out to matter just as much, and only one of those is something a host application can work around.

A Better FP4 Gradient Quantizer That Training Couldn't Notice

Seth Wheeler — Tue, 25 Aug 2026 14:00:00 +0000

Four-bit training is the current frontier of making LLM pretraining cheaper. NVIDIA's Blackwell GPUs do 4-bit matrix math several times faster than 16-bit, but 4 bits means every number gets rounded to one of 16 values. The NVFP4 format's menu is exactly {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}, stretched by one scale factor per block of 16 numbers. Choosing that scale is most of the game. The published state of the art for gradients (an estimator called MS-EDEN, from the Quartet II paper, arXiv:2601.22813) chooses it the safe way: scale so the block's largest value lands exactly on 6, and nothing clips.

I found a scale rule that beats it by 14% on its own metric, on every real gradient tensor I tested. Then I rented two GPUs to watch the improvement show up in training loss, and it never did; four estimators landed within 0.09% of each other at 2.8B parameters. The number that explains both halves cost nothing to measure, and I measured it last, which is the embarrassing part. My batches were between 35 and 643 times too small for the difference to be visible. The batch size where it does become visible is the one frontier labs actually train at.

The code isn't public, so this post carries the numbers instead. MS-EDEN here is my reimplementation (it reproduces the paper's error figures, not its unbiasedness proof), and the training corpus is FineWeb-Edu, tokenized with GPT-2's BPE.

The win, and where it comes from

A block of 16 gradient values is roughly bell-curved. Its largest value sits around 2 to 2.5 standard deviations. Scaling that value onto the grid's top level spends the two coarsest levels (4 and 6) on numbers that almost never occur, and it starves the middle of the distribution where nearly everything lives.

The alternative is what anyone tuning a lossy codec would try: sweep 17 candidate scales per block and keep the one that minimizes mean squared error. Values above the top level clip; sacrificing the rare outlier buys precision for the common case. On synthetic data that's worth +14.0% against MS-EDEN on Gaussian blocks and +25.0% on heavy-tailed ones. On 45 real weight-gradient tensors captured across three phases of training a small transformer, it wins on 45 of 45, mean +13.8% (worst tensor +9.6%, best +15.3%).

It stays hardware-valid, since the output is still the FP4 grid times one per-block scalar. And the caveat the experiment printed in its own output at the time matters for everything that follows: MSE is a proxy, and the training-loss impact was unvalidated.

The wall, then a landmine

Making the improvement provably unbiased failed, and the failure generalizes. On a fixed 16-level grid you can have a provably unbiased estimator (stochastic rounding, no clipping) or a low-error one (clipping), never both. The error reduction is the clipping, and stochastically rounding a clipped value is biased by construction. I later checked whether this was just NVFP4's menu being bad. It isn't: optimizing the palette itself moves the number but not the shape, and every 16-value palette I tested holds an unbiased-to-biased error ratio between 2.2× and 2.6×.

The standard escape is error feedback: keep the part you rounded off and add it back next step, Kahan summation applied to gradients. Each step stays biased, but the running sum doesn't. That works exactly as advertised under SGD. On a coarse 3-level quantizer, a quantized-plus-feedback run lands on the full-precision trajectory to three digits (2.31e-5 against 2.33e-5, distance to the optimum).

Under Adam it's a landmine. Adam rescales every update by a running second-moment estimate. The carried residual passes through that nonlinearity, and the guarantee quietly dies. Measured on the same harness, with orderings that hold at three sigma across seeds:

arm	distance to optimum (Adam)
full precision	8.86e-5
quantized, no feedback	9.59e-5
quantized + error feedback	1.78e-4
quantized + error feedback, second moment frozen after warmup	1.67e-5

Error feedback under Adam was worse than no correction at all. Freezing the second moment (the fix the 1-bit Adam paper used for compressed communication) produced the best arm in that table, and I originally wrote that it repairs error feedback completely. The follow-up post ran the ablation I skipped, and the credit was misassigned: on this harness a frozen v helps just as much with no feedback, and with no quantization in the run at all. The fix was real; it just wasn't fixing what I said it was. What survives is the landmine itself: don't bolt error feedback onto Adam. I'd been running exactly that broken configuration on the GPUs without knowing it. At FP4's error magnitudes it was too small to hurt, but the default was wrong.

Two GPUs that saw nothing

On a rented RTX 4090, a 0.5B-parameter GPT trained on 98M tokens. FP4 gradient quantization costs a real +2.14% in held-out loss against bf16, and my estimator differs from MS-EDEN by 0.01% (6.4442 against 6.4433, which is noise). The error-feedback arm never finished. Its fp32 residual buffer is a full parameter-sized copy, the card has 24 GB, and the log ends in a CUDA out-of-memory error trying to allocate 1.54 GiB with 773.69 MiB free.

So: a B200 with 192 GB, at $8.619 an hour, and a 2.8B-parameter model on 198M tokens, with the residual buffer halved to bf16 and activations checkpointed. All four arms ran, 2.06 hours of training time across them:

arm	final loss	vs bf16
bf16	7.6890	n/a
MS-EDEN	7.6868	-0.03%
optimal scale	7.6823	-0.09%
optimal scale + error feedback	7.6896	+0.01%

Everything sits inside a 0.09% band, which is seed noise. Even the 4090's 2.14% bf16-versus-FP4 gap vanished, because 800 steps into a 2.8B model is nowhere near convergence. There was very little converged signal for quantization error to corrupt.

(An operational note, because it cost an hour of B200 billing: a relaunch command that ran pkill -f scale_experiment.py in the same shell that then launched scale_experiment.py matches its own command line and kills its own launcher. Every relaunch mechanism I tried "mysteriously" died until I read the exit code properly. Two other sessions on this machine have since hit the same trap. The fix is pgrep -f "patt[e]rn", where the brackets break the self-match.)

At this point the honest summary was: real on the proxy, invisible on the metric that matters, at every scale I could afford. I nearly wrote exactly that and stopped.

The number I should have measured first

The reason every run came back "within noise" has a formula. A gradient estimated from a batch of B tokens carries sampling noise that shrinks as 1/B. Quantization error doesn't shrink with batch size at all. Divide one by the other and you get the batch size where a quantizer's error, or the difference between two quantizers, pokes above the sampling-noise floor. Both quantities are measurable on a tiny model in minutes, on a laptop, for free.

Measured on real gradients (64-microbatch statistics, three training phases):

	step 30	step 100	step 300
estimator gap (relative MSE)	+16.0%	+13.9%	+16.5%
gradient-noise scale (tokens)	1,035	5,778	2,469
estimator gap visible at batch ≈	868,331	5,264,559	1,915,591

My runs used batches of 8,192 and 24,576 tokens. Depending on the run and the training phase, that is 35 to 643 times below the batch where the gap could have appeared in loss. The experiments were not measuring a small effect; they were structurally incapable of measuring any effect. It's the benchmarking mistake in different clothes: evaluating a 1% optimization on a machine with 20% run-to-run variance, then concluding the optimization does nothing.

The crossover isn't in exotic territory either. One to five million tokens per batch is the standard operating point of frontier-lab pretraining. Measured where I could afford to train, the difference is invisible; extrapolated to where the labs train, it sits right at the threshold of visibility. Two caveats travel with that. The crossover was measured on a 3M-parameter model, so treat it as an order of magnitude. And the confirming experiment (a ~124M-parameter model at a ~1.6M-token accumulated batch, where quantization overhead amortizes to nothing) is specced and smoke-tested but has not run. The prediction is a prediction, not a result.

One consolation prize from the same afternoon of cheap experiments: after the standard Hadamard rotation makes blocks Gaussian, plain evenly-spaced INT4 beats NVFP4's float-style palette on both rounding modes (0.01445 against 0.01789 unbiased, 0.00621 against 0.00685 biased, normalized MSE). I first attached a condition to that, reasoning the uniform grid only wins after the rotation has removed the outliers the float spacing exists for. The follow-up that tested the condition on real tensors killed it: uniform INT4 wins on 41 of 45 gradient tensors with no rotation at all, because my heavy-tailed synthetic data had put the outliers somewhere real gradients don't. The practical half survives untouched; integer 4-bit math predates Blackwell by years.

What generalizes

A null result is a claim about your noise floor at least as much as a claim about the effect. I ran the training experiment twice, on two GPUs, before measuring the floor once. The floor measurement took minutes, cost nothing, and retroactively explained both runs; it also converted "needs a converged frontier run to settle" into "needs one afternoon at a large batch." The expensive experiments weren't wrong, but every dollar of them was spent below the detection threshold, and the threshold was computable in advance.

I keep relearning the same discipline in different domains. Before paying to measure an effect, price the noise you'll be measuring it through. The signal has to clear the floor, and the floor rarely announces itself. Mine was 35 to 643 times too high, and nothing in the loss curves even hinted at it.

Why a Small Transformer Can't Copy a Word It Hasn't Seen

Seth Wheeler — Mon, 24 Aug 2026 14:00:00 +0000

I have a small transformer that turns a one-line spec into a working web app. It is 11.9M parameters, 6 layers, d=384, trained on 4,176 generated programs. Given "a support ticket system with marking a ticket closed, a stats page, searching tickets and creating and viewing tickets" it writes a 1,158-token Python file that compiles, serves HTTP, implements search, stats and toggle, and does not implement comments, edit, delete or category. On held-out feature combinations it does that 5 times out of 6.

Ask it for a book catalogue and it writes an appointment booker.

I built this arm to justify a specific claim, and the claim turned out to be wrong. This post is that retraction, plus the three experiments it took to find out why the model fails, one of which corrected a diagnosis I had already written down and believed.

(I have written about this project once before, on how its verification sweep reports on itself. That post is about the symbolic half of the same system; this one is about the learned half and does not depend on it. The code is in a private research repo, so there is no link. Every figure below comes from a results file or a command I ran, and I say which.)

The claim I was defending

The shipped tool does not use the model at all. It uses compositional synthesis: a parser turns your request into an entity schema plus a feature set, and hand-written emitters assemble exactly that program. It is exact within its grammar and about 2 ms on a CPU.

Its cost is human labour. Each program kind is 160 to 287 lines of hand-written emitter, each language is another 104 to 231, and there is no transfer between them. I wrote in two separate experiment write-ups that this is "precisely the labour a learned generator would amortise." A model, the argument went, would learn the mapping once and cover new cases for free.

For a long time that was untestable, because the model scored 0%.

0% was the plumbing, four times over

Five experiments in a row scored 0/6 compile. The diagnosis I kept reaching for was capacity or training time, and it was wrong every time. Training to 3.5× lower loss left the failure distribution identical.

What actually fixed it was two one-line changes to the data pipeline:

Word-split docstrings. Every app in the corpus gets a spec-derived docstring, and docstrings were tokenised atomically, so every held-out app's docstring was <unk> by construction. 4,141 of 5,953 vocabulary entries were one-off docstrings. Splitting them into words took the vocabulary to 1,817.
An <eos> token. Training apps had nothing marking the end, so nothing taught the model to stop, and every previous arm's output was mechanically truncated at the 3,200-token cap.

Same architecture, same data, same optimizer. The results file (021-pipeline-fixes/fixed_combo.json) records what happened:

metric	before	after
emitted `<eos>`	n/a	6/6
compile	0/6	6/6
serves HTTP	0/6	5/6
all features, strict	0/6	5/6 (83%)

Strict means the requested features work and the unrequested ones are absent. The model does both. Dev loss improved as well, 0.034 to 0.026, from deleting 4,100 vocabulary entries: entries that can only ever be copied are pure liability, since they consume capacity and are unusable on held-out input.

So the generator worked, and the claim became testable.

The test, and the retraction

The library domain (entity book, table books) was held out of the corpus before any training. If the amortisation argument were right, this is where it would show: a new noun costs the symbolic system a schema synthesiser of about 15 lines, and should cost the model nothing at all.

From 025-generator-transfer/fixed_domain.json, on 8 held-out requests:

metric	seen domains	unseen domain
emitted `<eos>`	6/6	8/8
compile	6/6	7/8
serves HTTP	5/6	3/8
strict, all features	5/6	0/8
table hit / route hit	6/6	0/8
mean purity	1.00	0.00

Zero. Not a degradation, a floor.

I checked what it wrote instead by reading the eight generated files rather than trusting the summary. Every one names a table it was trained on:

000_library_basic.py      contacts
001_library_delete.py     tickets
002_library_edit.py       bookings
003_library_search.py     contacts
004_library_toggle.py     bookings   (with `tickets` leaking in elsewhere)
005_library_comments.py   bookings + requests
006_library_stats.py      bookings
007_library_category.py   bookings + requests

bookings for books is the tell: the model reaches for the lexically nearest entity it was trained on. The apps are internally consistent for the wrong entity, which is why 7 of 8 still compile while 0 of 8 are right.

Here is the part that made me stop defending the argument. This is the top of 006_library_stats.py, unedited:

# Spec:a simple book lending app offering a stats page and creating and viewing books
...
"""A simple address book app offering a stats page and creating and viewing bookings"""
PORT = int(os.environ.get('PORT', '8806'))
DB = os.environ.get('DB', 'bookings.db')
con.execute('CREATE TABLE IF NOT EXISTS bookings (id INTEGER PRIMARY KEY AUTOINCREMENT, guest TEXT, details TEXT)')

Line 1 is the prompt. The model then copies nine consecutive words of it verbatim ("offering a stats page and creating and viewing"), swaps book lending for address book, swaps books for bookings, and gives the table a guest column. The word it needed was in its context, on line 1, and I confirmed that across all eight files: the string books appears exactly once in each, always on line 1, never in the code.

So the model can copy a nine-word span and cannot copy the one word that decides correctness. The amortisation claim is dead: on the axis where a learned model was supposed to win, it scores 0/8 while 15 lines of symbolic schema synthesis score 10/10.

The diagnosis I got wrong

I wrote down the obvious conclusion: this is a copying problem, and the remedy is pointer attention or byte-level entity tokens.

Before building any of that, a 30-line probe against the checkpoint (029-entity-generalization/probe_vocab.py) checked whether copying was even the binding constraint. It was not.

books is not in the 1,817-token vocabulary. It never occurs in training, so it is <unk> on the way in and unreachable on the way out. Every SQL statement naming it is a single atomic string token, so 'INSERT INTO books (...)' is also one out-of-vocabulary token. Only 3.9% of a held-out app's tokens are OOV, and they are exactly the load-bearing ones.

No mechanism defined over that vocabulary could have produced the right program, pointer or otherwise. "It substitutes a trained entity" was a symptom of an unreachable output slot, not evidence about copying. That is the fifth representation defect in this line of work, and all five were invisible in the loss curve.

So: make the noun expressible. Spell the entity at character level (<w> b o o k s </w>) and split literals at the entity boundary so the template fragments stay in vocabulary. Vocabulary went down, 1,817 to 1,623, and dev loss was unchanged at 0.027.

It still scored 0/8 on fidelity. But it failed in a new and much more informative way. At epoch 1 the model visibly attempted the character channel:

booooooooooookings

Starting b, o, o, then degenerating into a repetition loop. By epoch 9 it wrote bookings cleanly, with no character attempt at all. More training made it abandon the channel it had started to use.

That is not a capability failure. It is the objective working as specified. Every training app's entity was one of nine in-vocabulary nouns, so the entity was always predictable from the app's own internal consistency, and spelling a novel noun was never required to be right. Predicting the familiar sequence b-o-o-k-i-n-g-s is simply lower loss. The model was never asked to copy, so it did not learn to.

Making it expressible was necessary and not sufficient. It also cost real quality: in-domain compile fell from 6/6 to 1/6, because spelling an identifier as eight tokens gives eight chances to break it.

Randomise the nouns, and the substitution disappears

The fix the diagnosis implies is to change the data, not the model: draw a fresh entity noun for every app, so the mapping cannot be memorised and copying out of the spec is the only way to be right.

I regenerated that corpus while writing this. It takes 9 seconds and reproduces byte-for-byte against the tracked manifest:

600 training nouns, 10 held-out nouns, 128 feature combos
wrote 4240 apps ({'train': 4200, 'test_combo': 20, 'test_entity': 20}), 0 skipped

The ten held-out nouns include book, so the number stays comparable rather than becoming an easier case, and the script asserts no held-out noun leaks into training.

Trained on that corpus and decoded greedily from the spec alone, substitution is gone: 0/10 generations write a trained noun, against 8/8 before. The model now attempts the novel noun. And the new failure names the next blocker precisely:

books      -> bs
sprockets  -> sckets
lectures   -> letes
harvests   -> heves

It gets the first character right and the ending right and loses the middle. Spelling a symbol character by character needs state for how far through it you are, and nothing in the architecture carries that. The blocker was never vocabulary, and after the randomised corpus it is no longer distribution; it is positional.

The zero-parameter mechanism that just does it

Meanwhile a different line of work in the same repo had a document cache: a count-based model over the text currently being written, with no parameters and no training.

Copying a noun out of the prompt is exactly what such a cache is for, and nobody had put the two together. The measurement walks each held-out program left to right and, at every position whose target is part of the entity noun, asks each system to predict it. I re-ran it (cache_copy.py, 63 seconds):

WHOLE-NOUN accuracy where the program FIRST names it:
  corpus   0/20 apps
  cache    20/20 apps
  mix      17/20 apps

system       spec (floor)    in the code      first in code
corpus             0.243          0.254             0.100
cache              0.662          0.970             1.000
mix                0.719          0.978             1.000

corpus-model errors that were a TRAINED noun's character: 3960

A cache with zero parameters gets the noun right 20 times out of 20. A count model trained on all 4,200 apps gets it right 0 times out of 20, and its errors are a trained noun's characters 3,960 times. The corpus arm is the control that makes this mean something: it fails the same way the neural model did, and it is not a data-scale problem, because 4,200 apps is the whole corpus and the score is zero.

The information was always extractable from the request. What was missing was a mechanism that emits what it just read.

A sweep over the mixing weight is worth putting next to that, because it is the kind of result that flips on the metric you pick. Per-position accuracy peaks at a blend (0.978 at β=0.70) while whole-noun correctness is monotone and peaks at pure cache (20/20 at β=1.00). Blend when the unit of correctness is a token; do not blend when it is a multi-token symbol that is wrong if any character is.

The honest catch, and it is a big one: all of that is teacher-forced. Under free generation the cache's best weight collapses from about 1.0 to about 0.3, because it begins reading the model's own output rather than true text. Pushed further, it amplifies whatever was just emitted: the gated variant runs degeneracy from 0.34 to 0.92 and takes "no table at all" from 1 of 10 to 10 of 10. The component that wins on prediction is the one whose input degrades under generation.

What did survive was constraining decoding to spell one of the spans the request actually contains: a trie over the user's own words, again with zero parameters. With the cache at low weight, that takes table hit from 0/10 to 10/10 and purity from 0.00 to 1.00.

The obvious objection is that masking to "spell one of the two nouns in the request" leaves so little freedom that a model which learned nothing would look correct. That was tested by injecting a second, equally novel, equally legal noun into each request: 9 of 10 chose the requested noun, 1 chose the distractor, against a chance rate of 0.5 (binomial p = 0.011). The constraint supplies legal spellings; the model supplies which one.

It fixes the entity, not the program. Whole-app validity is untouched, and a line like PORT = int(os.environ.get('PORT', deleting them in the same outputs is a fair reminder of that.

What generalises

Three things, and the first one is the one I would actually carry anywhere.

A 30-line probe beat five experiments of intuition, twice. "Needs more capacity" and "needs more training" were wrong at every stage, and the loss curve never once indicated the real defect. Both diagnoses that mattered came from cheap instruments: a teacher-forced per-position probe that localised two pipeline bugs to single token positions, and a vocabulary probe that showed the target string was unreachable before anyone built a copy mechanism for it. Five separate representation defects, all invisible in the loss.

A model will not learn a capability its training data never requires. The character channel was available, expressible, and visibly attempted at epoch 1, and training removed it, because being right never depended on it. That is not the model failing to generalise; that is me failing to specify. The fix was a corpus change, and it worked on the first try.

Check whether the failing thing is even representable before designing the fix. I had written "this is a copying problem, use pointer attention" into my notes as a conclusion. It was a plausible, standard, entirely wrong remedy for a slot that was <unk> on the way in and unreachable on the way out. The probe cost 30 lines and saved building the wrong mechanism.

The arm still does not do what I built it to do. Inside domains it has seen it writes correct software at 83%, which is a genuinely stronger result than I expected from 11.9M parameters, and it is compositional generalisation over 2⁷ feature subsets with no combination-specific code. It just is not the labour-amortising property that would have justified preferring it, and the thing that finally did the copying had no parameters at all.

DEV Community: Seth Wheeler

What a Language Needs Before It Can Compile Itself

The one that cost the most

The other five

What did not come up

The list was ordered, and two of them got built

The classification was wrong once, and the next stage proved it

What generalises

No Add-on Can Block Ads in This Browser

The measurement

I stated the opposite twice

What I have not ruled out

The thing that works is the thing I deleted

A related failure, for contrast

What generalises

Measuring the Wrong Process for Eight Months

The number that gave it away

Three mistakes, stacked

Why it survived eight months

The corrected numbers

The light pages inverted

What survives

What I changed, and what I did not

The part that generalises

When a SQL Engine Records Column Types but Never Reads Them

The query that finds nothing

Why the first query works

The part that has not bitten yet

What I would fix, and what that costs

What generalises

Choosing Video Frames by Content Instead of by a Clock

Where it does not win

Why I trust the numbers

On how it was built

Choosing an Unreleased API Over the One Already There

How the routes are found

The line that needed an API that did not exist

Four days later I did not need it

Why I put it back

What generalises

Reimplementing Enough of Kubernetes to Fool kubectl

kubectl's first move is discovery, and it will not proceed without it

kubectl get asks the server to do the formatting

The protobuf wire format is not protobuf

The failure mode is a wrong number, not an error

What I couldn't make work

The part worth generalizing

Checking a Cost Model Against a Stranger's Config File

The header states the fact outright

What the correction does

Two errors, pointing opposite ways

What I'd take from this

Predicting the Speed of a 276B Model Streamed From an SSD

Act 1: a prediction is worthless unless it can be wrong

Act 2: the benchmark was measuring RAM

Act 3: running it, and missing by 23×

The part that was worth the whole exercise

What a Hibernated Browser Tab Actually Costs

The ladder, and what each rung actually costs

Why COLD can't get below a whole process

A null result that read like a finding

Where the advantage went

The rule the simulation never surfaced

The instrumentation became the largest cost in the system

What I actually learned

A Better FP4 Gradient Quantizer That Training Couldn't Notice

The win, and where it comes from

The wall, then a landmine

Two GPUs that saw nothing

The number I should have measured first

What generalizes

Why a Small Transformer Can't Copy a Word It Hasn't Seen

The claim I was defending

0% was the plumbing, four times over

The test, and the retraction

The diagnosis I got wrong

Randomise the nouns, and the substitution disappears

The zero-parameter mechanism that just does it

What generalises

`kubectl get` asks the server to do the formatting