Every topup run broke batteries somewhere. The fixes broke different things. The pattern wasn't sloppy training. It was three architectural problems stacked on top of each other.
After Part 19's measurement-discipline reset, I went back to vocabulary work. Origin needed more concepts in its encoder. There were thirty-two candidates I'd been wanting to add for weeks. Words like holy, prime, cash, mobile, aircraft. The kind of vocabulary a conversational system needs to handle normal English.
I'd built a tool called train_encoder_topup.py that was supposed to add new concepts without retraining from scratch. Five quality gates. Preflight checks. Atomic promote with backup. Resume on failure. The whole discipline scaffolding from a week ago, applied to vocabulary growth.
The smoke run worked. Two concepts (holy and prime) lifted to high firing, all five batteries held within tolerance, atomic promote, hippocampus rekey, ten-out-of-ten on the conversational probe. Two and a half minutes wall-clock. Clean.
Then I ran the full version. Fourteen concepts at the same settings.
It failed. Two of the fourteen never lifted past the firing threshold. The working surface battery dropped one probe. The full set didn't clear the gates. Production stayed at two new concepts instead of fourteen, the tool aborted cleanly without overwriting anything, and I sat looking at the log trying to understand why the smoke had worked and the scale-up hadn't.
The first guess was hyperparameter sensitivity. Smoke ran with conservative learning rate and modest oversample. Maybe fourteen concepts at the same settings was just too many for the gradient budget. I tried raising the learning rate. That lifted the cold concepts but broke the batteries in a different place. I tried more anchor data to dilute the topup pressure. The cold concepts went cold again. I tried trimming the question-form templates that seemed to be teaching the encoder bad habits. That fixed two compositional failures and broke two identity probes.
Each mitigation moved the breakage around. Nothing eliminated it.
I'd built a sandbox specifically to test three different augmentation approaches in parallel. Text augmentation. Feature-space mixup. Graph-driven sentence templating. All three worked at lifting cold concepts. All three broke batteries. Different things broke under different approaches, which meant the breakage wasn't tied to any one technique. The system had a structural property that the topup process was provoking.
I stopped tuning and started auditing.
The first audit went after the headline number that had been driving the vocabulary work. We'd been saying for weeks that the encoder was "sixty percent dead." Five thousand four hundred and ninety-three concepts of eight thousand eight hundred and eighty-two were silent on the per-concept firing report. Three out of every five output slots, dark.
I wrote a tool that joined the encoder health report against the brain layer's reasoning data to see which dead concepts the brain layer actually needed for compositional reasoning. The brain layer's training pairs gave a direct measurement, how many times did this concept appear as a reasoning output? If a concept was never an output, the brain layer never needed it. If it was an output thirty times, the brain layer needed it a lot.
Two concepts had high pressure. perceptual appeared as an output five hundred and twenty-five times. evaporate appeared thirty-five times. Thirty-four concepts had medium pressure. Five thousand four hundred and thirty concepts had zero pressure. The brain layer never needed them. The composer never used them. The encoder had never trained on them properly because they weren't real concepts to begin with. They were nominal slots in V2C that had survived the vocabulary growth process but didn't correspond to anything the system was supposed to know.
I marked them as unrescuable. UNRESCUABLE_CONCEPTS, a new frozenset alongside the existing lists of deprecated and intent-only concepts. The effective vocabulary dropped from eight thousand eight hundred and eighty-two to three thousand three hundred and eleven. The "sixty percent dead" number had been a measurement artifact of counting empty slots that nothing was supposed to occupy.
That reframed the problem. The encoder wasn't dying. It had been carrying five thousand four hundred and thirty dead slots that everything still flowed through.
That's when the actual mechanism showed up. The encoder's output head is structured by domain. Each domain has its own classifier head producing logits across the concepts in that domain. The domain called other, the catch-all bucket for things that didn't fit anywhere else, had five thousand four hundred and thirty-seven slots in it, of which only two hundred and seventy-one were real concepts. Both perceptual and evaporate lived in other.
When the topup tool ran a gradient update on either of those concepts, the gradient flowed through every output row in the other domain head. All five thousand four hundred and thirty-seven of them. The two real concepts I was trying to lift had to share gradient with five thousand four hundred and thirty dead slots that contributed pure noise.
I called it gradient pollution because I was tired and that was what it looked like. The training signal for the concepts I cared about was getting drowned in noise from concepts I didn't.
The fix had three pieces. First, mask the dead slots. The loss function gets zeroed for any output row in the unrescuable set, so dead-slot gradients can't flow at all. Second, freeze the trunk. The trunk has signal for nearly every concept already. The heads are what need to learn. Freezing the trunk eliminates trunk drift by construction and makes battery regressions only possible through head output behavior. Third, isolate by domain. The mask zeros loss for any output slot in a domain that doesn't contain at least one of the concepts being lifted, so cross-domain head cross-talk gets killed too.
I stacked the three mitigations and re-ran the perceptual + evaporate topup.
Two firings above threshold. Identity battery forty-eight out of fifty-one. Working surface twenty-six out of twenty-six. Compositional reasoning twenty-nine out of twenty-nine.
The compositional battery had been at twenty-six out of twenty-nine before the topup. Lifting two concepts cleanly with all the gradient mitigations in place had fixed three compositional probes that had been failing for weeks. The brain layer needed those concepts to compose correctly, and once it had them, it composed correctly. The other batteries didn't budge. First all-gates-pass topup of the project.
I promoted it. Then I went looking for the next concept to lift, and the next architectural problem surfaced almost immediately.
I added trader to a different domain head. The topup gates passed. But on the way through the post-promote audit, I noticed evaporate's joint output logit had dropped sixty-eight percent on physics queries. evaporate wasn't in the same domain as trader. The per-domain row for evaporate hadn't moved. But the joint logit (the product of the per-domain classifier and a top-level domain selector) had collapsed because training trader had shifted the top-level domain selector's distribution.
This was a different architectural side-channel. The encoder's joint output combined per-domain concept logits with a log-softmax over which domain the input belonged to. Training a concept in domain eleven told the model "this input is in domain eleven," which proportionally reduced the model's prior on every other domain, which dragged down every concept in those domains by a constant offset. Hidden multiplicative coupling between concepts that had nothing to do with each other.
I added a fourth mitigation, surgical freeze on the domain classifier during topup. The classifier learns nothing during a concept lift, so it can't drag other domains down. I ran the same gates on five more concepts (chemical, thinker, ikon, rot, and a re-lift of trader). All five passed.
Then I tried the recipe on a sixth concept in a denser domain head, and hit the third structural limit. The within-domain heads use a shared bottleneck projection. When you lift one concept in a domain with thirty other live concepts, the shared projection learns the new concept's features at the expense of the existing concepts' features. The mask-and-freeze recipe couldn't isolate within a single head; the head had only one set of parameters and they all moved together.
The recipe had a ceiling.
Three architectural layers diagnosed in two days. Each fix exposed the next problem. We were tunneling down through layers of the encoder's design, and each layer was more fundamental than the last. The first one was a counting error masquerading as a capacity problem. The second one was a gradient routing problem inside a single head. The third one was a head-architecture problem that no amount of routing fix could touch. The wall was the architecture itself.
Nine concepts revived end-to-end. Compositional battery perfect. Identity off by two from baseline. Working surface unchanged. Real progress, real wins, real numbers. And the realization, sitting underneath the wins, that this approach to vocabulary growth was going to run out before Origin's vocabulary was anywhere close to where conversational use required it.
The architecture I was patching had become the thing that needed redesigning.
One guy. One GPU. One $1,800 computer in Arizona. Still building.
Origin is developed at Fallen Angel Systems with the Genesis framework — NVIDIA Inception member. (USPTO Application #64/016,973, #64/017,567). FAS Guardian defends production AI systems from prompt injection in under 3ms. FAS Judgement is the open-source attack console that finds the gaps. Defense. Offense. Creation.
fallenangelsystems.com | Judgement on GitHub | Guardian on GitHub
Questions or consulting inquiries: josh@fallenangelsystems.com
Top comments (0)