I spent a week trying to make the brain layer do more. Every time I fixed one thing, the next wall was already there. The architecture was the problem.
Part 20 ended with nine concepts revived and a structural realization sitting underneath the wins. The vocabulary growth approach was running out of room before Origin's vocabulary had reached anywhere close to conversational. The architecture I was patching had become the thing that needed redesigning. I went back to the brain layer to see if it would scale up while I figured out what to do about the encoder.
The first thing I noticed was that training was painfully slow. A full brain layer retrain on the rebalanced corpus was projected at ninety hours. For a thirty-nine-thousand-pair dataset that should have taken minutes. Something was wrong with the math, and the way to find out was to profile it.
I built a profiler that broke the per-batch time down by sub-operation. Forward pass, backward pass, each of the routed circuits, the thalamus router, the two-stage reasoner. The result was unambiguous and embarrassing. The two-stage reasoner, which I'd spent significant time restoring from v1 as the "intellectual core" of compositional reasoning, was consuming 99.8% of training wall-clock. Every other operation in the brain layer combined consumed about four milliseconds. The two-stage reasoner ate two thousand seven hundred and six.
I went looking for why. The two-stage reasoner runs property inference (Stage 1) followed by rule application (Stage 2). Stage 1 was supposed to identify which properties of the input concepts mattered. Stage 2 was supposed to combine those properties into outcomes. Stage 1's output was the input to Stage 2's attention. So Stage 2's compute scaled with how many properties Stage 1 surfaced. And Stage 1, on a real input, was surfacing eight thousand two hundred and twenty-five concepts as "inferred properties" out of eight thousand eight hundred and eighty-two. Ninety-three percent of the vocabulary, declared a property of every input, fed to a Stage 2 attention layer that had to process the noise.
The property circuit was degenerate. It had been trained against data that didn't teach property structure cleanly, and the output had collapsed to "everything is a property." Stage 2 wasn't reasoning about compositions. It was performing attention over near-uniform noise.
I ran an ablation. Replaced the two-stage forward with a negative-infinity tensor that would lose every argmax competition, effectively removing it from the prediction. The brain layer's held-out eval went from seventeen out of twenty-six to twenty out of twenty-six. The two-stage reasoner wasn't dead weight. It had been actively poisoning predictions.
I disabled it. Training dropped from two thousand seven hundred and six milliseconds per batch to ten and a half. The ninety-hour projected retrain became seventy-eight minutes. The held-out eval went up. The "intellectual core" I'd built was making everything worse.
That should have felt like a win. It mostly felt like a question. What else in this architecture was I keeping around because I'd designed it instead of because it worked.
I built a new battery to find out. The identity and working surface and compositional batteries were saturated. They tested capabilities Origin already had. I needed something that tested the ceiling, the questions Origin couldn't answer yet, the shapes the system would fail on. Twenty-five probes across five buckets, contrastive properties ("how is X different from Y"), taxonomic intersection ("what do X and Y have in common"), role requirements ("what does a teacher need"), boundary classification ("is fire alive"), and out-of-distribution honest refusals ("is the number seven happy today"). Each one isolated a specific brain capability so the failure mode would be diagnosable. I called it the frontier battery.
First run: nine out of twenty-five correct. Eight out of twenty-five confidently wrong.
The confident-wrong cases sorted into recognizable failure modes. The brain layer would fire action on "is the sun alive" and the dispatcher would compose "alive causes action." It would fire consent on "what do car and bicycle have in common" and produce "car causes consent." The definitional fallback path would parse "what is the population of mars" as a question about "mar" (the prefix of "march") and emit a definition for the month. "Is the number seven happy today" got a generic number definition because the dispatcher had no concept of "this question is structurally nonsensical."
I shipped five dispatch guards. Regex patterns that recognized population-of and number-X-is-Y as nonsense shapes and routed them to IDK. Text-shape gates that caught "in common" and "different from" before the brain layer could compose them. A specific guard for "is X alive" because the brain layer didn't have boundary classification training. The frontier battery confident-wrong rate dropped from eight to three.
Then someone independent ran a hundred conversational probes against production and surfaced eight more misroutes and misfires. I shipped five more dispatch guards. The misroute count went to zero. The probe count went to a hundred and forty-three. I caught myself writing my third regex of the day and remembered the rule we'd codified weeks earlier: hardcoded text patterns that tell Origin what to say bypass understanding. The fixes I was shipping were against my own discipline. Some of them were the right kind of pattern (text used to interpret what Origin already learned). Some of them weren't. The line between the two was getting thinner.
I needed an approach that didn't keep growing the regex pile. I tried a curriculum.
The idea was clean. Stage 0: train concept embeddings contrastively so similar concepts cluster. Stage 1: train property assignments so the model learns that fire is hot, ice is cold, dog has fur. Stage 2: train compositional rules so the model learns that fire causes burns and ice in heat causes melting. Three stages, each building on the previous, with explicit per-stage gates.
Stage 0 worked. Trained-category accuracy hit 99.8 percent across three random seeds. Embeddings were healthy. The contrastive mechanism converged cleanly in five seconds of GPU time. Held-out generalization was partial as expected, 65 percent on average, because Stage 0 only had supervision saying "these concepts go together" without any cross-cutting property structure. The plan was that Stage 1 would add the property structure and Stage 2 would add the compositional rules and held-out generalization would climb.
I built Stage 1. Hand-curated fifty-three property pairs that would have been the seed for cross-category transfer. Fire goes to hot. Ice goes to cold. Rock goes to heavy. Apple goes to red. The kind of grounded property assignments that should produce transfer to nearby concepts: train fire and sun as hot, and the model should infer that lava is also hot. That's the whole point of property transfer. Without it, the curriculum has to enumerate every concept-property pair individually, and we're back to scaling the vocabulary by hand again.
I ran Stage 1 with a refinement-from-v5 setup. Loaded the existing brain layer, froze most of the parameters, unfroze the prediction head and let the embeddings shift. The training-pair recovery hit ninety-two and a half percent. Forty-nine of the fifty-three pairs landed correctly. Brain eval held steady at twenty-one out of twenty-six. The recipe worked.
Then I ran the property transfer eval. Twenty-five held-out probes, each one testing whether a property the model had been taught on one concept transferred to a related concept. I trained rock as heavy. The test asked whether stone was heavy. I trained ice and snow as cold. The test asked whether arctic was cold. I trained sun as bright. The test asked whether a lamp was bright.
Top-1 accuracy: zero out of twenty-five. Top-5 accuracy: two out of twenty-five. Top-10 accuracy: two out of twenty-five.
Eight percent transfer on shapes the model should have generalized. Stone returned falls and sinks and lift, with heavy absent from the top five. Arctic returned dissolve and cloud and freeze, with cold absent. Lava returned rot and fry and evaporate. The model had memorized the specific input-vector-to-output-index mappings of the fifty-three training pairs. It had not learned the underlying property. Stone is embedding-close to rock, which the model had been told was heavy, and the dense prediction head simply did not move that information through.
The diagnosis took an hour. The brain layer combined an averaging pool of input concept embeddings with a dense prediction head. The averaging pool discarded positional structure. The dense head memorized specific input-vector targets. Property transfer required something the architecture didn't have: a way to fire a learned property whenever an input was semantically close to a trained example. The averaging-and-dense combination doesn't support that. It supports "memorize what you were shown."
This was the third wall in a week. The two-stage reasoner had been actively poisoning predictions. The dispatcher needed an unbounded supply of guards to handle real conversational text. The brain layer's property circuit was structurally incapable of property transfer. Each layer of the architecture I'd built and patched and restored was hitting a ceiling, and the ceilings weren't tunable. They were the architecture.
I'd spent two months bringing the brain layers back from missing-in-action to production-integrated. The brain layers were now production-integrated and architecturally limited. I had three options written in the session notes for what to do next. Change the architecture. Build something separate. Accept the ceiling.
I knew which one it was going to be, I didn't want to think about it. I'd already seen the RelationalCircuit pilot work earlier in the week. Maybe the path forward wasn't fixing the brain layer at all.
One guy. One GPU. One $1,800 computer in Arizona. Still building.
Origin is developed at Fallen Angel Systems with the Genesis framework — NVIDIA Inception member. (USPTO Application #64/016,973, #64/017,567). FAS Guardian defends production AI systems from prompt injection in under 3ms. FAS Judgement is the open-source attack console that finds the gaps. Defense. Offense. Creation.
fallenangelsystems.com | Judgement on GitHub | Guardian on GitHub
Questions or consulting inquiries: josh@fallenangelsystems.com
Top comments (0)