The data plan didn't move the encoder. The architecture sandbox did.
Part 10 ended with the abort condition triggering: top1 of 1.3% on held-out probes meant the architecture, not the data, was the bottleneck. The plan said "design contrastive next." We built a sandbox first.
The Sandbox
150 random concepts spread across six domains. The same training data filtered to that slice. The same held-out probe battery. Five concept_head variants tested side-by-side:
- baseline_flat_bce: current architecture (flat MLP + per-slot binary cross-entropy)
- contrastive: same MLP, but cross-entropy over the full concept space (the target must dominate every other concept, not just exceed a threshold)
- tree_hierarchical: predict domain first, then concept within domain
- domain_routed: soft-gate trunk features through per-domain sub-heads
- contrastive_tree: hybrid of tree structure plus contrastive global cross-entropy
All five shared the warm-started trunk. All had comparable parameter counts. The only difference was head topology and loss. The first round trained with the trunk frozen, head only, so any difference traced to the architectural choice, not optimization noise.
The Frozen Ceiling
The best variant under frozen trunk hit 10% top1. The baseline hit 4.5%. Real differentiation, but no variant came close to a useful threshold.
The reading: the trunk's 256-dimensional feature output was the ceiling. The trunk had been warm-started from v1 then trained on the new data, but it had never been shaped to discriminate 3687 concepts. No head topology could extract signal that wasn't there. Every variant was trying to read meaning from a representation that hadn't learned to encode it.
Before scaling further, we set a pre-defined pass criterion: top1 at or above 30% AND cross_fire under 30 at 500 concepts means "this is the architecture to scale." Hold the gate. Don't let a result you wanted to see become the result you accept.
Unfreezing the Trunk
One change: let the trunk co-adapt to the head's loss at a lower learning rate (1e-4 trunk vs 3e-4 head). Same data. Same head. Same epochs.
contrastive_tree at 30 epochs on 500 concepts: 28.4% top1 with cross_fire of 48.27. Just below criterion on both. The pattern said "more epochs needed." Larger concept counts take longer to converge. At 60 epochs: 34.8% top1, 26.26 cross_fire. Both criteria met, no other tuning needed.
Architecture locked: contrastive_tree + unfrozen trunk + 60 epochs.
Production Retrain
Same architecture, scaled to all 3687 concepts. 145,000 training pairs (the natural-positive corpus from Part 9 plus everything else in Phase A). 65 minutes on the 4070.
Phase 5 probe battery on the new encoder, same 50 random concepts as the baseline. The number that matters most: under the old encoder, the right answer was outvoted 14-to-1 by distractors. Under the new architecture, the target concept dominates by 22x. That's not gradual improvement. That's a different machine.
| Metric | Baseline | New Architecture |
|---|---|---|
| top1 | 1.3% | 31.3% |
| top3 | 4.0% | 50.0% |
| target activation | 0.012 | 0.249 |
| target / 2nd-best ratio | 0.07 | 22.67 |
Top1 went from 1.3% to 31.3%. Top3 from 4% to 50%. None of the absolute Phase 5 success gates are met yet (the plan's full-victory marks were 70% top1, 0.7 target activation, cross_fire under 2). But every metric moved dramatically in the right direction, and the lever was exactly where the plan said it would be if the data hypothesis failed.
What Comes Next
The new encoder is staged on disk but not yet live. Swapping it in turns out to be more than a file rename. The dispatcher that turns concept activations into responses was built around the old encoder's sigmoid output range. The new encoder uses softmax. Without changes, every query would route to IDK. We'll cover the fix in Part 12.
Origin is developed at Fallen Angel Systems with the Genesis framework — NVIDIA Inception member. (USPTO Application #64/016,973, #64/017,567). FAS Guardian defends production AI systems from prompt injection in under 3ms. FAS Judgement is the open-source attack console that finds the gaps. Defense. Offense. Creation.
fallenangelsystems.com | Judgement on GitHub | Guardian on GitHub
Questions or consulting inquiries: josh@fallenangelsystems.com
Top comments (0)