I Was Training the Model to Do the Wrong Job

I spent hours training a small local model before realizing I was asking it to do the wrong job. The training itself was not the problem. The model was learning. The evaluations were useful. I had a compact interface where the tuned model could produce the expected result across the test set, and eventually the base Qwen model could do the same thing without the tuning. That should have felt like success. Instead, it made me question why the model was being asked to solve so much of the problem in the first place.

The task contained two very different kinds of work that I had allowed to collapse together. Some of it was exact. Eterna already knew what evidence existed, which records were eligible, what the authoritative values were, how the output had to be reconstructed and which states were permitted. Those parts were deterministic whether I happened to be using a model or not. The remaining part was genuinely semantic. Given a bounded set of eligible evidence, which piece of meaning best matched what the user was asking for? I had been training the model to do both.

That is an easy mistake to make because modern models are capable of doing both. Give them structured data, formatting rules, selection criteria and an output schema, and they will often produce something that looks exactly right. The fact that a model can do a deterministic job does not mean it should own that job.

Once I separated the responsibilities, the architecture became much clearer. Eterna could decide what was eligible and supply exact authoritative data. The model could handle the narrow semantic mapping that actually benefited from learned judgment. Then Eterna could reconstruct and validate the final result deterministically. The model stopped being responsible for remembering the rules of the whole system. That changed the evaluation too.

On the compact test, the tuned model reached 24 out of 24. Then the base Qwen model also reached 24 out of 24. I had spent time training behaviour that the base model could already perform once the problem was represented properly. That was useful evidence, but it was not the end of the story. When I pushed into unresolved evidence, both models showed the same deeper weakness. They could over-select when the system had not established strongly enough whether a semantic decision was allowed to exist at all. That was the point where the lesson moved above the model.

The model was not necessarily bad at choosing between candidates. Eterna had failed to make the authority boundary explicit enough before asking it to choose. A model should not be deciding whether it has the right to make a decision when the surrounding system can determine that first. That sounds like a small architectural distinction. For me it connected several months of Eterna work.

I have spent a lot of this year learning that durable state matters more than conversation memory. One fact should have one owner. Software should do software-shaped work. Verification should not depend on the same mechanism that created the answer. Providers should be replaceable where practical. The system should carry authority rather than expecting whichever model happens to be active to infer it perfectly every time. This local-model experiment landed on the same idea from another direction. The best improvement was not teaching the model more rules. It was moving rules out of the model.

By Eterna Desktop v1.55, I could turn Qwen off for 21 prompt behaviours and watch what remained. Nineteen completed without it. Two genuinely semantic prompts failed closed because the semantic component they needed was unavailable. That result interested me much more than forcing all 21 through a local model. It meant the architecture was beginning to tell the truth about where intelligence was actually required.

For a long time, I thought of AI systems mostly in terms of which model was doing the work. A frontier model might be more capable. A local model might be cheaper, private or specialized. Fine-tuning might make the local model better at a particular task. Those are real design questions, but they are now downstream of another one for me. How much of a solved problem needs to remain an AI problem? If Eterna learns that a certain class of work can be represented as exact state, rules, schemas, tools and tests, keeping that work inside repeated probabilistic reasoning is not automatically sophistication. Sometimes it is just refusing to cash out what the system has already learned.

I started reading more about neuro-symbolic AI because I finally had a name for part of what I was converging toward. The phrase covers a broad family of approaches that combine learned or neural methods with explicit symbolic structures such as rules, logic, programs, knowledge or other inspectable representations. I am not claiming Eterna invented that idea. It clearly did not. What interested me was recognizing the pattern after arriving at it through practical failure.

The useful parts of Eterna increasingly live at different layers. Frontier intelligence is valuable when the problem is new, ambiguous, creative, poorly represented or changing quickly. Deterministic software is better when the system already knows exactly what should happen. Structured knowledge and authoritative state reduce how much the model has to infer. A smaller local model may make sense when a recurring semantic residue remains stable enough to justify it. And sometimes there is no reason to train a local model at all.

That last part matters because I had spent a lot of time assuming the local model was the destination. The experiment did not prove that. It showed me that better representation could remove model work entirely, and that the remaining learned component might be small enough for a base model, a specialist model, a frontier model or some future mechanism depending on the actual requirement. The architecture became more useful when I stopped trying to make every layer justify its own existence.

There is still a lot I do not know about where this goes. I do not have evidence that every recurring Eterna task can be compressed this way. Some tasks may remain frontier-heavy because the world keeps changing underneath them. Some semantic problems may not become stable enough to compile into anything simpler. The maintenance cost of a specialized local capability can also exceed the tokens or attention it saves. But I have a much better question now. When frontier intelligence solves something difficult, what durable capability should remain afterward?

Maybe the answer is software. Maybe it is a schema, a test, an authority rule, a tool, an index, a small model, or simply a better representation of the problem. Maybe nothing reusable can be extracted and the frontier model should handle it again next time. I no longer want to assume the answer is another model call.

That is a fairly drastic change from where I was even a few weeks ago. I thought I was training a model to become a better part of Eterna. The more important result was learning which parts of Eterna should never have depended on that model in the first place.