Fine-Tuning Your Facts Is Hardcoding Them Into a Binary Nobody Can Read

Every engineer already knows not to hardcode a price into a compiled binary. You put it in config or a database, because it will change, because someone needs to see where it came from, and because you want to fix it without a release.

Then an AI feature shows up, someone asks "can we train it on our data?", and that instinct quietly disappears. Fine-tuning a model on your facts is exactly that hardcoded price, except the binary is a pile of weights nobody can read, diff or grep.

I build AI features for clients at Null Studio, and "train it on our data" is one of the first things we hear. The honest answer is usually that you do not need to, and if you did, it would not do what you expect. The way I think about it now is as a storage decision: there are three places knowledge or behaviour can live, and each has very different properties.

Three places to put things

The prompt is config. It is the text the model gets before it answers: role, rules, tone, output format, worked examples. Change it and the behaviour changes on the next request. It costs nothing to deploy and it is trivially reversible. A surprising share of "the AI is not doing what we want" bugs are fixed here with clearer instructions, a few examples and a defined output structure.

Retrieval is the database. RAG means searching your own content for the passages relevant to a question and handing them to the model alongside it. The model writes the answer; your documents supply the facts. The important property is that the knowledge lives outside the model. When an opening hour changes, you edit a document and the next answer reflects it. You can show the source. You can enforce permissions at query time, the same way you would on any other read path.

Fine-tuning is compiled code. You take a base model and train it further on example inputs and the outputs you want. The result is a modified model that leans toward those patterns without being told. That is genuinely useful for behaviour: a strict output format, a classifier with your own categories, a house style that is easy to show and hard to describe, or getting a smaller, cheaper model to match a bigger one on one narrow task.

What it is bad at is facts. A fine-tuned model might reproduce something it saw in training, or it might produce something that sounds exactly like it and is wrong, and it cannot tell you which.

Why facts do not belong in weights

Put it in engineering terms and the problems are obvious:

• Every fact change is a release. New pricing means another training run, another test cycle, another deployment. With retrieval it is a row update.
• There is no stack trace. A RAG answer can point at the paragraph it used. A fine-tuned answer came from the weights, so nobody can check where it came from.
• There is no access control inside a model. Once restricted information is trained in, anyone who can call that model can potentially draw it out. "Do not reveal X" in a system prompt is not a security boundary. This gets worse once the model can take actions, which is why I give agents write access one verb at a time.
• Bugs are hard to patch. If a RAG system answers wrong, you can usually find the bad or missing document. If a tuned model answers wrong, the fix is more data and another training run, with no guarantee the mistake is gone.

So the rule I use is short: facts go in retrieval, behaviour goes in the prompt, and fine-tuning is for behaviour the prompt cannot hold.

What my own voice agents actually do

I went and checked this against the client voice agents I have exported from Retell, because it is easy to preach a rule you do not follow.

All nine of them have knowledge_base_ids: []. No retrieval at all. Every one has a ## Context section in the prompt instead. The dental practice agent, for example, lists its services, business hours by day, office address and phone number right there in the prompt text, with the practice name pulled in through a COMPANY_NAME dynamic variable.

Is that a contradiction? I do not think so, and the reason is the useful part. For a small business, the entire fact sheet is a few hundred words. It fits comfortably in the prompt, it is plain text anyone on the team can edit, and a change ships the moment the agent is saved. That is still config, not compiled code. What I have never done is fine-tune a model on a clinic's hours.

The prompt stops being the right home for facts when the facts stop fitting: a catalogue, a policy manual, program rules that differ by location, or anything where different callers are allowed to see different things. That is where retrieval earns its complexity. The multilingual voice and SMS intake we built behind Fortell AI, which serves Community Action Agencies in more than 100 languages, has to answer against real program rules, and those belong in content the team can update, not in a prompt that keeps growing. I wrote about what goes wrong when one prompt carries everything separately.

When fine-tuning is the right call

It is a real tool. I would consider it when:

• You have a narrow, repetitive task, like classifying messages into your own categories or extracting fields in a fixed shape, and prompting alone is not consistent enough at your volume.
• You want a smaller, faster or cheaper model to match a larger one on that one task. That matters for cost and, on a phone call, for latency.
• You have a style that is hard to specify but easy to demonstrate.
• You have hundreds or more high-quality input and output pairs, reviewed by someone who knows the domain.

That last one kills more fine-tuning projects than anything else. Train on a small, messy set and the model faithfully learns the mess.

The order I actually work in

1. Build an evaluation set first. Real inputs, defined pass criteria. Without it you cannot tell whether any of the following steps helped. This is the part most teams skip, and I wrote up how I test outputs you cannot unit test.
2. Fix the prompt. Examples, output structure, rules. Measure.
3. Add retrieval for anything that depends on your own information. Measure again. Most quality problems at this stage turn out to be content problems: duplicate, outdated or missing documents. Your RAG demo probably works because someone picked the documents.
4. Try a different model. Sometimes that closes the gap with no training at all.
5. Only then consider fine-tuning, against a specific measured gap, with a target score and a comparison to the untuned setup.

These stack. A fine-tuned model still gets a prompt, and where facts matter it still gets retrieved context. Fine-tuning does not replace the other two; it sits on top of them.

The costs nobody puts in the estimate

The training run is usually the cheapest line item. The rest:

• Data preparation. Collecting, cleaning and reviewing examples is real work, and most of it needs a domain expert, not just an engineer.
• Base model churn. Providers update and retire models. A tune on a retired model has to be rebuilt on its successor and retested. Think of it as a dependency you cannot pin forever.
• Lock-in. Some tunes only run on the provider that trained them. Ask where the tuned model lives, who can access it, and whether you can take it with you.
• Data handling. Your examples go to whoever does the training. Check what they do with them, especially with personal or regulated data.
• Ongoing evals. The tuned model still needs the eval set run on every change, and the set needs maintaining.

None of these are reasons to avoid fine-tuning. They are reasons not to reach for it by default.

Product features, same question

It is not just chatbots. For LectureNotes AI, which records lectures and produces summaries and outlines, what matters is faithfulness to what was actually said and a consistent outline structure. Faithfulness comes from grounding the output in the transcript; structure is a prompting job first. For Lifemaxxing AI, which builds personalised plans, personalisation comes from the user's own data passed in at request time, not from training a model per user.

In both, the useful question is not "should we train a model?" It is "where should this live so it stays correct and can change?" That is the same question you already ask about every other piece of state in your system.

If someone proposes fine-tuning first, ask them what it fixes that retrieval does not. If they cannot name a measured gap, you have your answer.

I am curious where others draw this line in practice. Have you shipped a fine-tune that actually beat a good prompt plus retrieval on your eval set, and what was the task?

Originally published on the Null Studio blog.