Inference is the process by which a language model generates output.
Context goes in — prompt, system instructions, conversation history — and the model produces a probability distribution of what the most likely output is.
The Agent isn't understanding my request. It isn't considering, planning or weighing. The LLM is playing a guessing-game always asking "which answer is most likely to be correct?"
Where do the probabilities come from?
The probabilities themselves come from training. During training, the model's output is compared against a desired output, and a training signal adjusts the model's parameters — more of this, less of that.
Fine-tuning applies the same process afterward with narrower data, shifting the probabilities again toward specific behaviours, such as instruction-following or a thorough assistant style.
Why Does This Matter?
1 - An agent is almost always going to be wrong.
Outputs are probabilistic and non-deterministic. This means that the same question asked twice, can get two different responses. In my dev projects, what this means is that the agent always misses something. It has a lot of predictions to make, so the likelihood of it missing something is high.
2 - Context Matters
More context increases the difficulty in predictions. I now keep file sizes much smaller. Planning files split into smaller slices <100 lines. Most .ts and .tsx files <250 lines.
When an agent starts to slow down, or give me poor results, it means something has snuck into context that I don't know about.
3 - Rules can backfire.
Because agents are always trying to guess at the "most correct" answer. Clear rules make that easier. But clear rules packaged with a creative task can stifle creativity. Creativity is subjective, so what is right for you might not be right for me.
"Make me a more visually stunning and interactive frontend dashboard. Here are 7 rules to follow...".
It could be a shocking dashboard, but if it follows all the rules the agent still scores 7/8 right.
What's Next
Standardising Prompts. Prompts go into into context. Variability in prompt formatting, language, and tone mean that my output will vary even if my underlying request is the same.
By controlling the input, I can fine-tune the way tasks are communicated, which will improve the reliability of output.
Additionally, creating a prompt schema for different task objectives. The prompt schema and context for a creative brainstorming session is quite different to one which is creating build-specs and architectural diagrams.
Top comments (0)