I spent three weeks building a feature that relied on an LLM to return structured data about a specific set of API endpoints. During testing, it worked perfectly. In production, it started hallucinating fields that did not exist, and the whole frontend crashed because the JSON was malformed.
My mistake was treating the model like a database. I expected it to recall specific facts and return them in a strict format every single time. That is not how these models work. They are probability engines, not lookup tables.
The Probability Trap
When you ask an LLM for a specific fact, it is not querying a disk. It is predicting the next most likely token based on weights. If the prompt is slightly different, or if the model version updates, that probability shifts.
If your application logic depends on the AI always remembering that "User ID is a UUID v4", you are building on sand. The model might get it right 99 times, but the 100th time it will return a string like "The User ID is a UUID v4: 123-abc" and your JSON.parse() call will explode.
The Pattern: Separation of Knowledge and Reasoning
To make AI features stable, you have to separate the knowledge (the facts) from the reasoning (the processing).
Instead of asking the AI to remember your business rules, feed those rules into the prompt as context. This is the core idea behind RAG, but you do not always need a complex vector database to do it. Sometimes a simple system prompt with a list of constants is enough.
Here is how I restructured my approach:
- Externalize the Truth: Put all factual data in a JSON file or a database.
- Inject Context: Fetch the necessary data and inject it into the prompt. Tell the AI: "Using only the following data, perform X task."
- Schema Enforcement: Stop hoping for valid JSON. Use a library that enforces a schema (like Zod in TypeScript) to validate the output before it ever touches your application state.
Handling the "Creative" Drift
Even with a strict schema, models drift. I noticed that as I added more context to my prompts, the AI started ignoring the middle section of the instructions. This is a known issue called "lost in the middle."
To fix this, I started using a technique called few-shot prompting. Instead of telling the AI "Return a list of errors", I gave it three concrete examples of a request and the exact expected response.
Example:
Input: "The server is down"
Output: { "category": "infrastructure", "severity": "high" }
Input: "I forgot my password"
Output: { "category": "user-auth", "severity": "low" }
This anchors the model. It stops guessing the format and starts mimicking the pattern.
Testing for Non-Determinism
Unit tests for AI are different. You cannot test for equality because the output changes. I started implementing "LLM-based evaluation".
I wrote a separate, more powerful model (like GPT-4) to act as a judge for my smaller, faster production model. The judge model does not check if the strings match. It checks if the intent and the facts are correct based on a rubric.
If the production model says "The price is 10 dollars" and the judge knows the price is 10.00, it passes. If it says "The price is 12 dollars", it fails. This is the only way to scale AI features without manually reading a thousand logs a day.
The Concrete Takeaway
If your code breaks because the AI forgot a detail or changed a word, your architecture is the problem, not the model.
Move your facts out of the model weights and into your code. Use the LLM for the transformation and reasoning, but keep the source of truth in a place where you can actually version control it. Treat the AI as a flaky intern: give them clear instructions, provide examples, and always double check their work with a validator.
Top comments (0)