Picture a shop owner at closing time asking their assistant one question: who still owes me money?
The assistant looks it up. The database returns the right customer and the right balance. Then the assistant writes its answer, and the customer’s name in that answer is one letter off. It’s still a real name, it looks fine, and the owner picks up the phone and calls the wrong person.
That was the bug we spent the most time on while building Dash, the assistant inside our bookkeeping app for small shops. It surprised us, because everything we had been told to worry about worked.
Everything we tested for worked
The usual worries about an AI agent are that it picks the wrong tool, passes it bad arguments, or invents data. We built tests for all three, and in real conversations they mostly held. The assistant chose the right lookup, the lookup returned correct data, and the data reached the model intact.
The errors were in the final step, where the model turns that data into a sentence for a person. It was retyping facts it had just been handed, and sometimes it retyped them wrong.
One letter, a different person
Our users write in Arabic, Kurdish and English. Names in those languages are often short and share most of their letters, so a single changed letter usually produces another real name instead of an obvious typo. A customer called Karim came back as Kiram. A staff role meaning “system administrator” came back as something closer to “commissioner of cleanliness”. A product name lost its last letter and turned plural.
None of these would have shown up in an English-only test set, which is a big reason we didn’t see them coming.
Two things made them worse than ordinary typos.
They spread. Each reply is saved to the conversation, and the model rereads the conversation on the next turn. Once it had written the wrong name, it trusted its own earlier message and kept using it. A one-off slip became the version of the truth for the rest of the chat.
Asking again didn’t help. Our first fix was the obvious one: when we caught a wrong name, ask the model to write the reply again. In our test runs it made the same mistake the second time, and each retry cost a full extra call and the wait that comes with it.
So we stopped asking and started correcting
We already knew the right answer, because it was sitting in the data the tool had just returned. The question was whether we could detect the error precisely enough to fix it without asking the model again.
We made the check deliberately narrow:
It only considers names that appeared in this conversation’s own data.
It flags a word only when the correct name is missing from the reply and a word in the reply is exactly one letter away from it, written in the same alphabet.
If the correct name appears anywhere in the reply, it leaves everything alone.
When all three conditions hold, it swaps that one whole word for the correct name and logs the correction. It never edits inside a word or touches anything else.
We had argued against editing model output at first, on the worry that an automatic fix could do more damage than the mistake. That reasoning held while errors were rare and a retry usually fixed them. Once neither was true, a precise, logged, one-word correction was the safer choice.
Numbers: never let the model do the typing
Names were a surprise. Numbers we had been wary of from day one.
So the model never formats money. Every amount our tools return comes with a finished, display-ready version made by ordinary code: the right decimal places for the currency, thousands separators and the currency code. The model is told to copy that text exactly.
Then we check that it did. Every amount in the final answer has to match one of those display values from the same conversation. The only exception is an amount clearly labelled as an example. Both sides are normalised before the comparison, so the same number written in Arabic numerals still counts as a match.
One set of checks, run in two places
Each of these checks runs twice.
Inline, on every reply before it reaches the user: the name correction and number formatting.
Offline, as tests over scripted conversations and real production conversations. They check four things:
names were copied faithfully;
money amounts match the data;
every record the assistant proposes to change came from real data and wasn’t invented;
every change the assistant made was approved by the user first.
Both run the same detection code, so the thing we measure and the thing we enforce can’t drift apart. When an offline check fires on real conversations, either the inline guard is missing a case or something has regressed, and we find out before a user does.
What we’d tell anyone building this now
Check the last step as well as the first. Compare the reply against the data it came from. That’s where our errors were.
Hand the model finished text for anything that matters. If a name or number must be exact, have code produce it and the model copy it, then check the copy.
Measure whether a retry fixes anything. Ours reproduced the same mistake.
Correct deterministically when you can be precise. Narrow the check until you’d trust it to edit text, then let it.
Test in your users’ languages. Most of what we found would never appear in English.
The assistant this is about
Dash lives inside Dashing, a point-of-sale and bookkeeping app for small shops, in Arabic, Kurdish and English. Owners ask it about their own business: who owes them, what sold this week, what’s running low. When it suggests a change, such as recording a payment, nothing happens until the owner approves it.
The checks in this article sit behind every one of those answers. If you want to see how it holds up, the free plan lets you open a shop and ask it yourself: Try Dashing free
Top comments (0)