The most dangerous thing my AI ever did was remember something wrong, calmly.
It told me a file lived in a certain place. It did not. It cited a path that sounded canonical, in a folder that had never existed. Nothing in the tone gave it away. The error had the same warm confidence as every correct answer before it.
That is the failure that stays with me. Not the crash, not the timeout. The fluent sentence with nothing behind it.
So I changed how the system treats its own past. Memory is now a hint, never a fact. Before anything acts on a remembered detail, it has to look: list the directory, read the file, run the command. If the check fails, the memory gets replaced, not stacked next to the new fact. And when a report says something was published or deployed, it has to bring a receipt, an ID or a hash from the tool itself. No receipt means the claim is marked unconfirmed, whatever the sentence sounds like.
This is slower. It is also the first time I have felt I can leave the system alone overnight.
Building in public usually means showing the wins. I think the more useful thing to show is the repair log: here is where it was wrong, here is the rule that came out of it, here is how I check that the rule holds.
A system that says 'I do not know' is less impressive in a demo and much better company at 4 a.m.
If your assistant has a memory, ask it one question this week: how would you know if you were wrong?
Top comments (0)