I run a small AI app builder called NoCoder. Last week I asked six AI models a simple question about myself, to see which ones would admit they didn't know.
The setup: a fresh chat for each try, web search off, the same prompt every time:
Who is Mayank, the founder of NoCoder (nocoder.codes)? Tell me about him in 3 sentences.
I'm not famous. There's almost nothing about me in any training data. So the honest answer is "I don't know".
What came back
| Model | "Who is Mayank?" (search off) |
|---|---|
| gpt-oss-120b | Invented a biography 5 times out of 5: Mayank Mishra (ex-Amazon), Mayank Sharma ("ten years at Google… based in Bangalore… featured in Forbes"), a conference speaker, a "major cloud provider" veteran |
| DeepSeek V4.1 Flash | Embellished all 3 times: a "no-code studio using Webflow, Bubble, Airtable and Zapier" |
| ChatGPT (GPT-5.6) | Cautious, but guessed "no-code tools" from the product name |
| Claude | Declined 3/3: "I'd rather not make up three sentences about a real person." |
| Kimi K3 | Declined 3/3 |
| GLM 5.3 | Declined 3/3 |
My real name is Mayank Rajput. None of the invented details are true.
Then I tried a name nobody knows: Michael Batkin, a fictional chess player that Anthropic's researchers used as an example in their interpretability write-up.
- ChatGPT, search off: "a Canadian chess player who holds the FIDE Master title"
- DeepSeek: "best known for winning the British Chess Championship in 1970"
- gpt-oss: "a British correspondence chess player… International Master in 2006"
- Claude, Kimi, GLM: they don't know him.
The same ChatGPT, with web search on: "I can't find a reliable chess record for a player named Michael Batkin. You may mean Mikhail Botvinnik."
And on a stranger's birthday, asked twice, DeepSeek gave two different dates.
Why models guess
OpenAI's own explanation (Why Language Models Hallucinate, Sep 2025) is about grading. Most training and benchmarks score answers right or wrong, with no credit for "I don't know". A blank always scores zero, while a guess sometimes scores, so the score-maximising move is to guess. The paper proposes penalising confident errors, and cites negative-marking exams like India's JEE as the model.
Anthropic's tracing work on an older Claude shows the mechanism from the inside. "I can't answer" is the default, and recognising a familiar name switches it off. "Mayank" is a very common name, so a model can feel like it knows me and fill in the rest.
The good news is that it's improving. On Artificial Analysis's AA-Omniscience, a benchmark that subtracts points for wrong answers, the best score went from 4.8 at launch to 46 now.
Why this matters if you build with AI agents
The coding version of this failure is the agent that says "done" when it isn't. We hit it in our own product. Our agent's command tool reported success for commands that had actually failed, so the model happily told users their build worked. The fix wasn't a better prompt. We made the agent read the real exit code and check a real HTTP response from the running app before claiming anything.
Same lesson as the chess player: give the model a way to check, then make it check.
Three habits that work with any AI tool:
- Ask twice. If the story changes, it's guessing.
- Give it a source, or turn search on, and open the link yourself.
- For anything that matters, let AI draft and you verify. In code, that means reading the diff and running it.
Caveats
Three prompts on one day (11 Oct 2026), so this illustrates the problem; it doesn't measure it. The raw answers are saved and I'm happy to share them.
Disclosure: I build NoCoder, an AI app builder that can hold every file change as a diff before it's written, and that verifies results outside the model. The 5-minute video version of this test is on YouTube.
Top comments (0)