You're not prompting it wrong.

I was listening to Grady Booch on The third golden age of software engineering episode of The Pragmatic engineer, and at one point during the episode he mentioned a website called Victorian Engineering Connections — an interactive diagram of how Victorian engineers knew and influenced each other.

I was in the middle of a session with Claude, so I asked claude to remind me of the name and it pointed me to sixdegreesoffrancisbacon.com. I then mentioned that I heard Grady Booch talk about it on a podcast. Still, it gave me three plausible-sounding wrong answers.(not quite hallucinations, the links worked. When I typed the actual name, it found it immediately.

I went and searched google for victorianengineeringconnections.net and the first link was to a blog post that reviewed the website, so I knew that Claude had references to the website in the training data.

It made me think of two things. The Library of Babel and "You're prompting it wrong.

There are virtually infinite strings of text one could use to prompt an LLM, and each will provide different responses. So in a sense, an LLM is a giant library of babel(with some randomness) where each person has their own unique index.

One of the reasons LLM's seem amazing is that they can answer almost any question we have. They feel right next to you the whole time. And that's because they are. People have often said "LLM's don't work for you, because you're prompting it wrong". But what if LLM's only work for people who are already subject matter experts? Because they're unique keys - the way they form their strings of text - are the only ones that lead the the correct answers.