📰 Originally published on Securityelites — AI Red Team Education — the canonical, fully-updated version of this article.
🗣️ LLM BASICS FOR BEGINNERS FREE
Day 3 of 5 · 60% complete
A lawyer once submitted a legal brief that cited six court cases as precedents. All six cases were completely made up. He’d asked ChatGPT to find supporting cases, and ChatGPT had confidently produced six plausible-sounding but entirely fictional court rulings — complete with case names, dates, and judges. The lawyer didn’t check them. The judge noticed. It became one of the most famous early examples of an LLM hallucination causing real damage.
That story isn’t about the lawyer being careless (though he was). It’s about something fundamental to how LLMs work: they produce confident-sounding text regardless of whether the underlying information is true. They can’t tell the difference between “something I know accurately” and “something that sounds like what would be said here.” The confidence is always the same.
Today I’m explaining exactly why LLMs make mistakes — not in a vague “AI isn’t perfect” way, but the specific mechanical reasons for each failure type. Understanding these will permanently change how you use LLMs.
🎯 What You’ll Learn in Day 3
What hallucinations are and exactly why they happen
Why an LLM’s confidence has nothing to do with its accuracy
How bias gets into LLMs and what it looks like in practice
Why LLMs struggle with maths, counting, and precise logic
The five categories of questions you should always verify
⏱ 20 min read · 3 exercises · Just a browser needed
📋 Before You Start:
- Completed Day 1 and Day 2
- Remember: LLMs predict the next word — they don’t retrieve facts
- Remember: LLMs are frozen after training — they don’t update their knowledge
Why LLMs Make Mistakes — Day 3 of 5
- Hallucinations — Confident Lies From a Prediction Machine
- The Confidence Problem — Why You Can’t Trust the Tone
- Bias — When the Training Data Leans One Way
- Why LLMs Are Terrible at Maths and Counting
- Knowledge Gaps — What the LLM Doesn’t Know
- Five Categories to Always Verify
- Questions and Answers
Days 1 and 2 built your model of how LLMs work. Today that model makes LLM failures make complete sense. None of these mistakes are random or mysterious — they all flow directly from the “predict the next word” mechanism and the training data quality issues from Day 2. The phishing URL scanner we run at SecurityElites is a great counterexample: it’s a specialised model trained specifically on phishing patterns, not a general LLM — which is why it gives reliable results on that specific task even when general LLMs would struggle.
Hallucinations — Confident Lies From a Prediction Machine
A hallucination is when an LLM produces false information as if it were true. Not a mistake the LLM knows is uncertain. A confident, fluent, plausible-sounding statement about something that isn’t real.
The court case story I opened with is a perfect example. The LLM produced six detailed, realistic-sounding case citations — names, dates, courts, judges. None were real. But they followed exactly the pattern of real case citations. The LLM was doing its job: predicting the most probable next words given “find court cases supporting this legal argument.” The most probable next words looked like real case citations. The fact that the specific cases didn’t exist was irrelevant to the prediction engine.
Why does this happen? From Day 1: the LLM is always predicting. There’s no internal check that asks “is this actually true?” The model has no fact-checking mechanism. It has no memory of what actually happened. It just has patterns — statistical relationships between words and phrases from its training data. If the pattern of words that would follow your question is “a confident, detailed-sounding answer,” that’s what it produces — regardless of whether the specific details are real.
Here’s my favourite analogy. Imagine someone who has read every crime novel ever written and can reproduce any plot point, character name, or detective technique from any of those books. Ask them “what happened in the plot of [book title]” and they can give you a detailed, convincing answer — because they learned the patterns of how crime novel plots work. Now ask them about a book that doesn’t exist. They’ll still give you a detailed, convincing plot summary — because they’re generating what a book like that would contain, not reporting what actually exists. That’s an LLM hallucinating.
securityelites.com
// WHY HALLUCINATIONS HAPPEN — THE MECHANISM
USER ASKS:
“What are the main findings of the Smith & Johnson 2019 AI safety paper?”
LLM THINKS:
“What words most likely follow a question about an academic paper’s findings?”
LLM PRODUCES:
“The Smith & Johnson (2019) paper identified three key findings: [detailed, plausible-sounding findings]…”
REALITY:
The paper may not exist. The LLM generated what an answer to that question should look like — not what is actually true.
📖 Read the complete guide on Securityelites — AI Red Team Education
This article continues with deeper technical detail, screenshots, code samples, and an interactive lab walk-through. Read the full article on Securityelites — AI Red Team Education →
This article was originally written and published by the Securityelites — AI Red Team Education team. For more cybersecurity tutorials, ethical hacking guides, and CTF walk-throughs, visit Securityelites — AI Red Team Education.
Top comments (0)