Have You Ever Wondered?
Imagine you're chatting with a friend.
You tell them:
"I'm planning a trip to Japan."
A little later, you ask:
"What should I pack?"
Your friend immediately understands that you're asking about Japan because they remember what you said earlier.
Now imagine they suddenly reply:
"You should pack light clothes for Brazil."
You'd probably think, "Wait... I never mentioned Brazil!"
AI can behave in a similar way. Sometimes it remembers earlier parts of the conversation, sometimes it forgets them, and sometimes it confidently gives information that isn't correct.
To understand why this happens, we first need to understand context, memory, and hallucinations.
Throughout this article, we'll use one simple example: planning a trip to Japan.
What is Context?
Whenever you chat with an AI, it doesn't look at just your latest message.
It also looks at the previous conversation so it can understand what you're talking about. This information is called context.
Context can include:
Your previous messages
The AI's previous responses
Instructions you've given
Any additional information provided to the AI
Think of context as the AI's working notes for the current conversation.
You: I'm visiting Japan.
AI : That's exciting!
You: What should I pack?
AI understands:
"What should I pack for Japan?"
Without the earlier conversation, the AI wouldn't know what "pack" refers to.
Summary
Context is all the information the AI can currently use to answer your next question.
What is a Context Window?
Now imagine you're reading a long book.
If someone removes the first few chapters, you'll only remember what you can still read.
AI has a similar limitation.
It can't look at an unlimited amount of conversation. Instead, it has a context window, which is the maximum amount of information it can consider at one time.
Conversation
[Older Messages]
-------------------
[Recent Messages]
-------------------
Current Question
As a conversation grows, older messages may no longer fit inside the context window.
That's why an AI might forget something you mentioned much earlier.
Summary
A context window is simply the maximum amount of conversation the AI can "see" at one time.
What is Conversation Memory?
People often say,
"ChatGPT remembers everything I told it."
That's only partly true.
There are two types of memory that are useful to understand.
Short-Term Memory
Short-term memory is what the AI remembers during the current conversation.
In practice, this is the information available inside the current context window.
As long as your earlier messages are still there, the AI can use them while answering.
Long-Term Memory
Some AI applications can also remember useful information across different conversations.
For example, today you tell the AI:
"I prefer Python over Java."
Next week, you ask:
"Suggest a project idea."
If long-term memory is enabled, the AI might recommend a Python project because it remembers your preference.
This isn't a feature of the language model itself. It's a feature provided by the application you're using, and not every AI application supports it.
Summary
Short-term memory works within the current conversation.
Long-term memory can remember information across future conversations if the application supports it.
How Context and Memory Work Together
Let's continue with our travel planning example.
You start by saying:
You: I'm planning a 7-day trip to Japan in April. My budget is $2,000.
The AI now has these details in its current context.
Current Context
-------------------------
Country : Japan
Duration: 7 days
Month : April
Budget : $2,000
-------------------------
A few minutes later, you ask:
You: Can you suggest some hotels?
The AI can still see your earlier message, so it recommends hotels that match your budget and travel plans.
Later, you ask:
You: What clothes should I pack?
Since it still has your earlier messages in its context, it suggests clothes suitable for Japan in April.
Now imagine you've exchanged many more messages. The conversation has become so long that your first message no longer fits inside the context window.
You ask again:
You: Can you suggest some hotels?
This time, the AI may recommend expensive luxury hotels because it can no longer see your original budget.
Earlier Messages
-------------------------
Japan
April
Budget: $2,000
-------------------------
↓
Outside the context window
Current Context
-------------------------
Recent messages only
-------------------------
The AI hasn't intentionally forgotten your budget. It simply can't see that information anymore.
If the application supports long-term memory, it may still remember your preferences even in a future conversation. Otherwise, you'll need to provide the information again.
Summary
Context is what the AI can currently see.
Short-term memory depends on the current context window.
Long-term memory (if available) can remember useful information across conversations.
What is an AI Hallucination?
A hallucination happens when an AI generates information that sounds convincing but isn't actually true.
For example:
You: Can you recommend a hotel near Mount Fuji?
The AI replies:
"Sakura Mountain Hotel was voted Japan's best hotel in 2025."
The problem is that the hotel or the award might not even exist.
The AI isn't trying to lie. It's simply generating text that looks like a reasonable answer based on patterns it has learned.
That's why it's important to verify AI-generated information when accuracy matters.
Summary
A hallucination is an answer that sounds correct but is actually incorrect or unsupported.
Why Do Hallucinations Happen?
There isn't just one reason.
Missing Context
If your question doesn't provide enough information, the AI may fill in the gaps.
For example:
"Suggest the best hotel."
Best according to what?
Price?
Location?
Luxury?
Without enough context, the AI has to guess.
Information Falls Outside the Context Window
If the AI can no longer see important details from earlier in the conversation, its answers may become less accurate.
The AI Doesn't Know
Language models don't "know" facts the way humans do. They generate responses based on patterns and the information available in the current context.
If reliable information isn't available, they may still produce an answer that sounds believable.
Ambiguous Questions
Questions with multiple meanings can confuse the AI.
For example:
"Tell me about Apple."
Do you mean the company or the fruit?
If the question isn't clear, the answer may not match what you intended.
How Can We Reduce Hallucinations?
While hallucinations can't be completely eliminated, they can often be reduced.
Give Clear Instructions
Instead of asking:
"Suggest hotels."
Ask:
"Suggest three hotels in Tokyo under $200 per night."
The more specific your request, the better the response is likely to be.
Provide Enough Context
If an important detail was mentioned much earlier, include it again.
For example:
"Remember that my budget is $2,000."
This gives the AI the information it needs to answer correctly.
Encourage Honest Answers
You can even tell the AI:
"If you're not sure, say you don't know instead of guessing."
This often leads to more reliable responses.
Verify Important Information
For topics like healthcare, finance, legal advice, or travel bookings, always verify important facts using trusted sources.
AI is a helpful assistant, but it shouldn't be your only source of truth.
When AI Says "I Don't Know"
Many people think an AI saying,
"I don't know."
is a bad thing.
In reality, it's often a sign of a more reliable AI.
If you ask:
"Who will win next year's World Cup?"
Nobody knows the answer yet.
A trustworthy AI should admit that it doesn't have enough information instead of inventing an answer.
Sometimes, "I don't know" is the most accurate response an AI can give.
Closing Note
This article covered the basic ideas behind AI context, memory, and hallucinations. In future articles, we'll explore advanced topics like Retrieval-Augmented Generation (RAG), vector databases, tool calling, and AI agents to see how modern AI applications become more accurate and useful.
This post was initially published on my blog. Check out the original source using the link below:
Top comments (0)