One of the interesting things about using a chatbot is how naturally a conversation can continue.
I can ask a question, provide additional information, refer to something mentioned earlier, and expect the assistant to understand what I mean.
But while studying how chatbots work, I learned an important distinction:
A continuous conversation is something the application must create. It is not automatically provided by every individual LLM call.
What happens when I send a follow-up message?
Imagine this conversation:
User: My food hasn't arrived.
Assistant: Would you like to wait or request a refund?
User: I want a refund.
The last message makes sense because of the earlier exchange.
But what happens if the application sends only “I want a refund” to the model?
The model may not have the information needed to understand the situation. The application needs to preserve the relevant conversation history and include it in the next request.
This was my first important takeaway: the application is responsible for maintaining the conversation.
Memory is not necessarily inside the model
Suppose I tell a chatbot that my preferred programming language is Java.
Later, I ask which language I prefer.
It might seem that the model has permanently learned something about me.
However, the application could have stored that preference, retrieved it when needed, and supplied it as context for the new request.
The process could look like this:
User Information
↓
Application Memory
↓
Retrieve Relevant Information
↓
Add to Current Context
↓
LLM Generates Response
This is different from training the model and updating its learned parameters.
That distinction changed how I think about chatbot memory. The user experience may look like persistent memory even when the application is retrieving stored information and passing it to the model.
More context is not always better
Another concept that stood out to me was the context window.
A model can process only a limited amount of information in a given interaction.
An application might store hundreds or thousands of messages, but it does not necessarily need to send every message with every request.
It can select recent messages, summarize older exchanges, or retrieve only information relevant to the current question.
Why does this matter?
Because repeatedly including a growing conversation can increase input token usage, which can affect cost and latency. It also consumes some of the model's available context capacity.
The important design question is not simply:
“How much context can I send?”
It is:
“Which information does the model actually need to answer this request well?”
Streaming is a user-experience decision
The lecture also connected next-token generation to streaming.
Instead of waiting for an entire answer before showing anything, an application can display generated content progressively as it arrives.
This can make an application feel more responsive, although it does not necessarily make the underlying model generate the complete response faster.
It is another reminder that the experience users see depends on application design, not just model capabilities.
And then there is hallucination
A model can generate an answer that sounds convincing without the answer being factually correct.
Fluent language and verified information are two different things.
For a real application, this matters enormously.
Imagine a support chatbot answering questions about an order. It should retrieve the actual order status rather than inventing one. If an action requires authorization, the application must enforce that authorization instead of trusting a prompt alone.
Depending on the use case, retrieval, external APIs, tool calling, validation, guardrails, and evaluation can improve reliability.
They reduce certain risks, but they do not guarantee that every answer will be correct.
What I'm taking away as a software engineer
I'm beginning to see chatbot engineering as more than sending prompts to a model.
The application needs to manage conversation history, select context, retrieve relevant information, control access to tools, and handle responses appropriately.
These are familiar software engineering concerns applied to a new kind of capability.
My biggest takeaway from this lecture is:
A good chatbot does not merely generate responses. Its application must provide the right context and manage those responses responsibly.
As I continue studying Forward Deployed Engineering and Generative AI, I want to understand not only how models work, but also how to build reliable applications around them.
Top comments (0)