Did you ever observed how LLMs chatbot like ChatGPT can’t recall older messages. LLMs usually struggle with this issue of memory loss in the middle of the conversation and forget the older conversation/memory. To understand why this happens we need to understand Context Window.
Context Window: The maximum amount of token the model can process and read. The context window depends on model to model. Early models take up to 4000-5000 token or less. The Current models have much larger windows: GPT-4 class models run 128k tokens, Claude and Gemini range from 200k up to 1M+.
At the beginning of the conversation, we have enough space for fresh new conversation. As the conversation will progress and it will fill up the context window, this is known as Context Window fill up. To make more room for the new conversation it will drop the previous conversation. This is especially a huge disadvantage when it comes for large scale project and research.
can we solve this by having larger context window? Sure, context windows will have more token, but it does comes with huge cost for tokens, take up alot of memory and security issues.
*How to overcome Context Window Limitation? *
Retrieval-Augmented Generation(RAG)-It retrieves the most relevant data from external sources with knowledge and only provides the relevant information to user’s query.It reduces the hallucination and provides relevant data
Tools-The models decides when to look up the data, chain multiple lookups and fetches the data on demand.It runs on loops and work for live data.Reduces unnecessary context that might take up space
Top comments (0)