Day 3/30: LLM Context Window Limits

I was working on a support bot that used a large language model (LLM) to generate responses to user queries. The bot was designed to have a conversation with the user, answering follow-up questions and providing additional information as needed. However, I noticed that the bot would often forget the context of the conversation, providing responses that were irrelevant or even contradictory to what it had said earlier. After digging into the issue, I realized that the problem lay in the way I was using the LLM: I was making single-shot calls to the model, providing it with a prompt and then using its response as the input for the next prompt.

The issue with this approach is that the LLM's context window is limited, typically to a few hundred tokens. When the user asks a follow-up question, the LLM has to process the entire conversation history, which can exceed its context window. As a result, the model starts to drop earlier parts of the conversation, effectively "forgetting" what was said earlier. This can lead to responses that are disconnected from the rest of the conversation.

To illustrate this problem, let's consider an example. Suppose we have a support bot that uses the LLM to answer user queries about a product. The user asks a series of follow-up questions, and the bot responds using a single-shot LLM call:

import langgraph

# Create an LLM instance
llm = langgraph.LLM()

# Define the conversation history
conversation_history = ""

# User asks a question
user_question = "What is the return policy for this product?"
conversation_history += user_question + "\n"

# Bot responds using a single-shot LLM call
response = llm.generate(conversation_history + "Bot: ")
conversation_history += response + "\n"

# User asks a follow-up question
user_question = "Can I return it after 30 days?"
conversation_history += user_question + "\n"

# Bot responds using a single-shot LLM call
response = llm.generate(conversation_history + "Bot: ")
conversation_history += response + "\n"

In this example, the conversation history grows with each user question and bot response. However, the LLM's context window is limited, so it starts to drop earlier parts of the conversation. As a result, the bot's responses become disconnected from the rest of the conversation.

To fix this issue, we need to use a more sophisticated approach to manage the conversation context. One way to do this is to use a state graph to keep track of the conversation history and use the LLM to generate responses based on the current state of the conversation. We can use the StateGraph class from the langgraph library to create a state graph and add nodes and edges to represent the conversation history.

For example:

import langgraph

# Create a state graph
state_graph = langgraph.StateGraph()

# Add nodes to represent the conversation history
state_graph.add_node("start")
state_graph.add_node("return_policy")
state_graph.add_node("return_after_30_days")

# Add edges to represent the conversation flow
state_graph.add_conditional_edges("start", "return_policy", condition="user asks about return policy")
state_graph.add_conditional_edges("return_policy", "return_after_30_days", condition="user asks about returning after 30 days")

# Use the LLM to generate responses based on the current state of the conversation
def generate_response(state):
    if state == "start":
        return "Please ask a question about the product."
    elif state == "return_policy":
        return "The return policy is 30 days."
    elif state == "return_after_30_days":
        return "Yes, you can return it after 30 days."

# Use the state graph to manage the conversation context
def manage_conversation(conversation_history):
    current_state = "start"
    for user_question in conversation_history:
        if user_question == "What is the return policy for this product?":
            current_state = "return_policy"
        elif user_question == "Can I return it after 30 days?":
            current_state = "return_after_30_days"
        response = generate_response(current_state)
        conversation_history += response + "\n"
    return conversation_history

In this example, we use a state graph to keep track of the conversation history and generate responses based on the current state of the conversation. This approach avoids the context-window-as-to-do-list problem and provides a more robust way to manage conversation context.

One practical gotcha to watch out for when using this approach is to make sure that the state graph is properly updated after each user question and bot response. If the state graph is not updated correctly, the conversation context can become disconnected, leading to irrelevant or contradictory responses.

As we continue to build more sophisticated agentic AI systems, we'll need to explore new ways to manage conversation context and generate responses that are relevant and coherent. Tomorrow, we'll dive into another critical aspect of building agentic AI systems, and explore how to use LangGraph and MCP to create more engaging and interactive conversations.