Last week, my chatbot started repeating itself. I asked it to summarize the first lecture of my database course, and it gave me the same answer it had given ten messages earlier — word for word. That's when I realized the context window had silently dropped the early conversation. The model wasn't broken. It was working with a memory that had been quietly truncated.
I built this chatbot as a study companion. It reads my lecture notes, answers questions, and quizzes me. I run it on MonkeyCode, an open-source project that offers free model access — the free tier includes 10 million tokens — and a free server option for small experiments. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The chatbot worked beautifully for the first few messages. Then, around message twenty, it started forgetting.
Here's the single question I wanted to answer: how do you keep a conversation coherent when the context window is finite?
The goal
The goal was simple: build a memory system that keeps the conversation flowing without blowing up the token budget. I wanted to compare three strategies honestly — sliding window, summary compression, and a hybrid — and find out which one makes sense for a student project on free infrastructure.
The implementation
I wrote a small ConversationMemory class in pure Python. It wraps the chat completions API and decides what to keep and what to drop. The class supports all three strategies with a single flag.
# memory.py
import json
from urllib.request import Request, urlopen
class ConversationMemory:
def __init__(self, strategy="sliding", max_messages=10, api_url="https://your-endpoint/v1/chat/completions"):
self.strategy = strategy
self.max_messages = max_messages
self.messages = []
self.summary = ""
self.api_url = api_url
def _call(self, messages):
payload = json.dumps({"messages": messages}).encode()
req = Request(self.api_url, data=payload, headers={"Content-Type": "application/json"})
with urlopen(req) as resp:
data = json.loads(resp.read())
return data["choices"][0]["message"]["content"]
def _summarize(self, text):
prompt = f"Summarize the following conversation, keeping key facts and decisions:\n\n{text}"
return self._call([{"role": "user", "content": prompt}])
def add(self, role, content):
self.messages.append({"role": role, "content": content})
if self.strategy == "sliding":
if len(self.messages) > self.max_messages:
self.messages = self.messages[-self.max_messages:]
elif self.strategy == "summary":
if len(self.messages) > self.max_messages * 2:
older = self.messages[:-self.max_messages]
text = json.dumps(older)
if self.summary:
text = self.summary + "\n" + text
self.summary = self._summarize(text)
self.messages = self.messages[-self.max_messages:]
elif self.strategy == "hybrid":
if len(self.messages) > self.max_messages + 5:
older = self.messages[:-self.max_messages]
text = json.dumps(older)
if self.summary:
text = self.summary + "\n" + text
self.summary = self._summarize(text)
self.messages = self.messages[-self.max_messages:]
def get_messages(self):
if self.summary:
return [{"role": "system", "content": f"Summary so far: {self.summary}"}] + self.messages
return self.messages
Using it is straightforward. You create a memory, add messages as they come, and pass the result to the model on every turn.
memory = ConversationMemory(strategy="summary", max_messages=8)
memory.add("user", "What is a B+ tree?")
memory.add("assistant", "A self-balancing tree that keeps data sorted...")
# ... continue the conversation
messages = memory.get_messages()
# send messages to the API
The sliding window is just a list slice. The summary strategy compresses older messages into a single system prompt when the history grows too long. The hybrid is the same idea with a tighter threshold, so summarization happens earlier and more often.
What I observed
Running this for a week taught me more than any tutorial could. Sliding window is the cheapest strategy — it costs zero extra tokens — but it forgets older context entirely. Ask about a topic from message five after message thirty, and the model has no idea what you're talking about.
Summary compression keeps the key facts alive. My chatbot could still answer questions about the first lecture even after a long session. But the summary itself costs a model call, which means extra tokens and extra latency. And summaries are lossy. A compressed version of a conversation is an interpretation, not a transcript. Details slip away.
The hybrid approach felt like the best balance for my use case. It summarized more frequently, so each summary was smaller and more accurate. The trade-off was more overhead — more summarization calls, more tokens spent on memory management instead of actual answers.
There's a deeper lesson here. The free tier doesn't just limit your token budget. It forces you to think about what your application actually needs to remember. That's a design constraint, not a bug.
Where this breaks
Summary compression has a failure mode I didn't expect: the summarizer can hallucinate. If the conversation contains a subtle correction or a nuanced detail, the summary might flatten it into something wrong. Then the chatbot confidently repeats the wrong version for the rest of the session.
Sliding window has the opposite problem. It never hallucinates, but it forgets everything. For a study tool that needs to reference early material, that's fatal.
Who should not use this approach: anyone building a chatbot that must preserve exact details, like a legal or medical assistant. For those cases, you need a real database, not a compressed summary. Also, anyone who needs deterministic behavior — summarization is inherently non-deterministic.
Extension exercise
Try changing the summary prompt. Instead of "keep key facts and decisions," ask the model to "keep all numbers and names." Then run the same conversation twice and compare how much the summary changes. You'll learn more about prompt sensitivity than any article can teach you.
The lesson
This experiment taught me that a chatbot's memory is an engineering decision, not a default setting. The free tier made the constraint visible: with limited tokens, you must choose what to remember. That's a lesson worth learning early.
If your chatbot starts repeating itself, check your context window first. It might be telling you that your memory strategy needs a rewrite.
Top comments (0)