Give Your Chatbot a Memory in Google Colab Before Your Next AI Interview

You don't need a vector database to understand LLM memory.

You don't need LangChain.

You don't need an API key.

You need:

• Google Colab
• Python
• A conversation that keeps growing
• About 15 minutes

In an earlier article, we built the retrieval half of RAG — chunking, embeddings, cosine similarity. This one builds the other half every multi-turn LLM app needs: memory. It's the exact mechanism behind the interview question, "how do you manage memory in an LLM application when conversations get long?"

Let's build it.

What Are We Building?

An LLM has no memory of its own. Every call only sees what you send it — the context window. Left alone, a growing conversation does this:

Turn 1 → Turn 2 → Turn 3 → ... → Turn 50
              ↓
   Send the full history every time
              ↓
   Context window fills, cost climbs

We'll build the fix: a token counter, a summarization trigger, and a remember() function that keeps a conversation coherent without sending everything, every time.

Step 1: A Conversation That Won't Stop Growing

Open a Colab notebook and simulate 50 turns of chat — no API needed yet.

def approx_tokens(text):
    return int(len(text.split()) / 0.75)  # a token is roughly 3/4 of a word

history = []
for i in range(1, 51):
    history.append({"role": "user", "content": f"Question {i} about our product."})
    history.append({"role": "assistant", "content": f"Answer {i}, referencing prior context."})

print("Total turns stored:", len(history))

Step 2: The Cost of "Just Send Everything"

The simplest memory strategy — an in-context buffer — resends the full history on every call. Let's measure what that actually costs.

def cost_of_turn(n_turns):
    sent = history[: n_turns * 2]
    text = " ".join(t["content"] for t in sent)
    return approx_tokens(text)

turn_2 = cost_of_turn(2)
turn_50 = cost_of_turn(50)

print("Turn 2 cost:", turn_2, "tokens")
print("Turn 50 cost:", turn_50, "tokens")
print("Turn 50 is", round(turn_50 / turn_2, 1), "x more expensive than turn 2")

Run it. Turn 50 costs roughly 25x what turn 2 costs — because turn 50 resends 49 prior turns as input. This is the token cost blowout, and it's a cost problem, not just a UX one.

Step 3: Add a Summarization Trigger

Once history crosses a token threshold, compress the old turns into a summary and drop the rest.

SUMMARY_THRESHOLD = 40  # tokens -- small on purpose, so it triggers a few times in 50 turns

def summarize(turns):
    goal = turns[0]["content"]
    facts = [t["content"] for t in turns if "plan" in t["content"].lower()]
    return f"Goal: {goal} | Facts: {facts or 'none stated'} | Compressed {len(turns)} turns"

def remember(user_msg, convo, summary):
    convo = convo + [{"role": "user", "content": user_msg}]
    full_text = summary + " ".join(t["content"] for t in convo)
    if approx_tokens(full_text) > SUMMARY_THRESHOLD:
        summary = summarize(convo[:-2])
        convo = convo[-2:]
    return convo, summary

A real summary can't just be "the last N words" — that's how you lose facts that fell in the middle of the conversation. LLMs already attend poorly to the middle of a long context (a property called "lost in the middle"); a lossy summary makes it worse. A summary that survives should always carry four things: the user's goal, key decisions made, open items, and any persistent facts (name, plan tier, stated preferences). Drop any of those and the assistant re-asks a question it already answered.

Step 4: Watch the Cost Curve Bend

Re-run the growing conversation, but through remember() this time.

convo, summary = [], ""
costs = []

for i in range(1, 51):
    convo, summary = remember(f"Question {i}", convo, summary)
    convo.append({"role": "assistant", "content": f"Answer {i}"})
    sent = summary + " " + " ".join(t["content"] for t in convo)
    costs.append(approx_tokens(sent))

print("Turn 2 cost:", costs[1])
print("Turn 50 cost:", costs[49])

Compare costs[49] here to turn_50 from Step 2 — roughly 28 tokens against roughly 666. It doesn't keep climbing — it stays bounded, because old turns keep getting folded into a fixed-size summary instead of piling up forever.

Make It Reusable

remember() is already the reusable piece. In a real app, summarize() is one extra LLM call instead of the string-matching stub above — same shape, smarter compression. Everything else — the threshold check, the token counter, the trigger — stays exactly as it is.

Five Colab Experiments to Try

1. Set SUMMARY_THRESHOLD = 100 and then 1000. Watch summarization trigger less and less — this threshold is a real production tuning knob.
2. Put a fact in turn 1 ("I'm on the Enterprise plan"), then run 30 turns and print summary after each compression. It survives the first compression — then quietly disappears on the second one, because summarize() only reads the current window, not the running summary. Watch it happen.
3. Now fix it: change summarize() to also accept the current summary string and fold its contents into the new one, instead of overwriting it. Rerun the same 30-turn test — the fact should survive indefinitely this time.
4. Print costs from Step 4 next to [cost_of_turn(n) for n in range(1, 51)] from Step 2. Chart both — that's the cost-vs-turn graph an interviewer is picturing when they ask about scaling memory.
5. Add a user_memory.json file that saves one fact outside history/summary entirely, and load it into a fresh convo/summary pair on a new run — that's semantic memory (persists across sessions), as opposed to episodic memory (resets with the conversation).

Interview Questions Hidden Inside This Notebook

Why not just use a bigger context window?

Cost and attention. A 200K-token context costs far more per call than a 4K one when full, and "lost in the middle" gets worse, not better, as the window grows.

What's the difference between an in-context buffer and summarization memory?

A buffer resends everything and is simplest but unbounded in cost. Summarization compresses old turns into a fixed-size summary once a token threshold is crossed, keeping cost roughly flat.

What is "lost in the middle"?

Even when a full context fits the window, models recall the beginning and end more reliably than the middle. Summaries have to explicitly preserve key facts — not just compress chronologically — or those facts vanish first.

Is memory design a cost decision or a UX decision?

Both, but cost drives the threshold. Turn 50 on a raw buffer can cost 25x turn 2 — that's a unit-economics problem before it's ever a UX one.

What's the difference between episodic and semantic memory?

Episodic memory is what happened this conversation and resets at session end. Semantic memory is persistent facts about the user (plan tier, preferences) that survive across sessions and get injected as context at the start of a new one.

The Memory Manager Is the Easy Part

The remember() function above is maybe fifteen lines. A production memory layer also has to handle:

• context poisoning — a bad instruction from turn 1 surviving into every later summary
• composite retrieval queries — for external/vector memory, a follow-up like "show me another one" has no retrieval signal on its own, so the query needs the summary plus the last couple of turns, not just the latest message
• validating that summaries never overwrite system rules with user-turn content
• knowing when to graduate: buffer for short sessions, summarization once you cross a few thousand tokens, external retrieval only once sessions span hours or multiple visits

Interviewers ask about these because they want to know you've operated a system like this, not just described one.

Want to Go Deeper?

If running this left you asking:

How do I pick a good summarization threshold for my actual traffic?

When do I need external memory instead of summarization?

How do I test that my summary isn't silently dropping facts?

Those are exactly the questions the full session covers — plus the 5-beat interview answer for "how do you manage memory in an LLM application":

https://confidentprep.com/courses/ai-ml-for-interview/4-context-and-memory-management/

And if you'd rather work through this with other developers and ask questions live, join one of the upcoming live sessions:

https://confidentprep.com/live/

Build the small version first. Watch it break at turn 50. Then learn how to explain why — that's a better way to prepare for an AI interview than memorizing another list of memory-management definitions.