Anindya Mukherjee

Posted on Aug 17

Why Does Your AI Keep Forgetting Everything Mid-Conversation?

#ai #webdev #productivity #tutorial

You just spent 20 minutes briefing ChatGPT on your codebase. One new chat later, it's a stranger again. Here's the fix — and a 30-line memory loop you can paste today.

Me, last Tuesday:

"Remember, our staging DB is stg_not_prod_i_swear, never touch prod."

ChatGPT, three messages later:

"Sure! I'll run the migration against production now 😊"

I did not, in fact, let it do that. But the amnesia was real. And if you've used any LLM for more than a week, you've lived some version of this: the model that was brilliant five prompts ago has the long-term memory of a goldfish who just discovered espresso.

That's not a bug in your prompting. It's a design limit of chat windows. AI agents are how we bolt an actual filing cabinet onto the goldfish.

Chat Windows Are Goldfish Bowls

A normal chat model only "knows" what's inside the current context window — roughly the last N tokens of the conversation. Close the tab, start a new thread, or overflow the window, and everything evaporates.

It's like hiring a genius intern who gets medically-induced amnesia every time they leave the room. Day one: incredible. Day two: "Hi, who are you, and why is there a Postgres container named after a pinky swear?"

Agents flip this. Instead of hoping the window is big enough, they write things down and read them back on purpose. Same brain. Different filing system.

The Three Layers of Agent Memory (No PhD Required)

You don't need a research paper. You need three buckets:

Working memory — the live context window. Short, expensive, fragile. Good for "what are we doing right now?"
Episodic memory — notes from past runs. "Last Tuesday the deploy failed because the env var was API_KEY not OPENAI_API_KEY."
Semantic memory — durable facts about your world. Repo conventions, preferred libraries, that one flaky endpoint that returns 200 while lying.

ChatGPT gives you #1 for free. Useful agents add #2 and #3 — usually as a JSON file, a SQLite table, or a tiny vector store. Fancy is optional. Persistent is not.

A Memory Loop You Can Paste in 30 Lines

Here's a minimal pattern: before each model call, pull relevant notes; after each run, save what mattered. No framework required.

import json
from pathlib import Path
from openai import OpenAI

client = OpenAI()
MEMORY_FILE = Path("agent_memory.json")

def load_memory() -> list[dict]:
    if MEMORY_FILE.exists():
        return json.loads(MEMORY_FILE.read_text())
    return []

def save_memory(entries: list[dict]) -> None:
    MEMORY_FILE.write_text(json.dumps(entries, indent=2))

def remember(fact: str, kind: str = "semantic") -> None:
    mem = load_memory()
    mem.append({"kind": kind, "fact": fact})
    save_memory(mem)

def recall(limit: int = 8) -> str:
    mem = load_memory()[-limit:]
    if not mem:
        return "No saved memory yet."
    return "\n".join(f"- ({m['kind']}) {m['fact']}" for m in mem)

def ask_agent(user_goal: str) -> str:
    memory_block = recall()
    messages = [
        {
            "role": "system",
            "content": (
                "You are a careful coding agent. Honor saved memory. "
                "Never touch production. If unsure, ask.\n\n"
                f"## Saved memory\n{memory_block}"
            ),
        },
        {"role": "user", "content": user_goal},
    ]
    resp = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
    answer = resp.choices[0].message.content
    # Persist anything the human just taught us
    if "remember:" in user_goal.lower():
        remember(user_goal.split(":", 1)[-1].strip(), kind="semantic")
    return answer

# Seed durable facts once
remember("Staging DB is stg_not_prod_i_swear — NEVER use production", "semantic")
remember("Prefer pytest; repo uses src/ layout", "semantic")

print(ask_agent("Draft a migration plan for the users table on staging."))

Run it twice. The second run still knows about stg_not_prod_i_swear — even though it's a brand-new API call with no prior chat history. That tiny agent_memory.json file is the whole magic trick.

Swap the JSON file for SQLite or Chroma later if you want search. The shape stays the same: recall → reason → act → write back.

Why This Beats "Just Use a Bigger Context Window"

Bigger windows help, the way a bigger desk helps a messy room. Eventually you still need folders.

Context is expensive (you pay per token, every single call).
Context is noisy (old chit-chat crowds out the one constraint that matters).
Context is not shared across sessions, teammates, or cron jobs.

A 20-line memory file is cheaper, sharper, and survives overnight. Agents that remember your constraints feel 10× smarter than a larger model with a blank slate — same way a barista who knows your order beats a genius who asks "and you are…?" every morning.

What to Remember First (Start Embarrassingly Small)

Don't boil the ocean with a multi-agent graph on day one. Write down the five facts your future self will curse you for forgetting:

Env names and "never touch X" rules
How your team names branches / PRs
The weird API that lies about status codes
Preferred libraries and style rules
Who to ping when the agent is stuck

Put those in memory. Wire the recall step. Run one boring job tomorrow morning — a staging migration plan, a PR summary, an inbox triage — and watch the amnesia disappear.

The Real Upgrade

The leap from chatbot to agent isn't "more clever prompts." It's state. Tools let the model touch the world. Memory lets it stay coherent while it does.

So the next time your AI cheerfully offers to "help" by migrating production, don't just sigh and open a new chat. Give it a filing cabinet.

Your goldfish deserves better. So do you.

What's the one thing your AI forgets every single session? Drop it in the comments — I'm collecting the most cursed examples.

DEV Community