DEV Community

Cover image for I Built an AI That Forgets on Purpose
Adedotun Akindele
Adedotun Akindele

Posted on

I Built an AI That Forgets on Purpose

#ai #hackathon #llm

Every chat with an AI feels like a fresh start, doesn’t it? It doesn’t remember what you mentioned yesterday, can’t connect notes across sessions, and treats a casual "good morning" the same as a critical lab result. It’s a little frustrating, honestly.

So I built a memory system where information has to earn its place.

Every message gets scored for importance the moment it arrives. If it turns out useful later, its lifespan extends. Once a night, a background job consolidates the valuable stuff into permanent summaries and facts, and lets the rest fade away.

The name Àtúnbí means "reborn" in Yoruba. Raw memory comes and goes. The good stuff sticks around.

I built this for the Qwen Cloud Hackathon, and tested it by tracking two diabetic patients — Ada and Kai — across visits, medication changes, and lab results. But it works for more than healthcare: swap patients for codebases, legal cases, research papers, or your own notes, and the core model stays exactly the same.

The Architecture: Two Paths, One Brain

The system is split into two complementary rhythms to balance speed and depth: one for real-time responses, another for slow, nightly learning. This split is what makes long-term memory feel snappy, not clunky.

Serving Path (Real-Time)

This loop responds in under a second:

  1. Ingest: Normalize every input (voice, PDFs, images, text) into a single text stream
  2. Process: Embed, score, and extract entities from incoming messages
  3. Working Memory: Store raw messages with dynamic expiration timestamps
  4. Retrieval: Query all memory tiers to build context, then generate a response
  5. Feedback loop: Every time a memory is used, its lifespan extends — it earns its spot.

Learning Path (Nightly Dream Phase)

Once every 24 hours, EventBridge triggers a separate job for slow, deep work:

  • Pulls all Working Memory entries older than 24 hours
  • Clusters related messages into conversation summaries → saved to Episodic Memory
  • Extracts standalone facts → saved to Semantic Memory (contradictions mark old facts as superseded, never deleted)
  • Updates relationships in the Entity + Procedural Memory graph
  • Prunes low-score, unused entries — this is the intentional forgetting.

The Five Memory Tiers

Tier Path Purpose Lifespan
Working Memory Serving Raw messages + embeddings 1h–168h, extended by use
Episodic Memory Learning Conversation summaries Permanent (unless updated)
Semantic Memory Learning Verified standalone facts Permanent, full audit trail
Entity + Procedural Shared Relationship graph + learned patterns Evolves continuously

The Model Router: Why One LLM Isn’t Enough

I used seven specialized Qwen models, split cleanly across the two paths. The biggest lesson: picking the right tool beats using one giant model for everything.

Serving Path Models (fast, frequent use)

  • qwen3-asr-flash: Audio transcription — handled medical terminology far better than expected
  • qwen3.5-omni-flash: Vision/image description — great for context, never trust it for exact numbers
  • text-embedding-v4: 1536-dim vectors for semantic search (8192 token support, rock solid)
  • qwen-flash: Real-time scorer — rates importance (0–1) and emotional valence (-1 to +1)
  • qwen-plus-latest: Workhorse — entity extraction, PDF parsing, final response generation
  • qwen-turbo: Binary relevance filter — keeps only retrieved memories that actually matter

Learning Path Model (slow, daily use)

  • qwen-max: Heavy summarization and fact extraction for the Dream Phase — accurate, so used sparingly.

The Bug That Hid For Three Days

Most of my debugging time went here: three silent failures, each masking the next.

The demo: A doctor uploads Ada’s blood work PDF showing her HbA1c is 8.2%. The system confidently replies "7.1%" — and doubles down when questioned.

1. Vision Hallucination

I first sent rendered PDF images to the vision model, which invented plausible-sounding but wrong numbers.

Fix: Extract raw text with pymupdf first, then send that text to a text-only model. Text models don’t make up numbers from pixels.

# Before
pages = _render_pdf_pages(file_content)
Enter fullscreen mode Exit fullscreen mode 
# After
pdf_text = _extract_pdf_text(file_content)
Enter fullscreen mode Exit fullscreen mode

2. Poisoned Cache

Fixed… but the system still said 7.1%. That first wrong answer had been stored in Working Memory as a "fact," and every query pulled the cached mistake instead of re-reading the file. I had to manually delete it:

DELETE FROM workingmemory WHERE message ILIKE '%HbA1c%7.1%';
Enter fullscreen mode Exit fullscreen mode

(This is exactly why Working Memory expires quickly — bad entries usually flush themselves, but this one got retrieved so often it kept extending its stay!)

3. Silent Truncation

Fixed again… still wrong. Finally, I checked the database: the model output correct values, but my ingestion pipeline cut every chunk at 500 characters. The HbA1c row sat at position 600, so it never made it to disk.

# The culprit
message=f"[{source_label}]: {para[:500]}",
Enter fullscreen mode Exit fullscreen mode 
# Now: full documents stored whole; chat messages still chunked
if source_type in ["pdf", "document"]:
    message=f"[{source_label}]: {para}"
else:
    message=f"[{source_label}]: {para[:500]}"
Enter fullscreen mode Exit fullscreen mode

Takeaway: Never trust just the chat response — check the source, model output, and database.

Deployment: Boring Is Beautiful

Everything runs on Alibaba Cloud, kept simple and maintainable:

  • ECS: Single Docker container (FastAPI + Nginx) for the Serving Path, deployed via GitHub Actions
  • ApsaraDB RDS: PostgreSQL + pgvector holds all memory tiers
  • OSS: Stores raw uploads from Ingest
  • EventBridge: Triggers the nightly Dream Phase cron, which scales to zero when done

One docker run command ties it all together; credentials live in GitHub Secrets. No Kubernetes, no over-engineering — just what works.

Hard-Won Lessons (So You Save Time)

  1. Check your dependencies: pymupdf once missing from requirements.txt crashed my deploy on startup.
  2. Vision describes, text transcribes: Need exact numbers from a document? Extract raw text first.
  3. One size doesn’t fit all: Chat messages and lab reports are different — don’t force the same chunk size on both.
  4. Test background jobs: I polished the real-time pipeline but barely tested the nightly cron. Always verify the parts that run when you’re asleep.

Àtúnbí. Reborn.

Built for the Qwen Cloud Hackathon. View the code on GitHub.

How do you approach long-term memory in LLM apps? What’s the trickiest bug you’ve chased? Let’s talk in the comments!

Top comments (0)