I Gave My AI a Memory. It Changed Everything.

I got tired of explaining my own codebase to an AI every single session.

"Here's the architecture. Here's the README. Here's what I tried last time and why it didn't work." Every. Single. Time.

Six months in, I realized I wasn't chatting with an assistant — I was onboarding a new intern who quit after every conversation and came back with total amnesia.

So I built a memory system. Local. Free. And honestly, it's the single most useful AI upgrade I've made.

The Problem Nobody Talks About

LLMs don't remember you. They don't remember your projects. They don't remember that you tried asyncio.gather last week and it deadlocked, or that your Garmin watch face has a hard 64KB memory limit, or that you always forget how your Telegram bot handles rate limits.

What they have is a context window — a temporary scratchpad that gets wiped when the conversation ends. For coding workflows, this is broken by design. You're not having a chat. You're doing ongoing work on a codebase that spans months.

I tried the obvious fixes:

• Pasting huge chunks of context every time. Burned through token limits, got truncated responses.
• Keeping "project briefs" in a text file and pasting them in. Worked until I had ten projects.
• Using cloud memory features. Worked until I hit another subscription and another privacy question.

None of it felt right. What I wanted was simple: ask a question about my stuff, get an answer based on my docs, with zero setup friction.

The Setup: RAG That Actually Runs Locally

RAG (Retrieval-Augmented Generation) isn't new. What's new is that you can run the entire stack — embedding model, vector database, and LLM — on a Mac Mini without touching a single cloud API.

Here's what I ended up with:

My Documents (markdown, code, notes, PDFs)
  → Chunked into ~400-token pieces
  → Embedded with nomic-embed-text (Ollama)
  → Stored in Chroma (local vector DB, no server needed)

Question
  → Embedded with the same model
  → Top-5 relevant chunks pulled from Chroma
  → Fed into Qwen 3.5 9B with a prompt template
  → Answer

Total stack: Ollama + Chroma + a 40-line Python script. No Docker. No cloud. No API keys.

What I Actually Indexed

Not the whole internet. Just the stuff I actually need:

• Project docs — READMEs, architecture decisions, API specs
• My notes — Obsidian vault, scratchpads, debugging logs
• Code snippets — Functions I keep reusing, config templates
• Bookmarked solutions — Stack Overflow answers I always forget
• Deployment notes — "How did I set up the VPS again?"

About 4,800 chunks total. Query time: under 2 seconds on the Mac Mini, under 500ms if I route it through the Windows PC's RTX 3060.

The Code (It's Embarrassingly Simple)

Indexing:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

loader = DirectoryLoader("~/projects/", glob="**/*.md", recursive=True)
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=50)
chunks = splitter.split_documents(docs)

embeddings = OllamaEmbeddings(model="nomic-embed-text")
db = Chroma.from_documents(chunks, embeddings, persist_directory="./knowledge-base")
db.persist()

Querying:

import ollama

def ask(question: str) -> str:
    results = db.similarity_search(question, k=5)
    context = "\n\n".join([r.page_content for r in results])

    response = ollama.chat(
        model="qwen3.5:9b",
        messages=[{
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {question}"
        }]
    )
    return response['message']['content']

That's it. No LangChain chains. No agents. No frameworks that require a PhD to configure.

What Changed

Before: "How does my Garmin watch face fetch stock data?" → paste 200 lines of code → wait → hope the model doesn't hallucinate.

After: ask("How does my Garmin watch face fetch stock data?") → "It uses Background.exit() to pass a dictionary with the ticker and price to the app-side view, because the background service has a 64KB memory limit." — in 1.8 seconds. Correct. Specific. Zero hallucination.

Before: "What's the rate limit for the crypto bot?" → dig through files → find nothing → guess.

After: "3 requests per second with a 1.5-second cooldown between calls, enforced in the throttle() decorator in bot/utils.py." — pulled straight from my own code comments.

The AI stopped being a generalist that needed onboarding. It became a specialist that already knew my stuff.

The Honest Downsides

Indexing is slow. Embedding 4,800 chunks on a Mac Mini CPU takes ~20 minutes. On the RTX 3060 it's ~4 minutes. You don't do it often — I run incremental updates via cron every hour — but the first run is painful.

Chunking is an art. Too small (100 tokens) and you lose context. Too big (1000 tokens) and you hit the retrieval limit too fast. I settled on 400 with 50-token overlap after way too much trial and error.

It doesn't replace search. If you ask "what's the syntax for Python list comprehension," RAG is overkill and probably retrieves some random list-handling function from your old projects. This is for your knowledge, not general knowledge.

Storage grows. 4,800 chunks ≈ 300MB of vector data. Manageable, but not nothing. Chroma handles it fine locally.

One Trick That Made It Actually Useful

I added incremental updates with file hashes instead of re-indexing everything:

def update_index(docs_dir, db, cache_file="./index-cache.json"):
    cache = json.loads(Path(cache_file).read_text()) if Path(cache_file).exists() else {}
    changed = []

    for f in Path(docs_dir).rglob("*.md"):
        h = hashlib.md5(f.read_bytes()).hexdigest()
        if cache.get(str(f)) != h:
            changed.append(str(f))
            cache[str(f)] = h

    if changed:
        # ... re-index only changed files
        Path(cache_file).write_text(json.dumps(cache))

Runs every hour via cron. Most of the time it does nothing. When I edit a file, it catches it. My knowledge base is never more than an hour stale.

When This Makes Sense

Do this if:

• You work on multiple long-term projects
• You keep notes/docs/code that you reference repeatedly
• You want context-aware answers without pasting walls of text
• You have a machine that can run Ollama (Mac Mini M4, any modern PC)

Don't do this if:

• You only do one-off queries ("what's the weather")
• Your projects are small and you remember everything anyway
• You can't spare 300MB of disk space
• You expect it to replace Google (it won't)

Getting Started (15 Minutes)

# 1. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull models
ollama pull qwen3.5:9b
ollama pull nomic-embed-text

# 3. Python deps
pip install chromadb langchain ollama

# 4. Index your docs (adapt the script above)
# 5. Ask questions

Total time: 15 minutes. Total cost: $0. Monthly cost: $0.

The Real Win

The technical setup is neat. But the real change is mental.

Before, every AI session started with context-building: "Here's what I'm working on, here's what I tried, here's the constraints." Now I just ask the question. The system already knows the constraints because it read my docs.

It's the difference between calling a consultant and calling a teammate who was in the meeting.

That gap — the gap between "AI that answers questions" and "AI that knows your work" — is bigger than I expected. And closing it costs exactly zero dollars.

---

Sam Hartley is a solo dev building local AI tools on a 3-machine home lab. Writes about the infrastructure that makes AI actually useful for real work.

→ Custom automation setups on Fiverr
→ Follow CelebiBots on Telegram

ai #rag #ollama #selfhosted #productivity #buildinginpublic