My Local AI Assistant Got Worse When I Remembered Too Much
I moved a personal AI assistant onto a small local model last week and immediately hit a boring problem: the model was fine, but my memory layer was not.
The old version persisted raw conversation history and replayed it back into the prompt. That worked well enough with hosted models. Then I pointed the same app at a local OpenAI-compatible server running Qwen3-4B-4bit through Swama on the Mac mini.
After 196 accumulated messages, the assistant started doing the classic small-model failure mode: parroting its own previous replies, over-weighting stale context, and sounding less useful the more “memory” I gave it.
The fix was not a vector database. It was deleting most of the memory.
I split memory into two different things:
- short-term conversation state
- long-term user facts
Short-term history now stays in RAM only. It resets after an idle gap, and it has a hard cap so a marathon session cannot poison every future turn.
self.session_memory: Dict[str, ConversationBufferMemory] = {}
self._last_activity: Dict[str, float] = {}
idle_limit = int(os.getenv("SESSION_IDLE_MINUTES", "120")) * 60
last = self._last_activity.get(user_id)
if last is not None and now - last > idle_limit:
del self.session_memory[user_id]
Long-term memory is not chat logs. It is a small list of distilled facts: preferences, people, devices, recurring activities, that kind of thing. Maximum 30 facts per user.
_FACT_EXTRACTION_PROMPT = """
Update the fact list. Add only stable, personal facts worth remembering across
conversations: preferences, interests, people, pets, places, devices, recurring
activities. Ignore small talk, one-off requests, and anything the assistant said
about itself.
Return ONLY a JSON array of strings.
"""
The fact extraction runs in a background thread after each exchange. The chat path should not wait for memory housekeeping.
threading.Thread(
target=self._extract_facts,
args=(user_id, msgs[-2].content, msgs[-1].content),
daemon=True,
).start()
I also deliberately use a hosted model for the distillation step. The local 4B model is good enough for fast interaction, but long-term memory cleanup is one of those places where quality matters more than latency. It is off the response path anyway.
The other local-model tweak was tool bias. Small models are much more likely to answer from stale weights even when tools exist, especially if the system prompt says anything like “use your knowledge first.” So the Swama handler adds a blunt override for live data:
_TOOL_BIAS = (
" IMPORTANT OVERRIDE: for anything happening NOW - weather, sea or"
" kitesurfing conditions, device/home status, prices, news, live data"
" of any kind - you MUST call the matching tool. Never answer those"
" from memory. /no_think"
)
Qwen3 also emits <think> blocks even when asked not to, including mid-stream after tool calls, so the streaming handler strips those tags incrementally. Not glamorous, but necessary if you do not want raw reasoning markup leaking into a voice/chat UI.
The useful lesson was this:
Memory is not “more previous tokens.”
For a personal assistant, raw transcript replay is the cheapest thing to build and one of the easiest ways to make the system worse. The assistant needs enough recent context to hold the current conversation, plus a tiny set of stable facts that survive across sessions.
Everything else is prompt pollution with a better name.
Source: Recent personal assistant backend work: Swama local model support, Qwen3-4B-4bit via an OpenAI-compatible endpoint, RAM-only session history, 2-hour idle reset, 200-message cap, background fact extraction, and 30 persisted user facts.
Tags: ai, python, llm, devops
Status: published
Top comments (3)
The two-layer split is the right move and it trades a loud failure for a silent one, which is worth flagging because it changes what you have to watch. Raw-history degradation announces itself: the model parrots, over-weights stale turns, you can see the pollution in the output. Distilled-fact degradation does not. A fact extracted wrong, or a relevant one never extracted, just quietly is not there, and the model proceeds confidently over the gap with nothing in the output flagging that something is missing. You moved from a failure you can see to one you cannot.
That is usually the right trade, but it relocates the risk onto the distiller, and the distiller is now an unaudited model deciding what matters, whose output a second model trusts as ground truth. Offloading extraction to the hosted model buys quality, but it is still one pass grading importance with no check on what it dropped. The instrument that closes it is provenance from each fact back to the turn it came from. Then a wrong fact is traceable and, more importantly, a query that should have hit a fact and missed becomes detectable, because you can go back to the source turn and see the fact was there and the distiller lost it. Without that link, the distilled layer is cleaner than raw history and fails in a place you have no way to look.
I like the distinction between "conversation history" and "durable memory." A lot of assistants treat memory as a larger context window, but they're solving two different problems. Recent context helps with coherence, while long-term memory should represent stable facts that survive across sessions.
One thing I'd add is that durable memory also benefits from confidence and expiry. Not every extracted fact deserves to live forever preferences change, devices get replaced, and assumptions become stale. Having promotion, decay, and periodic re-validation often improves quality more than simply increasing the memory budget.
Small local models especially seem to reward cleaner context over larger context.
Great read! Learned more about local AI and memory.