Your Bot's Memory File Is a Liar. Build a Nightly Auditor to Catch It

Yesterday my bot told me I had already submitted the assignment. I hadn't. The memory file said submitted with a timestamp from last week, and the bot repeated that line as if it were gospel. The bot wasn't deceiving me on purpose. It was faithfully reading a memory entry that had been overwritten by a hallucinated summary from an earlier conversation.

That incident pushed me to change how I think about memory in small AI projects. Memory is not a source of truth. It is a mutable cache that can be corrupted by the same model that writes it. Once I accepted that, I wanted a cheap, repeatable way to detect when that cache drifts from reality before the bot acts on it.

The Failure Mode

My setup is a tiny RAG assistant for course notes. Each chat session appends or updates entries in a memory.json file. The bot reads this file, injects the entries into the prompt, and answers. The file is the bot's only persistent state, so every wrong summary, every duplicated deadline, and every contradictory note becomes future context.

The critical flaw is trust. The bot never checks whether an existing entry matches a newer one. It just overwrites. One afternoon, a conversation about the reading list made the model decide I had finished Chapter 4. It wrote "chapter4": "completed, notes extracted". I hadn't touched that PDF. The next time I asked about the chapter, the bot confidently told me it was done.

The core problem is not hallucination. It is that a single wrong write permanently pollutes the memory file, and nothing ever challenges it.

Auditor Design

I decided to build a nightly auditor that would snapshot the memory file, compare it with yesterday's snapshot, and ask an LLM to classify each change. The output would be a markdown report showing which entries became contradiction, safe, or expansion.

For the experiment, I used MonkeyCode's free models and the free server option as the execution environment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The quotas and server availability can change, so verify current limits on their documentation before relying on them. The technique itself is model-agnostic and works with any OpenAI-compatible endpoint.

The auditor had three responsibilities: capture a hash of the current memory file, store the full previous state, and generate a diff of every changed entry. The hash tells me whether anything changed. The stored state tells me what changed. The LLM tells me whether that change looks healthy or dangerous.

The Script

Here is the complete script I ran. It requires Python 3.11 or newer, the requests library, and four environment variables: LLM_ENDPOINT, LLM_API_KEY, LLM_MODEL, and optionally MEMORY_PATH.

import hashlib
import json
import os
import requests
from datetime import datetime, timezone
from pathlib import Path

MEMORY_PATH = Path(os.getenv("MEMORY_PATH", "memory.json"))
SNAPSHOT_PATH = Path("snapshot.json")
API_URL = os.getenv("LLM_ENDPOINT")
API_KEY = os.getenv("LLM_API_KEY")
MODEL = os.getenv("LLM_MODEL")

def read_json(path: Path):
    with open(path) as f:
        return json.load(f)

def compute_hash(data) -> str:
    raw = json.dumps(data, sort_keys=True).encode("utf-8")
    return hashlib.sha256(raw).hexdigest()

def load_snapshot():
    if not SNAPSHOT_PATH.exists():
        return {"hash": None, "old_data": {}}
    return read_json(SNAPSHOT_PATH)

def diff_entries(old: dict, new: dict):
    added = {k: v for k, v in new.items() if k not in old}
    removed = {k: v for k, v in old.items() if k not in new}
    changed = {}
    for key in new:
        if key in old and old[key] != new[key]:
            changed[key] = {"before": old[key], "after": new[key]}
    return added, removed, changed

def classify_changes(changed: dict) -> str:
    if not changed:
        return "No changes found."
    lines = []
    for key, vals in changed.items():
        lines.append(f"- {key}: before={vals['before']!r} after={vals['after']!r}")
    prompt = [
        {"role": "system", "content": "You audit a chatbot's memory file. For each change, answer with JSON: {\"key\": {\"label\": \"safe|contradiction|expansion\", \"reason\": \"short reason\"}}."},
        {"role": "user", "content": "Changes to classify:\n" + "\n".join(lines)}
    ]
    resp = requests.post(
        API_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": MODEL, "messages": prompt},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

def audit():
    now = datetime.now(timezone.utc).isoformat()
    data = read_json(MEMORY_PATH)
    digest = compute_hash(data)
    snap = load_snapshot()
    if snap["hash"] == digest:
        print(f"{now} — no change")
        return
    added, removed, changed = diff_entries(snap["old_data"], data)
    report_lines = [f"# Audit {now}", f"hash: {digest}"]
    if changed:
        report_lines.append("## Classified Changes")
        report_lines.append(classify_changes(changed))
    if added:
        report_lines.append("## Added")
        report_lines.extend(f"- {k}: {v}" for k, v in added.items())
    if removed:
        report_lines.append("## Removed")
        report_lines.extend(f"- {k}" for k in removed)
    with open("audit_log.md", "a") as f:
        f.write("\n".join(report_lines) + "\n\n")
    with open(SNAPSHOT_PATH, "w") as f:
        json.dump({"hash": digest, "old_data": data}, f, indent=2)
    print(f"{now} — audit written")

if __name__ == "__main__":
    audit()

The crucial design choice is storing the entire previous memory JSON as old_data, not just a hash. A hash can only tell you that something changed. The old_data lets you compare actual values and feed them to a classifier. Without that, the auditor would be a blind alarm.

Scheduling on a Free Server

The free server stays online, which suits a nightly job. I used cron with this line:

0 3 * * * cd /path/to/auditor && /usr/bin/python3 audit_memory.py

If cron is not available, a simple loop works:

import time
while True:
    audit()
    time.sleep(86400)

Cron is better because it fails loudly when the job does not run. A Python loop that crashes will not retry, and you might not notice for days.

What the Logs Showed

After three nights, the auditor flagged a change I never expected. The memory contained two keys for the same assignment: assignment_status and submission_deadline. The bot had changed assignment_status from "draft" to "submitted" during a conversation where I was only asking about the deadline. The classifier marked it as contradiction with the reason: "The deadline is still in the future and the user never said 'submitted'." That single entry would have caused my bot to tell me the assignment was done when it was not.

The auditor also caught useful expansion changes, like adding a new key for a reading list. Those were harmless. The value of the auditor is not that it stops writes. It forces you to see what the model is writing into your permanent state.

Decision Table for Using This Approach

Situation	Use the auditor?	Why
Personal notes, course assistant	Yes	Cheap to run, catches obvious contradictions
Production customer data	No	Needs deterministic validation, not probabilistic classification
Memory file smaller than 100 KB	Yes	Diff fits easily in a prompt
Memory file in megabytes	Maybe	Sampling or batching required
Financial or medical decisions	No	You need formal verification, not an LLM opinion

The table reflects my own limits. I would not trust an LLM-based classifier for anything regulated.

Where This Falls Apart

The auditor only catches changes between snapshots. If the memory file is corrupted from the very first run, the baseline is already wrong. It cannot detect a false memory that never changes. Also, the classifier itself may mislabel a change. I saw one case where a deadline extension was called a contradiction, simply because the wording differed. The auditor is a smell detector, not a proof system.

Another limitation is operational. If the free model endpoint rate-limits your request or the server runs out of memory, the cron job will silently fail unless you read the logs. For this experiment that was acceptable. For a real assistant, I would add a webhook or a separate watchdog.

The Real Lesson

I stopped asking my bot to explain itself and started asking it to show its memory timestamps. A confident answer built on a corrupted memory entry is worse than a slow answer that admits uncertainty. The nightly audit gave me a low-effort way to notice when the memory file became the liar.

The next improvement is to add an expires_at field to every memory entry. Then the bot can be instructed to treat expired entries as untrusted. This is a small change, but it moves the system from blind trust to scheduled doubt.

Try the script on a test memory file with a few intentionally wrong entries. Watch what the classifier says. You will learn more about your bot's memory hygiene in one night than in a month of prompt engineering.