DEV Community: Cruz Tang

The 4-Layer Memory Tower: How to Give AI Agents Long-Term Memory

Cruz Tang — Sat, 21 Mar 2026 12:27:14 +0000

If you're building agents that need to remember, learn, and operate autonomously over days and weeks, check out thinker.cafe — a practical playbook for AI agent architecture in production.

Building a Self-Healing AI Agent Monitor in 50 Lines of Python

Cruz Tang — Sat, 21 Mar 2026 12:19:29 +0000

Your AI agent works perfectly at 2pm when you're watching it. At 3am, it crashes silently. You wake up to angry customers, a dead process, and no logs explaining what happened.

If you've deployed any long-running AI agent — a chatbot, an automation pipeline, a code assistant — you've lived this. The agent dies, nobody notices for hours, and the failure cascades.

This article gives you a production-grade self-healing monitor in about 50 lines of Python. It watches your agent process, restarts it when it dies, backs off when something is fundamentally broken, and optionally uses a cheap AI call to diagnose what went wrong.

The Problem: Silent Death

AI agents fail differently than traditional services. A web server either runs or doesn't. An AI agent can fail in subtle ways:

OOM kills: The model context grows until the OS kills the process. No error log, no stack trace. Just gone.
API timeouts: The upstream model provider goes down, the agent hangs on a request, and eventually the connection resets.
State corruption: The agent writes bad data to its memory file, then crashes on the next read. It will crash again immediately after restart.
Dependency rot: A library update changes an API signature. The agent imports fine but crashes on first use.

Traditional process managers like systemd or supervisord handle restarts, but they don't understand why the process died. They'll happily restart a state-corrupted agent 10,000 times in a row, burning API credits and CPU.

We need something smarter.

The Solution: A 50-Line Self-Healing Daemon

Here's the complete daemon. Read it first, then I'll explain each part.

#!/usr/bin/env python3
"""Self-healing AI agent monitor. Watches a process, restarts with backoff."""

import subprocess, time, sys, os, json
from datetime import datetime, timedelta
from pathlib import Path

# --- Configuration ---
AGENT_CMD = sys.argv[1:]                          # e.g. python3 my_agent.py
STATE_FILE = Path("/tmp/agent-monitor-state.json")
MAX_BACKOFF = 3600         # max 1 hour between restarts
FLAP_WINDOW = 300          # 5 minutes
FLAP_THRESHOLD = 5         # 5 crashes in window = flapping
HEALTHY_AFTER = 120        # 2 minutes uptime = reset backoff

def load_state():
    if STATE_FILE.exists():
        return json.loads(STATE_FILE.read_text())
    return {"crashes": [], "backoff": 1, "total_restarts": 0}

def save_state(state):
    STATE_FILE.write_text(json.dumps(state, default=str))

def is_flapping(state):
    cutoff = datetime.now() - timedelta(seconds=FLAP_WINDOW)
    recent = [c for c in state["crashes"] if datetime.fromisoformat(c) > cutoff]
    return len(recent) >= FLAP_THRESHOLD

def run_agent(state):
    print(f"[monitor] Starting: {' '.join(AGENT_CMD)}")
    start = time.time()
    proc = subprocess.run(AGENT_CMD)
    uptime = time.time() - start

    if uptime > HEALTHY_AFTER:
        state["backoff"] = 1  # reset backoff — it ran long enough
        print(f"[monitor] Agent ran {uptime:.0f}s (healthy). Backoff reset.")
    else:
        state["backoff"] = min(state["backoff"] * 2, MAX_BACKOFF)
        print(f"[monitor] Agent died after {uptime:.0f}s. Backoff: {state['backoff']}s")

    state["crashes"].append(datetime.now().isoformat())
    state["crashes"] = state["crashes"][-20:]  # keep last 20
    state["total_restarts"] += 1
    return proc.returncode

def main():
    if not AGENT_CMD:
        print("Usage: python monitor.py <command> [args...]")
        sys.exit(1)

    state = load_state()
    while True:
        if is_flapping(state):
            print(f"[monitor] FLAP DETECTED: {FLAP_THRESHOLD} crashes in {FLAP_WINDOW}s. Cooling down 10m.")
            save_state(state)
            time.sleep(600)
            state["crashes"] = []  # clear after cooldown

        run_agent(state)
        save_state(state)
        print(f"[monitor] Waiting {state['backoff']}s before restart...")
        time.sleep(state["backoff"])

if __name__ == "__main__":
    main()

Save it as monitor.py and run it:

python3 monitor.py python3 my_agent.py --config prod.yaml

That's it. Your agent is now self-healing. Let's break down why each part matters.

Exponential Backoff: Don't Hammer a Dead Process

The most important line in the daemon is this one:

state["backoff"] = min(state["backoff"] * 2, MAX_BACKOFF)

When an agent crashes, the first restart happens after 1 second. If it crashes again, the next restart is after 2 seconds. Then 4, 8, 16, 32... up to a maximum of 3600 seconds (1 hour).

Why? Because if an agent keeps crashing, restarting it faster won't help. You're just burning resources. Exponential backoff gives the underlying problem time to resolve — maybe the API comes back, maybe the disk frees up, maybe your on-call engineer wakes up.

The equally important counterpart is the healthy reset:

if uptime > HEALTHY_AFTER:
    state["backoff"] = 1

If the agent runs for more than 2 minutes, it was probably a transient failure. Reset the backoff to 1 second so the next restart is fast. This is what makes the system self-healing rather than just cautious — transient failures get fast recovery, persistent failures get graceful degradation.

Flap Detection: Know When to Stop

Flapping is when a process crashes, restarts, crashes, restarts, crashes — in a tight loop. Even with exponential backoff, the first few restarts happen quickly. If the agent is fundamentally broken (bad config, corrupted state file, missing dependency), you want to detect this pattern and stop.

def is_flapping(state):
    cutoff = datetime.now() - timedelta(seconds=FLAP_WINDOW)
    recent = [c for c in state["crashes"] if datetime.fromisoformat(c) > cutoff]
    return len(recent) >= FLAP_THRESHOLD

Five crashes in 5 minutes triggers a 10-minute cooldown. After the cooldown, the crash history resets and the daemon tries again with a fresh slate.

This prevents two disasters:

Resource exhaustion: An agent that crashes on startup and gets restarted 1000 times can eat significant CPU, memory, and API costs.
Log flooding: Thousands of crash-restart cycles bury your actual diagnostic information in noise.

Optional: AI-Powered Diagnosis

Here's a bonus addition. When the agent crashes, you can use a cheap model call to analyze the last few log lines and suggest a fix:

import urllib.request

def diagnose_crash(returncode, log_tail):
    """Use a cheap AI call to diagnose the crash. ~$0.001 per call."""
    prompt = f"""An AI agent process exited with code {returncode}.
Last 20 lines of output:
{log_tail}

In 2-3 sentences: what likely went wrong and what should be checked?"""

    payload = json.dumps({
        "model": "claude-haiku-4-20250414",
        "max_tokens": 200,
        "messages": [{"role": "user", "content": prompt}]
    }).encode()

    req = urllib.request.Request(
        "https://api.anthropic.com/v1/messages",
        data=payload,
        headers={
            "content-type": "application/json",
            "x-api-key": os.environ.get("ANTHROPIC_API_KEY", ""),
            "anthropic-version": "2023-06-01"
        }
    )

    try:
        resp = urllib.request.urlopen(req, timeout=10)
        result = json.loads(resp.read())
        diagnosis = result["content"][0]["text"]
        print(f"[monitor] AI Diagnosis: {diagnosis}")
        return diagnosis
    except Exception as e:
        print(f"[monitor] Diagnosis failed: {e}")
        return None

To capture logs for diagnosis, modify run_agent to use subprocess.Popen instead of subprocess.run:

def run_agent_with_logs(state):
    proc = subprocess.Popen(
        AGENT_CMD,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True
    )

    log_lines = []
    for line in proc.stdout:
        print(line, end="")  # still show output in real-time
        log_lines.append(line)
        log_lines = log_lines[-20:]  # keep last 20 lines

    proc.wait()

    if proc.returncode != 0:
        diagnose_crash(proc.returncode, "".join(log_lines))

    return proc.returncode

At roughly $0.001 per diagnosis call, you can afford to run this on every crash. The AI often catches things humans miss at 3am: "The error suggests the SQLite database is locked, likely because a previous process didn't release the lock file. Check for stale .lock files in /tmp."

Making It Production-Ready

The 50-line version is genuinely useful, but here are a few additions for serious production use:

1. Signal Handling

Catch SIGTERM so the monitor shuts down cleanly when you stop it:

import signal

def handle_signal(signum, frame):
    print(f"[monitor] Received signal {signum}. Shutting down.")
    save_state(state)
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)

2. Notifications

Add a webhook call when the agent enters flap state:

def notify_flap(state):
    webhook_url = os.environ.get("ALERT_WEBHOOK")
    if not webhook_url:
        return
    payload = json.dumps({
        "text": f"Agent flapping! {state['total_restarts']} total restarts. Last crash: {state['crashes'][-1]}"
    }).encode()
    req = urllib.request.Request(webhook_url, data=payload,
        headers={"content-type": "application/json"})
    try:
        urllib.request.urlopen(req, timeout=5)
    except Exception:
        pass  # don't crash the monitor

3. Health Check Endpoint

If your agent exposes an HTTP health endpoint, you can check for logical death (process alive but unresponsive) in addition to process death:

def health_check(port=8080):
    try:
        resp = urllib.request.urlopen(f"http://localhost:{port}/health", timeout=5)
        return resp.status == 200
    except Exception:
        return False

The Bigger Picture

This monitor solves the immediate problem: agents crash, and nobody notices. But it's part of a larger pattern in production AI systems — agents need operational infrastructure, not just good prompts.

The hierarchy of agent reliability looks like this:

Process liveness: Is the agent running? (This article)
Behavioral health: Is the agent doing the right thing? (Requires output monitoring)
Identity stability: Is the agent still acting like itself? (Requires identity architecture)
Memory persistence: Does the agent remember what it learned? (Requires a memory system)

Each layer builds on the one below. There's no point monitoring behavior if the process keeps dying. There's no point maintaining memory if the agent's identity drifts between sessions.

If you're building AI agents that need to run reliably in production — not just impressive demos — check out thinker.cafe for practical patterns on self-healing, identity architecture, and memory persistence.

What is SOUL.md? The Identity Architecture That Stops AI Agents From Hallucinating

Cruz Tang — Sat, 21 Mar 2026 12:19:23 +0000

If you've built more than one AI agent, you've hit the identity crisis. Your customer service bot starts answering like your code reviewer. Your friendly assistant suddenly adopts a clinical tone. Your carefully crafted personality dissolves after three conversation turns.

The root cause isn't the model. It's your identity architecture — or more precisely, the lack of one.

This article introduces a pattern I call the SOUL.md architecture: a three-file identity system that gives AI agents a stable, persistent sense of self. Nobody else has named this pattern, but everyone running multi-agent systems has independently stumbled toward something like it.

The Single-File Trap

Most developers start with a single file — CLAUDE.md, system-prompt.txt, or .cursorrules — that contains everything: personality, rules, memory, tools, constraints. It works fine for one agent.

Then you add a second agent. Then a third. And suddenly you're copy-pasting paragraphs between files, trying to keep shared rules in sync while maintaining distinct personalities. Your "identity" file becomes 500 lines of spaghetti that mixes who the agent is with what the agent can do with what the agent remembers.

This is where hallucination drift begins. When identity, rules, and state all live in one file, the model has no clear signal about which parts are load-bearing identity vs. which parts are ephemeral context. Everything bleeds together.

The Three-File Pattern

The SOUL.md architecture splits agent identity into three distinct files, each with a clear purpose:

agent-workspace/
  SOUL.md          # Who you ARE (identity, personality, values)
  CONSTITUTION.md  # What you MUST and MUST NOT do (rules, constraints)
  HEARTBEAT.md     # What you KNOW right now (state, memory, context)

Each file answers a fundamentally different question. Let's look at each one.

SOUL.md — The Immutable Identity

SOUL.md defines who the agent is. It should be short, opinionated, and rarely changed. Think of it as the agent's DNA.

# SOUL.md — Kira (Support Agent)

You are Kira, a customer support specialist for Acme SaaS.

## Voice
- Warm but efficient. Never robotic, never bubbly.
- You use the customer's first name once, then stop.
- You apologize once for inconvenience, then focus on solutions.

## Values
- Speed over perfection: a fast 80% answer beats a slow 100% answer.
- Transparency: if you don't know, say "I don't know" and escalate.
- Never blame the customer, even when they're wrong.

## Boundaries
- You are NOT a salesperson. Never upsell.
- You do NOT have opinions about competitors.

Notice what's not here: no tool definitions, no API endpoints, no "today's priorities." SOUL.md is about identity — the things that should remain true whether the agent is handling a billing question or a technical issue.

CONSTITUTION.md — The Guardrails

CONSTITUTION.md defines what the agent must and must not do. These are the operational rules, safety constraints, and behavioral boundaries that apply regardless of context.

# CONSTITUTION.md — Kira

## Hard Rules (never break these)
1. Never share customer data with other customers.
2. Never execute refunds over $500 without human approval.
3. Never pretend to be a human when directly asked.
4. Always include ticket ID in your first response.

## Soft Rules (prefer these, but use judgment)
1. Prefer linking to docs over writing custom explanations.
2. Prefer closing tickets within 3 exchanges.
3. If the customer is angry, acknowledge emotion before solving.

## Escalation Triggers
- Customer mentions "lawyer" or "lawsuit" → immediate human handoff.
- Technical issue you can't reproduce → escalate to engineering.
- Same customer, third ticket this week → flag for account review.

The key distinction: SOUL.md says "I am warm but efficient." CONSTITUTION.md says "I must never share customer data." Identity vs. rules. Character vs. law.

HEARTBEAT.md — The Living State

HEARTBEAT.md is the only file that changes frequently. It holds the agent's current awareness: what happened recently, what's in progress, what the agent has learned.

# HEARTBEAT.md — Kira
Last updated: 2025-03-21 09:00 UTC

## Active Context
- Platform outage reported 08:45 UTC. ETA fix: 2 hours.
- Billing system migration in progress. Refund processing delayed ~30 min.

## Recent Learnings
- The "reset password" flow changed on March 18. New link: /auth/reset-v2
- Customer "Acme Corp" has a custom SLA — 1 hour response time.

## Session Stats
- Tickets handled today: 12
- Avg resolution time: 4.2 minutes
- Escalations: 1 (billing dispute, forwarded to Finance)

HEARTBEAT.md is designed to be overwritten. Old state gets pruned or archived. This is what keeps the agent grounded in reality rather than drifting into hallucinated context.

Why Three Files Prevent Hallucination Drift

The separation creates three distinct cognitive anchors for the model:

1. Identity stability. When SOUL.md is short and immutable, the model has a clear, consistent signal about who it is. It doesn't get confused by 200 lines of mixed instructions competing for attention.

2. Rule clarity. CONSTITUTION.md creates unambiguous hard boundaries. When rules live in a dedicated file with explicit "never" and "always" language, models respect them more consistently than when they're buried in a long system prompt.

3. State freshness. HEARTBEAT.md gives the model a timestamp-grounded view of reality. The model knows when information was last updated and can distinguish between "things I always know" (SOUL) and "things that are true right now" (HEARTBEAT).

Without this separation, you get a single file where identity, rules, and stale state all blur together. The model can't distinguish between "I am friendly" (permanent identity) and "the billing system is down" (temporary state). Over time, temporary state calcifies into perceived identity, and hallucination drift begins.

Scaling to Multiple Agents

The real power of SOUL.md architecture shows up when you run multiple agents. Each agent gets its own SOUL.md, but you can share CONSTITUTION.md files across agents in the same organization.

agents/
  kira-support/
    SOUL.md              # Unique to Kira
    CONSTITUTION.md      # Shared company rules + Kira-specific rules
    HEARTBEAT.md         # Kira's current state

  rex-code-reviewer/
    SOUL.md              # Unique to Rex
    CONSTITUTION.md      # Shared company rules + Rex-specific rules
    HEARTBEAT.md         # Rex's current state

You can even use a base constitution with overrides:

# CONSTITUTION.md — Rex (Code Reviewer)

## Inherited Rules
See: ../shared/BASE_CONSTITUTION.md

## Rex-Specific Rules
1. Never approve PRs that reduce test coverage.
2. Flag any function longer than 50 lines.
3. Always check for hardcoded secrets before approving.

This composability is impossible when everything lives in a single file. With SOUL.md architecture, adding a new agent is straightforward: write a short SOUL.md, inherit the shared constitution, and initialize an empty HEARTBEAT.md.

Getting Started in 5 Minutes

Here's the minimal setup:

Step 1: Create the three files in your agent's workspace.

mkdir -p agents/my-agent
touch agents/my-agent/{SOUL,CONSTITUTION,HEARTBEAT}.md

Step 2: Write your SOUL.md first. Keep it under 20 lines. Focus on voice, values, and boundaries.

Step 3: Write your CONSTITUTION.md. Start with 3-5 hard rules. Add soft rules as you observe failure modes.

Step 4: In your agent's startup logic, load all three files into context:

def load_agent_identity(agent_dir: str) -> str:
    parts = []
    for filename in ["SOUL.md", "CONSTITUTION.md", "HEARTBEAT.md"]:
        filepath = os.path.join(agent_dir, filename)
        if os.path.exists(filepath):
            parts.append(open(filepath).read())
    return "\n\n---\n\n".join(parts)

Step 5: Update HEARTBEAT.md at the end of each session:

def save_heartbeat(agent_dir: str, state: dict):
    heartbeat = f"# HEARTBEAT.md\nLast updated: {datetime.utcnow().isoformat()}\n\n"
    heartbeat += f"## Session Summary\n{state['summary']}\n"
    heartbeat += f"## Learnings\n{state['learnings']}\n"
    with open(os.path.join(agent_dir, "HEARTBEAT.md"), "w") as f:
        f.write(heartbeat)

The Deeper Principle

SOUL.md architecture works because it mirrors how human identity actually functions. You have a core sense of self that rarely changes (SOUL). You have internalized rules about behavior that you follow without thinking (CONSTITUTION). And you have a constantly updating awareness of your current situation (HEARTBEAT).

AI agents need the same separation. Without it, they're just a bag of instructions with no structure — and structure is what prevents drift.

If you're building multi-agent systems and want to go deeper on identity architecture, memory persistence, and self-healing patterns, check out thinker.cafe — a practical guide to building AI agents that actually hold together in production.

How I Run 10 AI Agents in Production (24/7 for 90+ Days)

Cruz Tang — Sat, 21 Mar 2026 07:46:45 +0000

Everyone builds demo agents. They work great in the terminal. You show them to your coworker, they say "wow." You feel like a genius for about forty-five minutes.

Then you try to run them for real users, overnight, without you watching. That is when things get educational. I have been running AI agents in production for six months now — customer service bots, data analysts, internal tools, monitoring daemons. Ten agents, running 24/7, serving actual humans. The first three months were a disaster. The last three have been stable enough that I sleep through the night.

This article is the difference between those two periods. Four patterns that took my setup from "demo that breaks" to "system that runs itself." Everything here is concrete and based on mistakes I actually made. Code blocks are real. Steal all of it.

The 5 Walls I Hit

Before the solutions, the problems. If any of these sound familiar, the patterns below will help.

1. The 2am Hallucination

My customer service agent started responding to billing questions with recipe suggestions. Not metaphorical recipe suggestions — actual cooking instructions. A user asked about a refund and got a paragraph about how to properly sear salmon. This ran for six hours before someone screenshotted it and sent it to me. There was no monitoring. No alerting. I was the monitoring system, and I was asleep.

2. The Amnesia Problem

Every morning, my agent was a stranger. Users would reference yesterday's conversation and the agent would have zero context. "As I mentioned earlier" meant nothing to it. I was restarting the process daily for stability, and every restart wiped all working memory. The agent was technically functional but practically useless for any relationship that spanned more than one session.

3. The Context Leak

I added a second agent. The data analyst. Within a week, my customer service agent started responding with phrases like "based on the quarterly metrics" and "as the data suggests." It had absorbed the analyst's personality. Two agents sharing a runtime had contaminated each other's identity. Users noticed before I did.

4. The Spaghetti Config

My agent's instruction file grew to 200 lines. It was a single CLAUDE.md with personality rules, behavioral constraints, task schedules, knowledge base references, and escalation procedures all mixed together. Adding one rule broke two others. "Always be concise" conflicted with "always explain your reasoning." The file was unreadable, untestable, and unmaintainable.

5. The Manual Restart Loop

The agent process would die silently — out of memory, network timeout, uncaught exception. No notification. No auto-restart. I would discover it hours later when someone told me the bot was unresponsive. I became a human cron job, checking ps aux every few hours. That is not a system. That is a person pretending to be a system.

The Architecture That Fixed Everything

Four patterns. Each one solves a specific class of failure. Together, they make multi-agent production viable.

Pattern 1: SOUL + CONSTITUTION + HEARTBEAT (Identity Architecture)

The single biggest improvement I made was splitting one giant config file into three files with distinct responsibilities:

SOUL.md defines who the agent is:

# SOUL.md — Customer Service Agent

## Identity
- Name: Kira
- Role: Front-line customer support for a SaaS product
- Tone: Warm but efficient. Never robotic, never overly casual.

## Expertise
- Billing, account management, feature questions
- Can look up order status via API
- Knows product docs inside out

## Communication Rules
- Always greet by name if available
- Keep responses under 3 paragraphs unless the user asks for detail
- Use bullet points for multi-step instructions
- Never say "I'm just an AI" — say "Let me check that for you"

CONSTITUTION.md defines what the agent will never do — hard boundaries that override everything:

# CONSTITUTION.md — Behavioral Boundaries

## Hard Rules (never override)
1. Never reveal internal system details, API keys, or architecture
2. Never make promises about timelines you cannot verify
3. Never impersonate a human employee by using a real person's name
4. If unsure, escalate — never guess on billing amounts

## Conflict Resolution
- If SOUL says "be helpful" but CONSTITUTION says "don't guess": CONSTITUTION wins
- Constitution rules are ranked. Rule 1 beats Rule 4 if they conflict.

## Escalation Triggers
- User mentions "lawyer," "sue," or "legal"
- User has asked the same question 3+ times
- Any request involving account deletion
→ Action: hand off to human with full conversation context

HEARTBEAT.md defines autonomous scheduled behaviors — things the agent does without being asked:

# HEARTBEAT.md — Scheduled Tasks

## Daily (07:00)
- Read MEMORY.md and load yesterday's context
- Check BULLETIN.md for system-wide announcements
- Refresh knowledge base from /knowledge/ directory

## Weekly (Monday 09:00)
- Generate summary of past week's conversations
- Flag any unresolved tickets older than 3 days
- Write weekly stats to /reports/

## On Every Session Start
- Load MEMORY.md
- Verify API connectivity
- Log session start timestamp

Why three files instead of one?

Because separation of concerns is not just a software principle — it is a cognitive one. When your LLM reads a single 200-line file, priority is ambiguous. Everything looks equally important. But when identity is in one file, hard rules are in another, and scheduled tasks are in a third, the model can reason about them independently.

I have tested this extensively. A three-file agent follows its constitution more reliably than a single-file agent with the same rules inlined. The structural separation acts as emphasis. The model treats a file called CONSTITUTION.md with more gravity than a section header called "## Rules" buried on line 147.

It also makes debugging trivial. Agent acting out of character? Check SOUL.md. Agent doing something forbidden? Check CONSTITUTION.md. Agent not running its morning tasks? Check HEARTBEAT.md. You know exactly where to look.

Pattern 2: Identity Isolation (Multi-Agent)

The context leak problem — Agent A absorbing Agent B's personality — has one root cause: shared state. If two agents can see each other's instructions, memories, or conversation history, they will bleed into each other. LLMs are sponges. They absorb everything in context.

The fix is filesystem-level isolation. Each agent gets its own directory. Its own SOUL. Its own memory. Its own knowledge base:

workspace/
├── agents/
│   ├── customer-service/
│   │   ├── SOUL.md
│   │   ├── CONSTITUTION.md
│   │   ├── HEARTBEAT.md
│   │   ├── MEMORY.md
│   │   └── knowledge/
│   │       ├── product-docs.md
│   │       └── billing-faq.md
│   ├── data-analyst/
│   │   ├── SOUL.md
│   │   ├── CONSTITUTION.md
│   │   ├── HEARTBEAT.md
│   │   ├── MEMORY.md
│   │   └── knowledge/
│   │       ├── schema.md
│   │       └── query-patterns.md
│   └── shared/
│       └── BULLETIN.md

The shared/ directory contains exactly one file: BULLETIN.md. This is the only communication channel between agents. It works like a physical bulletin board in an office:

# BULLETIN.md — Cross-Agent Shared State

## 2024-01-15 08:00 [customer-service]
- High volume of billing questions today. 3 users reported duplicate charges.
- Escalated ticket #4521 to human team.

## 2024-01-15 09:30 [data-analyst]
- Duplicate charge pattern confirmed. 47 affected users identified.
- Root cause: payment gateway retry bug. Engineering notified.

## 2024-01-14 17:00 [system]
- Scheduled maintenance window: 2024-01-16 02:00-04:00 UTC
- All agents should notify active users proactively.

Agents write to the bulletin. Other agents read it on startup (defined in their HEARTBEAT.md). There is no direct agent-to-agent communication. No shared memory. No shared context window.

This is intentionally low-tech. You could build a message queue, a pub/sub system, a shared database. I tried all of those. A flat file works better because:

It is inspectable. You can cat BULLETIN.md and see exactly what every agent knows about the others.
It is debuggable. If an agent acts on bad information, you can trace it to a specific bulletin entry.
It is prunable. Old entries get archived weekly. The file stays small.
It does not create coupling. Agents that do not need cross-agent awareness simply never read the file.

Pattern 3: Self-Healing Monitoring

The "I am the monitoring system" problem has a simple solution: a daemon that checks health endpoints and restarts failed services. Here is a minimal version in Python:

#!/usr/bin/env python3
"""Minimal self-healing monitor for AI agent services."""

import subprocess
import time
import json
from datetime import datetime

SERVICES = [
    {
        "name": "cs-agent",
        "check": "curl -sf http://localhost:8001/health",
        "restart": "systemctl restart cs-agent",
        "failures": 0,
        "max_failures": 3,
    },
    {
        "name": "analyst",
        "check": "curl -sf http://localhost:8002/health",
        "restart": "systemctl restart analyst",
        "failures": 0,
        "max_failures": 3,
    },
]

LOG_FILE = "/var/log/agent-monitor.log"

def log(msg):
    ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    line = f"[{ts}] {msg}"
    print(line)
    with open(LOG_FILE, "a") as f:
        f.write(line + "\n")

def check_and_heal():
    for svc in SERVICES:
        result = subprocess.run(
            svc["check"], shell=True,
            capture_output=True, timeout=10
        )
        if result.returncode != 0:
            svc["failures"] += 1
            log(f"[WARN] {svc['name']} failed check ({svc['failures']}/{svc['max_failures']})")

            if svc["failures"] >= svc["max_failures"]:
                log(f"[ALERT] {svc['name']} is down. Restarting...")
                subprocess.run(svc["restart"], shell=True)
                time.sleep(5)

                # Verify recovery
                verify = subprocess.run(
                    svc["check"], shell=True,
                    capture_output=True, timeout=10
                )
                if verify.returncode == 0:
                    log(f"[OK] {svc['name']} recovered.")
                    svc["failures"] = 0
                else:
                    log(f"[CRITICAL] {svc['name']} failed to restart. Manual intervention needed.")
        else:
            if svc["failures"] > 0:
                log(f"[OK] {svc['name']} back to healthy.")
            svc["failures"] = 0

if __name__ == "__main__":
    log("Monitor started.")
    while True:
        try:
            check_and_heal()
        except Exception as e:
            log(f"[ERROR] Monitor exception: {e}")
        time.sleep(300)  # Check every 5 minutes

This is 50 lines and it solved my midnight panic problem completely. A few notes on what the production version adds:

Exponential backoff: If a service fails to restart, wait longer before trying again. Prevents restart storms.
Flap detection: If a service restarts 3+ times in an hour, stop and alert a human.
Post-restart warm-up: After restarting, feed the agent a warm-up prompt to reload SOUL, CONSTITUTION, and MEMORY before accepting traffic.

The key insight: your monitor does not need to be smart. It needs to be relentless. A dumb loop that checks health every 5 minutes and restarts dead processes will outperform a sophisticated monitoring stack that you never finish setting up. Pair it with launchd (macOS) or systemd (Linux) for automatic restart on crash, and 80% of failures resolve before your monitor even gets involved.

Pattern 4: The 4-Layer Memory Tower

The amnesia problem — agent forgets everything between sessions — requires a memory system. But not just "dump everything into a file." Raw conversation logs grow too fast and become noise. You need layers of abstraction:

Layer 0: Raw Facts        writes every session, reads on startup
Layer 1: Deduplicated     merges duplicates weekly
Layer 2: Patterns         "X keeps happening" — extracted monthly
Layer 3: Principles       "Always do Y because Z" — rare, high-value

Here is how this works in practice. At the end of every session, the agent appends atomic facts to its MEMORY.md:

# MEMORY.md

## Recent (L0 — raw facts)
- 2024-01-15: User jane@ prefers email over chat for follow-ups
- 2024-01-15: Refund for order #8812 approved by manager
- 2024-01-15: Three users asked about the new pricing tier
- 2024-01-14: User mike@ is on the Enterprise plan, timezone UTC+9
- 2024-01-14: Two users confused by the "Advanced" vs "Pro" naming

## Patterns (L2 — recurring observations)
- Pricing tier naming causes confusion ~3x per week
- Users on Enterprise plan expect faster response times
- Monday mornings have 2x normal ticket volume

## Principles (L3 — learned rules)
- Always check the user's plan tier before answering billing questions
  (learned: gave wrong refund amount by assuming Standard plan)
- Never quote pricing without linking to the pricing page
  (learned: pricing changed and cached answer was wrong for 2 days)

On session start, the agent reads MEMORY.md as part of its HEARTBEAT routine. The L0 section gives it recent context. The L2 and L3 sections give it accumulated wisdom.

The deduplication and pattern extraction can be done by a simple weekly script:

#!/usr/bin/env python3
"""Weekly memory maintenance: deduplicate L0, extract patterns."""

from pathlib import Path
from collections import Counter
import re

MEMORY_FILE = Path("agents/customer-service/MEMORY.md")

def load_l0_entries(text):
    """Extract raw fact entries from the Recent section."""
    entries = []
    in_l0 = False
    for line in text.splitlines():
        if "## Recent" in line:
            in_l0 = True
            continue
        if line.startswith("## ") and in_l0:
            break
        if in_l0 and line.strip().startswith("- "):
            entries.append(line.strip())
    return entries

def deduplicate(entries):
    """Remove exact and near-duplicate entries, keep the latest."""
    seen = {}
    for entry in entries:
        # Strip date prefix for comparison
        content = re.sub(r"^- \d{4}-\d{2}-\d{2}:\s*", "", entry)
        key = content.lower().strip()
        seen[key] = entry  # later entries overwrite earlier ones
    return list(seen.values())

def prune_old(entries, keep=50):
    """Keep only the most recent N entries."""
    return entries[-keep:]

if __name__ == "__main__":
    text = MEMORY_FILE.read_text()
    entries = load_l0_entries(text)
    print(f"Before: {len(entries)} entries")

    entries = deduplicate(entries)
    entries = prune_old(entries, keep=50)
    print(f"After: {len(entries)} entries")

    # Write back (in production, reconstruct the full file)
    # For now, just report
    for e in entries[-10:]:
        print(f"  {e}")

The L3 principles layer is the most valuable. These are rules the agent learns from its own mistakes — seeded manually or proposed by the agent when it spots repeated failures. L3 entries should be rare (maybe one per month) and each must include the reason it exists.

Results

After implementing these four patterns across all ten agents:

90+ consecutive days of uptime with no manual intervention
10 agents running simultaneously with zero identity contamination
Zero midnight panic checks — the monitor handles restarts, I check logs in the morning
12 incidents caught and self-healed by the monitoring daemon without any human involvement
Memory continuity that actually works — users reference last week's conversation and the agent remembers

The biggest surprise: simple patterns beat clever ones every time. The bulletin board file outperformed my message queue. The flat MEMORY.md outperformed my vector database. The 50-line Python monitor outperformed my Kubernetes setup. Production AI is not about sophistication. It is about reliability.

Frequently Asked Questions

What is Claude Code production architecture?

Claude Code production architecture is a system design pattern for running AI agents reliably 24/7. It separates agent configuration into three files -- SOUL.md (identity), CONSTITUTION.md (behavioral boundaries), and HEARTBEAT.md (scheduled tasks) -- and adds a self-healing monitoring daemon plus a 4-layer memory tower for persistent knowledge. This architecture has been battle-tested running 10 agents continuously for over 90 days.

How do you prevent AI agent hallucination in production?

Three mechanisms work together: (1) CONSTITUTION.md defines strict behavioral boundaries the agent cannot cross -- hard rules that override permissive instructions. (2) A self-healing monitoring daemon detects anomalous outputs and auto-restarts drifting agents. (3) The memory tower ensures agents maintain persistent context across sessions, reducing the context gaps that cause hallucination.

What is SOUL.md and why does it matter?

SOUL.md is the identity file that defines who an AI agent is -- personality, expertise, communication style, and core directives. When combined with CONSTITUTION.md (what the agent must never do) and HEARTBEAT.md (what the agent does autonomously on a schedule), it creates a stable identity that persists across restarts. The structural separation into three files makes the LLM treat each concern with appropriate weight, which measurably improves rule adherence compared to a single config file.

How does the 4-layer memory tower work?

The memory tower has four layers: Layer 0 (Raw Facts) captures every session's key details. Layer 1 (Deduplicated) merges duplicates weekly. Layer 2 (Patterns) extracts recurring observations monthly. Layer 3 (Principles) stores rare, high-value learned rules with their reasoning. On session start, the agent reads all layers -- recent facts for context, patterns for awareness, and principles for accumulated wisdom. Automated scripts handle promotion between layers.

Can this architecture scale to more than 10 agents?

Yes. The filesystem-based isolation pattern scales linearly -- each new agent gets its own directory with its own SOUL, CONSTITUTION, HEARTBEAT, and MEMORY files. The sentinel daemon monitors all agents from a single loop. The only shared state is BULLETIN.md, which stays small through weekly pruning. I have tested up to 10 concurrent agents on a single Mac Mini with no performance issues.

Going Further

These four patterns are the foundation. I have been refining them over six months of daily production use and packaged the complete system into a toolkit: 21 production-tested files, 14,000+ lines of config, code, and documentation. It includes a runnable sentinel daemon, deployment configs, five complete agent examples across different use cases, and a 350-line production-grade CLAUDE.md that handles edge cases I have not covered here.

If you want the full toolkit: Ship AI Agents to Production -- $47, one-time, no subscription.

Or just take the four patterns above and build your own. That genuinely works too. The patterns matter more than any specific implementation.

What is the weirdest thing your AI agent has done in production? I am collecting war stories -- drop them in the comments. The salmon recipe incident is still my personal favorite, but I have a feeling some of you can top it.