Every tutorial on building AI agents starts with: "First, set up your PostgreSQL database." Then Redis for caching. Then a message queue. Before you've written a single line of agent logic, you're managing three infrastructure components.
I've been running autonomous AI agents for months. They maintain state across sessions, hand off context to each other, and track tasks reliably. The entire state layer is JSON files on disk.
Here's why that's not crazy — and when you should actually reach for a database.
The Problem with Database-First Thinking
When you're prototyping an AI agent, your state needs are simple:
- "What was I doing last session?"
- "What tasks are pending?"
- "What did I learn that I need to remember?"
This is a few KB of JSON. Spinning up Postgres for this is like renting a warehouse to store your groceries.
The File-Based Pattern
Here's the core of what actually works:
import json
from pathlib import Path
from datetime import datetime
class AgentState:
def __init__(self, state_dir: str = "./agent_state"):
self.state_dir = Path(state_dir)
self.state_dir.mkdir(parents=True, exist_ok=True)
self.state_file = self.state_dir / "state.json"
self.state = self._load()
def _load(self) -> dict:
if self.state_file.exists():
with open(self.state_file) as f:
return json.load(f)
return {
"session": 0,
"energy": 100,
"last_activity": None,
"active_tasks": [],
"completed_tasks": [],
"timestamp": datetime.now().isoformat()
}
def save(self):
self.state["timestamp"] = datetime.now().isoformat()
with open(self.state_file, 'w') as f:
json.dump(self.state, f, indent=2)
def new_session(self):
self.state["session"] += 1
self.state["energy"] = 100
self.save()
Nothing magical. No ORM, no migrations, no connection pooling. Just structured JSON that your agent reads at startup and writes before shutdown.
Handoffs: The Killer Feature
The real power shows up with session handoffs — when one agent instance needs to pass context to the next:
class HandoffManager:
def __init__(self, handoff_dir: str = "./handoffs"):
self.dir = Path(handoff_dir)
self.dir.mkdir(parents=True, exist_ok=True)
def create_handoff(self, context: dict) -> Path:
handoff = {
"timestamp": datetime.now().isoformat(),
"what_i_did": context.get("completed", []),
"what_needs_attention": context.get("pending", []),
"key_context": context.get("context", []),
"next_priorities": context.get("priorities", [])
}
path = self.dir / "latest_handoff.json"
with open(path, 'w') as f:
json.dump(handoff, f, indent=2)
return path
def receive_handoff(self) -> dict:
path = self.dir / "latest_handoff.json"
if path.exists():
with open(path) as f:
return json.load(f)
return None
This solves a real problem: when your agent restarts (crashes, scheduled shutdown, context window fills up), the next instance knows exactly what was happening.
Task Tracking Without a Task Database
class TaskTracker:
def __init__(self, tasks_file: str = "./agent_state/tasks.json"):
self.file = Path(tasks_file)
self.tasks = self._load()
def _load(self) -> list:
if self.file.exists():
with open(self.file) as f:
return json.load(f)
return []
def add(self, description: str, priority: str = "medium"):
self.tasks.append({
"id": len(self.tasks) + 1,
"description": description,
"status": "pending",
"priority": priority,
"created": datetime.now().isoformat()
})
self._save()
def complete(self, task_id: int):
for task in self.tasks:
if task["id"] == task_id:
task["status"] = "completed"
task["completed_at"] = datetime.now().isoformat()
self._save()
def next_task(self) -> dict:
priority_order = {"high": 0, "medium": 1, "low": 2}
pending = [t for t in self.tasks if t["status"] == "pending"]
if pending:
return sorted(pending, key=lambda t: priority_order.get(t["priority"], 1))[0]
return None
def _save(self):
with open(self.file, 'w') as f:
json.dump(self.tasks, f, indent=2)
Your agent can now track what it needs to do, pick the highest-priority task, and mark things done — all with zero infrastructure.
When You Actually Need a Database
File-based state has real limits. Reach for a database when:
- Multiple agents write simultaneously — file locks get messy fast
- You need to query across thousands of records — JSON files don't have indexes
- Audit trails matter — you want transactional guarantees
- Your state exceeds ~10MB — file I/O starts to feel it
For a single agent running sequentially? Files are simpler, faster to develop, and easier to debug (you can literally cat state.json to see what your agent is thinking).
The Debug Advantage
This is underrated: when your agent does something weird, you open a JSON file and read it. No query tools, no admin panels. Just:
cat state.json | python -m json.tool
You can even version control your state files to track how your agent's behavior evolves over time.
Putting It Together
# Agent startup
state = AgentState()
tasks = TaskTracker()
handoffs = HandoffManager()
# Receive context from previous session
previous = handoffs.receive_handoff()
if previous:
print(f"Picking up from: {previous['what_i_did']}")
for p in previous.get("next_priorities", []):
tasks.add(p, priority="high")
# Start new session
state.new_session()
# Work loop
while task := tasks.next_task():
print(f"Working on: {task['description']}")
# ... agent does work ...
tasks.complete(task["id"])
# End of session handoff
handoffs.create_handoff({
"completed": [t["description"] for t in tasks.tasks if t["status"] == "completed"],
"pending": [t["description"] for t in tasks.tasks if t["status"] == "pending"],
"priorities": ["Continue from where I left off"]
})
state.save()
The Bottom Line
Start with files. Add complexity when you hit a real wall, not when a tutorial tells you to. Your agent's first job is to be useful — not to have a proper data layer.
The best infrastructure is the simplest thing that works. For most AI agents starting out, that's a directory of JSON files.
I build and run autonomous AI agent systems. This is based on real production patterns, not theory. If you're building agents, I'd love to hear what state management approach works for you.
Top comments (0)