Building AgentTrace: How I Made AI Agents Remember, Recover, and Resume with FalkorDB

SAIF MODAN — Sun, 18 Oct 2026 14:33:43 +0000

AI agents are getting better at planning, writing code, and using tools. But there is still a frustrating problem: what happens when an agent forgets what it has already done?

Imagine an AI agent working on a software project. It analyzes the source code, runs tests, encounters an error, and starts fixing it. Then its process crashes.

When the agent restarts, several questions appear:

Which tasks were completed?
Which actions failed?
What caused the failure?
Which dependencies are still blocked?
Is it safe to retry an interrupted operation?
How can another agent continue the work without starting from scratch?

A traditional conversation history or a simple task list isn't enough to answer all these questions reliably.

That is the problem behind AgentTrace, a graph-native execution-memory and coordination system built with FalkorDB for the WeMakeDevs × FalkorDB Graph Hacks 2026 hackathon.

GitHub repository: AgentTrace — Open Source

The Idea: Give Agents an Execution Memory

Instead of depending entirely on the LLM's context window, AgentTrace records execution state inside a persistent graph.

The graph stores tasks, dependencies, attempts, failures, recovery procedures, decisions, verification results, and checkpoints.

More importantly, these are not just historical logs.

The relationships actively determine what the agent can do next.

An agent can ask:

What steps are ready to execute?
What is preventing this task from running?
Which earlier failure is relevant?
What approved recovery procedure applies?
What work was already verified before the process stopped?

FalkorDB answers these questions through graph queries.

Why a Graph Database?

An agent workflow naturally contains relationships.

Consider a simple task sequence:

Inspect Application → Run Tests → Build Application → Inspect Artifact

The build must not execute if the tests have not passed.

When the tests fail, a recovery action might need to be inserted as a new dependency.

These connections are important. They should not exist only inside an LLM-generated plan.

In AgentTrace, a task points toward its prerequisites through a DEPENDS_ON relationship.

The execution graph includes twelve node labels:

Node	Responsibility
AgentRun	Overall execution goal
Step	Planned actions and dependencies
Attempt	Individual tool executions
Agent	Planner, executor, verifier, and recovery workers
Event	Execution history
Failure	Recorded errors
Procedure	Reusable recovery knowledge
Fact	Facts learned during execution
Artifact	Generated files and outputs
Decision	Actions and supporting evidence
Checkpoint	Persistent progress snapshots
Resource	Files and workspaces

This graph becomes the shared execution state for the entire system.

How AgentTrace Works

The architecture combines a React dashboard, a FastAPI backend, and FalkorDB.

There are four cooperating agent roles.

1. Planner

Creates a validated task plan with dependencies and expected outcomes.

2. Executor

Selects eligible tasks, claims them, runs registered tools, and records actual results.

3. Verifier

Independently checks whether a task achieved its expected result.

4. Recovery

Investigates failures, retrieves supporting graph evidence, and introduces approved corrective actions.

All four roles work against the same graph-backed memory manager.

The important distinction is that a worker cannot simply choose any pending task.

It must first consult the dependency graph.

Graph-driven scheduling

A simplified version of the readiness query looks like this:

MATCH (r:AgentRun {id:$run})-[:HAS_STEP]->(s:Step)
OPTIONAL MATCH (s)-[:DEPENDS_ON]->(d:Step)
WITH r, s, collect(d) AS deps
WHERE r.status = 'running'
  AND s.status IN ['pending', 'ready', 'retryable']
  AND s.retry_after_ms <= $now
  AND s.attempt_count < s.max_attempts
  AND all(
    d IN deps
    WHERE d.status = 'succeeded'
    AND d.verification_status = 'passed'
  )
RETURN s, deps
ORDER BY s.priority DESC
LIMIT 30

The production implementation adds lease and resource checks.

The actual claim repeats eligibility checks inside an atomic graph write, creates an execution attempt, and records ownership information.

This helps prevent two workers from claiming the same task.

FalkorDB is not merely storing the workflow. It helps enforce the workflow.

The Demo: An Agent Encounters a Real Failure

To demonstrate AgentTrace, the project includes a controlled Python calculator application.

The calculator contains a deliberately broken addition function.

def add(a, b):
    return a - b

The expected behavior is addition, but the function subtracts.

The executor runs real Python unit tests against this source.

The tests fail.

Rather than discarding that result, AgentTrace records the unsuccessful execution as an Attempt, together with its error evidence.

The recovery component investigates the failure and checks whether it matches a supported, approved correction.

In the demonstration, the failure is recognized as an addition defect.

AgentTrace introduces a new repair task that changes the function to:

def add(a, b):
    return a + b

But there is an important detail.

The failed test step now depends on the repair step.

The graph prevents the test from proceeding before the correction is applied and verified.

This is where graph-based execution memory becomes useful: recovery actually changes the workflow's dependency structure.

Verification matters

After the repair, the tests run again.

The verifier independently checks the result rather than trusting a tool's success message.

The graph retains both the original failure and the successful attempt.

The process also produces a checked build artifact.

The original failure is not erased just because the workflow eventually succeeds.

It remains part of the agent's execution memory.

Making Decisions Explainable

Another goal was to make agent actions traceable.

For example, why did the recovery component choose a particular repair?

AgentTrace stores Decision nodes linked to relevant evidence, including Failure and Procedure nodes.

A decision can therefore be investigated through the graph.

A simplified evidence query is:

MATCH (d:Decision {id:$id})
OPTIONAL MATCH (d)-[r:BASED_ON]->(e)
RETURN d, collect(e), collect(r)

The dashboard uses this information to show the relationships supporting a decision.

This is different from asking an LLM to invent an explanation after executing an action.

The evidence is associated with the decision when it is recorded.

AgentTrace also implements multi-hop dependency impact analysis.

For example:

MATCH p=(affected:Step)
  -[:DEPENDS_ON*1..30]->
  (failed:Step {id:$id})
RETURN affected, nodes(p)
LIMIT 300

This query identifies downstream tasks that depend on a particular step.

It helps answer questions like:

"If this task fails, which other tasks are affected?"

And:

"Which independent tasks can still proceed safely?"

What Happens When the Agent Crashes?

An execution-memory system should not depend on an uninterrupted process.

AgentTrace uses persistent graph state, execution attempts, leases, and reconciliation logic to support resuming interrupted work.

The repository includes a real worker-restart integration test.

The test:

Executes and verifies an initial task.
Starts a separate worker process.
Allows that worker to claim another task.
Terminates the worker process.
Waits for the execution lease to expire.
Starts a new worker.
Checks that execution can continue.
Confirms that previously completed work was not repeated unnecessarily.

The distinction between an interrupted action and a failed action matters.

An interrupted read or test operation may be safe to retry.

An external action with an uncertain outcome might not be.

The system therefore performs reconciliation instead of blindly replaying every interrupted tool call.

This demonstration covers a specific controlled interruption point, not every possible crash scenario.

A Real Challenge: FalkorDB Persistence

One of the most interesting issues encountered during development was database durability.

During testing, the chosen FalkorDB container image experienced a segmentation fault while replaying an AOF graph-effect record involving a constraint.

This was observed with the pinned image used by AgentTrace; it is not a claim that every FalkorDB version has the same issue.

The implementation was adjusted to use RDB snapshot persistence with synchronous checkpoint saves.

Container stop, restart, and replacement tests subsequently passed with the persistent volume configuration.

However, the tradeoff is important.

RDB snapshots do not guarantee that every recent write survives an abrupt database or host failure.

The project explicitly documents this limitation rather than claiming perfect durability.

The complete details are available in the recovery design documentation.

This experience reinforced an important lesson: persistent agent memory requires thinking about database recovery, not just application-level checkpoints.

Testing and Validation

AgentTrace was tested locally using real FalkorDB instances, backend tests, browser automation, and a controlled end-to-end demonstration.

The recorded validation results were:

Validation	Result
Backend tests	30 passed
Browser tests	3 passed
Frontend production build	Passed
Python lint	Passed
Frontend lint	Passed
Docker Compose validation	Passed
Worker crash/restart test	Passed
Database restart/replacement test	Passed

The implementation also tested concurrent task claims, lease expiry, stale-result rejection, dependency traversal, tenant isolation, and graph-backed event replay.

Representative local query measurements were:

Query	Median
Run lookup	3.232 ms
Step lookup	3.013 ms
Readiness traversal	5.466 ms
Multi-hop dependency traversal	8.778 ms

These measurements came from small synthetic graphs in the local development environment, with client and network overhead included. They should not be interpreted as production-scale benchmarks.

The detailed report is available in validation.md.

What I Learned

Building AgentTrace highlighted several important lessons about reliable AI-agent systems.

Memory should influence execution

Storing conversation history is useful, but operational memory becomes more valuable when it helps determine which actions are safe and which dependencies remain unresolved.

Failure is useful information

A failed attempt contains evidence about the state of the system.

Preserving that evidence allows the agent to investigate problems instead of repeatedly starting from incomplete context.

Verification and execution are different responsibilities

A command finishing does not automatically mean a task has achieved its intended objective.

Independent verification makes execution records more trustworthy.

Recovery is more than retrying

Retrying an unsuccessful action without examining the cause can repeat the same failure or duplicate an external side effect.

Graph-backed recovery can introduce new prerequisites and preserve the reasoning behind corrective actions.

Durability needs to be tested

A system that survives a worker restart is not necessarily protected against every form of database or host failure.

These are different reliability problems that require different tests.

Current Limitations and What's Next

AgentTrace is a working hackathon implementation, but it is not yet a general-purpose production agent platform.

The current demonstration uses a deterministic plan and supports one controlled recovery scenario.

Optional planning through an OpenAI-compatible model endpoint exists, but live-model behavior was not validated in the published local test report.

The execution environment uses restricted tools but is not a hardened operating-system sandbox for arbitrary untrusted code.

The system also does not yet provide automatic revalidation of every downstream result when a previous verification becomes stale.

Future development could explore broader recovery procedures, more sophisticated memory retrieval, live-model evaluation, and stronger durability guarantees.

Built with FalkorDB

AgentTrace demonstrates how a graph database can provide more than connected data storage.

FalkorDB powers task dependencies, execution eligibility, failure relationships, coordination, memory retrieval, and recovery evidence.

Without those graph operations, the central workflow cannot function in the same way.

For this project, the most important idea is simple:

An AI agent should not only remember what it said. It should remember what it did, what actually happened, why something failed, and what evidence supports its next action.

That is the foundation AgentTrace explores.

Explore the Project

GitHub: https://github.com/saifmodan2006/agenttrace

Graph schema: https://github.com/saifmodan2006/agenttrace/blob/main/docs/graph-schema.md

Cypher queries: https://github.com/saifmodan2006/agenttrace/blob/main/docs/cypher-queries.md

Local demo setup: https://github.com/saifmodan2006/agenttrace/blob/main/docs/demo-guide.md

AgentTrace was developed with AI coding assistance, including OpenAI Codex. The implementation, automated validation, and assistance disclosure are documented in the public repository.

This project was created for the WeMakeDevs × FalkorDB Graph Hacks 2026 Hackathon, primarily under the Agent Memory and Coordination track, with additional graph-driven capabilities relevant to Agents That Act on Connected Data.

If you are working on long-running AI agents or multi-agent workflows, I would be interested in hearing how you handle execution memory, failures, and recovery.

DEV Community: SAIF MODAN