An Evaluation Is Not an Approval: Building AFTERIMAGE with FalkorDB

stephen-gabriel — Sun, 18 Oct 2026 07:21:50 +0000

A saved investigation in my hackathon project ended with NO_ACTION_NEEDED. The policy evaluator could see passing evidence. The current release revision still had no recorded decision.

Those facts should not have been allowed to collapse into the same conclusion.

That bug became the clearest explanation of why I built AFTERIMAGE: an approval needs a record of the exact facts it depended on, and a way to recognise when those facts no longer apply.

I built the prototype for Graph Hacks by WeMakeDevs and FalkorDB, for developers and release reviewers who need to revisit decisions after dependencies change.

The approval that outlived its context

Imagine a checkout release approved against payment-api version 2.0.0. Its integration evidence passes, its requirement is satisfied, and the decision is recorded.

Now the dependency changes to 3.0.0.

The old test result has not become false. It still describes the earlier configuration. What has changed is whether that result supports the release we are reviewing now.

AFTERIMAGE creates a new release revision and marks it Needs review. The old approval remains visible as history. A separate documentation release, which does not depend on that API, stays approved.

The demonstration follows this sequence:

What happens	What AFTERIMAGE records or shows
Checkout is approved against payment-api v2	A decision supported by revision-specific evidence
The dependency changes to v3	A new revision; the earlier decision is no longer current
The agent investigates	Findings with record references and proposed review actions
I apply an evidence-request proposal	A persistent request for the missing evidence
Matching synthetic PASS evidence is recorded	Evidence for the new revision; approval still requires a decision
I apply a reassessment	A new recorded decision, with its supporting evidence

“Needs review” does not claim the dependency is broken. It identifies a gap between an earlier decision and the evidence available for the current configuration.

Why the graph is part of the mechanism

FalkorDB stores the application’s authoritative records. Release revisions, dependency versions, requirements, evidence, sources, and decisions are connected through directed relationships.

Some of the relationships that matter most are:

Relationship	Question it answers
`ReleaseRevision —DEPENDS_ON→ DependencyVersion`	Which dependency version belongs to this configuration?
`Evidence —FOR_REVISION→ ReleaseRevision`	Which revision does this evidence describe?
`Evidence —SATISFIES_CHECK→ Requirement`	Which requirement is the evidence attached to?
`Evidence —FROM_SOURCE→ Source`	Where did it come from?
`Decision —ASSESSES→ ReleaseRevision`	What was actually reviewed?
`Decision —SUPPORTED_BY→ Evidence`	What supported the recorded outcome?

Attaching evidence to a requirement is not enough to approve it. The policy still checks the outcome, revision, dependency scope, source authority, and withdrawal or supersession state.

Impact lookup follows the graph backwards from a changed entity. This excerpt from the application’s findAffectedReleases query follows a dependency to the release revisions that use it, then selects releases for which those revisions are current:

MATCH (entity {id: $entityId})
WHERE entity.workspaceId = $wsId
OPTIONAL MATCH (entity)<-[:DEPENDS_ON]-(depRev:ReleaseRevision {workspaceId: $wsId})
OPTIONAL MATCH (depRev)<-[:CURRENT_REVISION]-(depRel:Release)

This is an excerpt, not a standalone query: the full implementation includes additional paths and returns the affected releases.

The same graph supplies context to the agent and records to the policy evaluator. The Connections tab makes those paths readable, but the database also determines which records the application can investigate and evaluate. Removing it breaks the core reads and writes.

What the agent can do

The live agent uses an OpenRouter model through six narrow tools:

get_release_context
find_affected_releases
inspect_evidence
get_decision_history
evaluate_release
propose_review_action

These tools expose validated operations over the application’s query layer. The model does not receive unrestricted database access.

An investigation can propose creating an evidence request, resolving a satisfied request, or recording a reassessment. The user applies the proposal, and the backend checks the current revision and the action’s requirements again.

Runs, proposals, requests, and decision history persist in FalkorDB. A later session can recover the earlier work rather than starting from an empty conversation.

That separation matters: the model helps investigate and choose the next action, while deterministic application code evaluates the evidence and guards writes.

The bug that clarified the design

The most useful failure was the one in the opening paragraph.

An earlier completion rule effectively treated an APPROVED evaluation as sufficient reason to stop. It did not verify that a decision had actually been recorded for the investigated revision. Another part of the code classified completion from the model’s findings text, which could omit actions the run had submitted.

The fix needed more than a stronger prompt.

The backend now checks the release’s current decision pointer and verifies that the referenced decision assesses the investigated revision. The relationship check uses this Cypher pattern:

MATCH (d:Decision {id: $decisionId, workspaceId: $wsId})
      -[:ASSESSES]->
      (rev:ReleaseRevision {id: $revisionId, workspaceId: $wsId})
RETURN d.id as id

When evaluation is complete but the relevant decision is missing, the validator requires a reassessment proposal unless one is already pending. It gives the model bounded corrective feedback. If the required action is still missing when that allowance runs out, the run fails validation. The application does not invent a proposal or silently write an approval.

Completion classification also uses the run’s submitted proposals, rather than relying on whether its prose mentions them.

The resulting distinction is precise:

An evaluation computes what the policy says about the evidence now. A recorded decision preserves an explicitly applied assessment of a particular revision.

The validation and orchestration code makes that distinction enforceable.

A valid proposal can become stale

A second lesson appeared when an investigation proposed two actions based on the same workspace state.

Applying the first action advances the workspace revision. The other proposal was prepared against the earlier state. Letting both apply without checking would make the second action depend on assumptions that might no longer hold.

AFTERIMAGE stores the revision each proposal expects. The backend compares it with the current workspace revision at application time. Outdated proposals become stale, and the interface offers another investigation against the refreshed state.

Release revisions and workspace revisions serve different purposes: one identifies a release configuration; the other tracks changes to the shared application state.

The prototype serializes mutations in one backend process. These guards are not a claim of distributed, multi-instance concurrency support.

What testing—and a free model—taught me

I used FalkorDB in Docker and a free OpenRouter model for the live demo. Runtime spending stayed at $0, but model availability was a real constraint: a later investigation hit the provider’s daily quota.

That made it important to separate three kinds of evidence:

Deterministic policy tests check applicability, conflicts, withdrawals, supersession, and revision handling.
A clearly labelled test model exercises action flows repeatably without consuming live-provider requests.
Live-provider runs check that a real model uses the tools and produces usable proposals.

A passing test double does not prove that a live model followed the same path. Similarly, a quota failure does not prove that a release failed its checks. The interface reports it as an investigation failure.

Focused regression tests cover stale proposal application and the missing-decision case, including a deliberately uncooperative test model. The important outcome is that failure remains visible without fabricating successful work.

What the prototype does—and what comes next

AFTERIMAGE currently uses synthetic evidence and a small demonstration policy. It does not run CI, deploy software, or certify that a release is safe. A resolved evidence request closes an internal task; approval still requires a recorded decision.

The next useful extension would be ingesting real CI results while preserving their source and exact revision scope. A production version would also need authentication, stronger operational controls, and a durable execution design before running across multiple backend instances.

For this build, the useful result is already concrete: I can change a dependency, see which approval needs review, investigate the evidence gap, and record a new decision without erasing the old one.

The record of why we approved something should remain useful after the facts change.

Explore the project

AI assistance: I directed the product scope, reviewed the running behaviour, and used AI coding assistants extensively for implementation, debugging, and test development. AI assistance also helped draft this article from the repository and build records. The repository includes a fuller disclosure.

DEV Community: stephen-gabriel