Your AI agent says the tool failed. What if the action actually happened?

A tool call returning an error does not necessarily mean the action failed.

That sounds obvious if you have spent time around distributed systems.

It gets much more dangerous when an AI agent is the thing deciding what to do next.

Imagine this:

1. An agent decides to make a payment.
2. The payment request reaches the provider.
3. The provider creates the payment.
4. The response times out.
5. The agent receives an error.

From the agent's point of view:

PAYMENT FAILED

From the provider's point of view:

PAYMENT HAPPENED

Now the agent has to decide what to do next.

Retry?

If it retries blindly, you may now have two payments.

That is the problem I have been thinking about.

A tool response is not always evidence of the effect

We tend to collapse several things into one event:

agent calls tool
      ↓
tool returns result
      ↓
agent decides what happened

But consequential systems do not always work like that.

The real sequence can be:

agent calls tool
      ↓
provider accepts request
      ↓
real-world state changes
      ↓
response is lost
      ↓
tool reports an error

The tool may be completely honest.

The transport really did fail.

But the external effect also really happened.

Both facts can be true at the same time.

That makes the response useful evidence about the call, but not necessarily authoritative evidence about the final effect.

Idempotency helps, but it does not answer the whole question

The obvious answer is idempotency.

And yes, consequential tools should use idempotency wherever the underlying system supports it.

If the same payment request is repeated with the same idempotency key, the provider should not create a second payment.

That prevents one very important class of failure.

But the agent still has a question:

Did the first action actually happen?

Before deciding whether to retry, escalate, compensate, or stop, something has to establish the downstream state.

That may mean:

• reading the provider state;
• checking a transaction receipt;
• querying the resource independently;
• observing the physical side effect;
• reconciling against another authoritative system.

This is where I think agent testing needs to go further than checking the model output or tool response.

There are at least three different questions

1. Was the action allowed?

This is an authorization problem.

Can this agent perform this action against this resource under this policy?

2. Was the evidence behind the action still current?

This is the problem FreshCtx addresses.

An agent may make a correct decision at 10:01.

Reality changes at 10:03.

The action executes at 10:04.

The original reasoning was not necessarily bad. It was simply no longer valid when execution happened.

FreshCtx revalidates the declared evidence immediately before the consequential action.

Conceptually:

CURRENT
→ proceed

STALE_REASONING
→ block

UNVERIFIABLE
→ block under the default policy

That is intentionally a narrow job.

FreshCtx does not claim that an authorized action will execute correctly.

And it does not claim that a tool response proves what happened afterward.

3. What actually happened?

This is a different assurance problem.

Once the action crosses the execution boundary:

• Did the effect occur?
• Did it occur once?
• Did the agent report the same thing reality shows?
• Can we establish the result independently?
• If we repair the problem, does the same execution path now behave correctly?

I did not want to keep expanding FreshCtx until it became an authorization system, observability platform, execution tester and audit system at the same time.

So I separated the problem.

That became Revera

Revera is the system I built for the execution side of this problem.

The question it tries to answer is simple:

What did the AI agent actually execute?

The workflow is roughly:

discover consequential action
        ↓
reproduce execution
        ↓
observe actual effect
        ↓
diagnose
        ↓
remediate where supported
        ↓
rerun the exact path
        ↓
produce evidence

One design principle became especially important:

The system under test should not be the only authority on whether its own action succeeded.

If the agent reports:

PAYMENT FAILED

while an independent observation establishes:

PAYMENT EXECUTED

I want both facts preserved.

Not one silently replacing the other.

The uncomfortable state is sometimes "we don't know"

There is another case that matters just as much.

Suppose:

• the tool returned an error;
• the provider cannot currently be queried;
• no authoritative receipt is available;
• the effect may have happened.

Calling that FAILED is dangerous.

Calling it SUCCESS is equally dishonest.

The safest answer may simply be:

UNVERIFIABLE

or, depending on the system:

MAY_HAVE_EXECUTED

That state changes what the agent is allowed to do next.

Blind automatic retry may be exactly the wrong response.

Why exact retesting matters

There is one more place where I think agent security work often stops too early.

A failure is reproduced.

Someone generates a patch.

The code looks correct.

And the issue is marked fixed.

But:

patch generated != fix proven

For a consequential execution problem, I want the repaired path to actually run again under the relevant execution conditions.

Then observe what happened.

And also verify that legitimate behavior still works.

That is why Revera includes an exact retest step rather than treating remediation generation as the end of the workflow.

FreshCtx and Revera solve different problems

The distinction I use now is:

FreshCtx
Is the evidence still current before the action?

Revera
What actually executed, what effect occurred, and can the repair be proven?

FreshCtx remains open source.

Revera is the separate verification product.

I think keeping that boundary explicit makes both systems more useful.

I want to test Revera on real external systems, not just examples we built ourselves.

Revera is live now:

https://reveralab.com

At this stage, I am not trying to maximize signups.

I want a small number of external developers building agents that actually change something:

• payments;
• infrastructure;
• customer records;
• access;
• deployments;
• bookings;
• approvals;
• workflows.

If you have one public repository, MCP tool, or consequential action where a wrong execution would matter, send it to me.

Especially if you think Revera will get the answer wrong.

That is much more useful right now than another test against a system we built ourselves.

And I am curious about one thing from other builders:

What is one action in your agent that you would absolutely not trust it to retry blindly?