DEV Community: Achir othmane

Your AI Agent Passed the Approval Check. Did the Side Effect?

Achir othmane — Tue, 22 Sep 2026 09:08:27 +0000

Most AI agent safety systems ask a question like:

«Is this action allowed?»

That is necessary.

But for action-taking agents, it may not be enough.

Imagine that your intended production path looks like this:

agent
↓
approval:P
↓
deploy_to_production

You add the approval. You test it. The expected tool path respects it.

Everything looks fine.

But the same agent also has access to a shell.

Now another path exists:

agent
↓
shell
↓
gh workflow run deploy.yml
↓
production

The approval system may be perfectly correct.

The problem is that the consequence was reachable through another path.

That is the problem I have been experimenting with in an open-source project called Agent Action Guard:

https://github.com/othy19904-eng/agent-action-guard

Authorization correctness is not consequence completeness

There are two different questions:

Did the policy correctly authorize this action?
Can the agent reach the same real-world consequence
through another execution path?

Those are not equivalent.

I represent the second one as:

expected:

agent
-> approval:P
-> production_deploy

versus:

counterexample:

agent
-> shell.exec
-> workflow:deploy.yml
-> production_deploy

The static scanner in the project looks for that second path.

It can return:

COUNTEREXAMPLE_FOUND
COVERED_WITHIN_MODEL
UNKNOWN

Importantly:

COVERED_WITHIN_MODEL != proven safe
UNKNOWN != impossible

The scanner is deliberately conservative.

But static analysis creates another problem.

A route being possible does not mean the agent actually used it.

So I added a second layer.

Runtime Witness

The runtime layer asks:

«What path was actually observed when the protected consequence occurred?»

That produces a different set of results:

BOUNDARY_OBSERVED
OBSERVED_BYPASS
UNRESOLVED_TRACE

For example:

root:agent
↓
boundary:approval:P
↓
tool:deploy
↓
consequence:production_deploy

becomes:

BOUNDARY_OBSERVED

But:

root:agent
↓
effect:shell.exec
↓
workflow:deploy.yml
↓
consequence:production_deploy

becomes:

OBSERVED_BYPASS

There is an important third state.

If the execution evidence is incomplete, the system does not infer safety and does not infer a bypass.

It returns:

UNRESOLVED_TRACE

That distinction matters.

Absence of an approval event in a broken log is not evidence that approval was bypassed.

Connecting it to OpenAI Agents SDK

The project now has an experimental adapter for the OpenAI Agents SDK.

It observes the real local tool lifecycle through "RunHooks".

The basic idea is:

hooks = OpenAIAgentsWitnessHooks(
trace_path,
tool_consequences={
"write_marker": "local_file_write"
},
)

result = await Runner.run(
agent,
"Write the marker file.",
hooks=hooks,
)

If the application observes the expected approval boundary:

hooks.boundary("approval:P")

After the run completes:

hooks.finalize()

Runtime Witness can then test whether the observed consequence path contained that boundary.

Try the complete demo

Clone the repository and install the optional adapter:

python -m pip install ".[openai-agents]"

Then run the guarded case:

python examples/openai_agents_runtime_witness.py --mode guarded

Expected:

BOUNDARY_OBSERVED

Now run exactly the same agent/tool pipeline without the approval boundary:

python examples/openai_agents_runtime_witness.py --mode bypass

Expected:

OBSERVED_BYPASS

The demo executes a real Python function tool that performs a temporary filesystem side effect.

It does not require an OpenAI API key or a model request.

The test uses the Agents SDK's deterministic testing model so that the actual Runner and tool orchestration can be exercised reproducibly.

I intentionally do not record everything

An execution-security tool can easily become a new data-leak surface.

So the OpenAI Agents adapter currently records only minimal lifecycle evidence such as:

tool name and tool-call identity
observed boundary events
successful tool completion
mapped consequence
trace-completeness attestation

It does not persist tool arguments, prompts, model outputs, stdout, or stderr by default.

For many systems, those values may contain credentials, customer information, proprietary code, or other sensitive data.

Why require trace completeness?

Suppose this is all you see:

agent
↓
tool
↓
production consequence

Was approval bypassed?

Maybe.

Or perhaps the approval event simply disappeared because your telemetry pipeline dropped it.

So an "OBSERVED_BYPASS" requires an explicit completed trace.

Without sufficient evidence:

UNRESOLVED_TRACE

I would rather return “I don't know” than silently convert missing evidence into a security claim.

Static evidence + runtime evidence

The direction I am exploring is therefore:

repository
↓
static consequence graph
↓
possible alternate path
↓
real agent execution
↓
runtime witness
↓
observed boundary verdict

The static layer tells you:

this bypass CAN exist

The runtime layer tells you:

this path WAS observed

Neither is meant to replace authorization engines, guardrails, or approvals.

The question is different:

«After you created the safety boundary, is that boundary actually complete around the consequence you care about?»

The project is still early and intentionally narrow.

If you are building action-taking agents with OpenAI Agents SDK, I am particularly interested in real-world tests.

Repository:

https://github.com/othy19904-eng/agent-action-guard

If you test it, the most useful feedback is simple:

Did it report a real path?
Did it produce a false positive?
Would this check be useful enough to keep in CI?

That evidence will determine what gets built next.

Your Shopify Store Can Be Selling While GA4 Is Wrong — I Built a Tool to Check

Achir othmane — Mon, 21 Sep 2026 19:10:56 +0000

Before changing the tracking stack again, I’d separate two different questions:

How many Shopify orders were actually eligible to be attributed to Google Ads?
Of the orders/conversions that should exist in the measurement layer, are any missing, duplicated, or carrying the wrong value?

Those are different problems. 80 Shopify orders vs 16 Google Ads conversions does not by itself prove that 64 conversions were lost, because not every order necessarily came from an eligible Google Ads click.

I’m building an open-source tool called ConversionGuard specifically to reconcile authoritative ecommerce orders against measured purchase records by transaction ID, value and currency.

It can flag:

missing purchase records
duplicate transaction IDs
value/currency mismatches
measured conversions with no corresponding order

For the first 3 Shopify stores I’m doing the audit free to validate it on real data. No production credentials are required for the first pass — sanitized CSV exports are enough.

Project:
https://github.com/othy19904-eng/conversionguard

Pilot:
https://github.com/othy19904-eng/conversionguard/issues/1

I built a scanner for AI-agent approval bypass paths

Achir othmane — Mon, 21 Sep 2026 10:31:23 +0000

AI agents can have valid access to tools and APIs, but access is not the same thing as authority.

Suppose an agent is expected to reach production only through an approval boundary:

agent → approval → production deploy ✅

But the same repository may contain another execution path:

agent → shell → gh workflow run → production deploy ❌

I built an early open-source scanner to look for this specific class of problem.

It builds a modeled consequence path across Python and GitHub Actions and asks:

Is there another modeled route to the same real-world consequence that does not cross the expected approval boundary?

When it can prove such a path, it returns:

COUNTEREXAMPLE_FOUND
the path it followed
source-code evidence

It deliberately does not claim full repository completeness.

UNKNOWN is not interpreted as safe, and COVERED_WITHIN_MODEL is limited to what the current model could analyze.

The current MVP is intentionally narrow:

Python
GitHub Actions
production-deployment consequence paths

First run:


bash
python -m pip install .
consequence-boundary . --consequence production_deploy --boundary P

Repository:
https://github.com/othy19904-eng/agent-action-guard

I’m specifically looking for people building action-taking AI agents or automated workflows who can run it on a real repository.

The most useful feedback would be:

Did you get COUNTEREXAMPLE_FOUND, COVERED_WITHIN_MODEL, or UNKNOWN?

If it found a path, was it real or a false positive?

Would a check like this be useful enough to keep in CI?

This is an early demand test before I add more features.