Why I stopped trusting “exit code 0” from AI coding agents

Ilya Golovenkov — Mon, 28 Sep 2026 05:55:03 +0000

AI coding agents are getting much better.

They can inspect repositories, change multiple files, run commands, write tests, and sometimes complete work that would have taken a developer hours.

But I kept running into the same problem:

the agent would finish, return exit code 0, and say that everything was successful.

Sometimes it really was.

Sometimes it wasn't.

That made me realize that I did not only need a better coding agent.

I needed an independent way to verify what the agent had actually done.

The problem

When an AI coding agent works on a project, several things can go wrong:

required files may be missing before the run even starts;
the agent may misunderstand the project structure;
it may modify files it was not supposed to touch;
tests may not actually run correctly;
the agent may report success even when verification is incomplete;
a successful process exit does not necessarily mean the project is actually in a good state.

The stronger coding agents become, the more important this problem becomes.

If an agent changes 10 or 20 files, I do not want to manually inspect every line after every run.

And if agents become even more autonomous, manual verification becomes even less practical.

That is the problem I wanted to solve.

What I built

I built a small open-source CLI called ELY Agent Input Preflight.

👉 Try ELY Agent Input Preflight on GitHub:

https://github.com/Golovenkov79/ely-agent-input-preflight

The idea is simple:

Check whether the project is ready before the AI agent starts.
Detect the project type.
Prepare the right context and instructions.
Run the coding agent in read-only mode by default.
Monitor whether project files changed unexpectedly.
Run independent verification after the agent finishes.
Record the result locally.

The important part is this:

an agent exit code of 0 is not automatically treated as success.

ELY can return:

VERIFIED
FAILED
NEEDS_REVIEW

So the agent's own result is only one signal.

It is not the final authority.

Before the agent starts

The first layer is Preflight.

Before the coding agent runs, ELY checks whether the project has the inputs it is supposed to have.

That can include things such as:

required files;
required directories;
file patterns;
environment variables;
valid JSON files;
non-empty files;
optional files that should produce warnings instead of blocking the run.

If something critical is missing, ELY can stop the run before the coding agent starts.

That sounds simple, but it avoids wasting an expensive agent run just to discover later that the required context was incomplete.

Project detection and skills

ELY also detects the type of project it is looking at.

The current release includes project detection for:

Python;
Android;
.NET.

Based on that detection, ELY can select matching built-in guidance before preparing the handoff to the coding agent.

The goal is not to replace the agent.

The goal is to make sure the agent starts with better context and clearer rules.

Read-only by default

One design decision was especially important to me:

the coding agent should not get write access by default.

With the current Codex integration, ELY starts the agent in an explicit read-only sandbox unless write access is deliberately enabled.

If I only want an agent to inspect a project, I do not want it silently modifying files.

Write access has to be explicitly requested.

That gives me a much clearer separation between:

inspection;
modification.

File integrity monitoring

ELY also takes a snapshot of project files before an agent run.

After the agent exits, it compares the workspace again.

This allows ELY to detect:

created files;
modified files;
deleted files.

That matters especially for read-only runs.

If a supposedly read-only agent run changes protected project files, ELY can treat the result as a failure.

This is independent of what the agent claims it did.

A real example

During development I had a Codex run where the agent completed with exit code 0.

Inside its sandbox, however, many tests could not run correctly because temporary writable storage was unavailable.

The agent still completed its inspection and reported the limitation.

ELY did not simply trust the agent result.

After Codex exited, ELY ran its own Postflight verification outside the agent sandbox and independently checked the project.

The Postflight verifier was able to run the project's tests normally and verify the state independently.

That was the moment when the idea became useful to me.

The interesting part was not that the coding agent was "bad".

It was that the agent and the verifier were operating under different conditions.

A successful agent run and a successfully verified project are not always the same thing.

Independent Postflight verification

After the coding agent exits, ELY performs its own verification step.

For Python projects, the current built-in verifier can run the project's unittest suite when a tests directory is available.

The final result can then be classified as:

VERIFIED

The independent verification passed.

FAILED

The agent failed, the project changed when it should not have, the Preflight checks became blocked, or the independent verifier failed.

NEEDS_REVIEW

ELY does not currently have enough automated verification for that project type.

I prefer this to pretending that every run can be automatically proven correct.

Sometimes the right answer really is:

this needs a human review.

Current flow


text
Preflight
  -> Project detection
  -> Skill selection
  -> Agent handoff
  -> Codex
  -> File integrity check
  -> Independent Postflight verification
  -> Run history

DEV Community: Ilya Golovenkov