Leaving Everything to AI Feels Risky. But Codex Became Much More Reliable Once I Put Up Some Guardrails
https://github.com/tosane932/sales_data_app
Introduction
This article was originally published in Japanese on Qiita and has been translated and adapted for DEV Community.
I currently work as a truck driver while teaching myself web application development using Python and Flask.
Recently, I've been using Codex in VS Code while working on my personal application, including:
- improving security
- strengthening pytest coverage
- validating database behavior
- checking migrations
- handling invalid input
When I read about AI-assisted coding, I often come across warnings like:
If you leave everything to AI, things can go badly wrong.
I think that's true.
But after using Codex for a while, I started to feel that there is a big difference between:
letting AI roam free
and:
defining its working area before assigning the task.
In this article, when I say "guardrails," I mean deciding not only what Codex should do, but also what it must not do and where it should stop.
I'm still a beginner in programming, but these are the six guardrails I've started using when working with Codex.
1. Define What It Is Allowed to Touch
The first thing I do is limit the scope of the task.
In other words:
What exactly are we changing this time?
For example:
- only validate input for the sales POST endpoint
- only handle rollback when a database save fails
- only validate POST requests for the product master
I try to keep each task focused on one small theme.
I also tell Codex:
If you find another problem outside the current scope, do not fix it automatically. Report it instead.
There is nothing wrong with Codex discovering another issue while investigating.
But if it keeps expanding the task and fixing additional problems along the way, I can eventually end up wondering:
What exactly changed?
So I try to separate:
discovering a problem
from:
fixing a problem.
2. Create a Failing Test First
Recently, I've increasingly asked Codex to write pytest tests before modifying the production code.
My basic workflow is:
- Reproduce the dangerous condition with a test
- Run the test before making the fix
- Confirm that the test fails
- Make the smallest necessary change
- Confirm that the target test passes
- Run the entire pytest suite again
For example, when I strengthened input validation for my product master, I added 17 invalid-input cases.
Before the fix:
17 failed
After the fix:
17 passed
By confirming that the test fails first, I can verify more than just:
"It seems safer now."
I can confirm that the problem actually existed, that the test reproduced it, and that the change closed the gap.
Recently, I've stopped thinking of pytest as just a way to confirm that the application works.
Instead, I think of it more like an:
incident-prevention log.
Once I discover a dangerous condition, I record it as a test so the application cannot silently return to that state later.
3. Don't Let It Directly Touch the Normal Database or Production
During testing, I generally avoid connecting Codex-driven tests directly to:
- the PostgreSQL database used during normal development
- the normal Docker database
- the actual Gemini API
- the production environment
For ordinary pytest runs, I separate the test database from the normal database and use a disposable in-memory SQLite database:
sqlite:///:memory:
For regular tests, this gives me a fast and disposable environment.
However, I do not assume SQLite can fully reproduce PostgreSQL-specific behavior.
When I need to verify PostgreSQL-specific behavior or migrations, I create a separate isolated PostgreSQL environment.
I separate things such as:
- containers
- networks
- databases
- volumes
from the normal environment.
My thinking is not:
"It's only a test, so it's probably fine."
Instead:
"Test in a place where failure cannot easily spread into the normal environment."
The goal is not simply to avoid using the production or normal database.
The real goal is:
to reduce the impact if something goes wrong.
4. Don't Let an Unexpected Problem Expand the Task Automatically
While Codex is investigating one issue, it sometimes discovers another unrelated problem.
In the past, I might have said:
Go ahead and fix that too.
Now I usually stop there.
I increasingly tell Codex:
If you discover an unexpected problem, do not expand the current scope. Stop and report it.
For example:
Current task:
Validate the sales POST endpoint
↓
Codex discovers another database issue
↓
Do not fix it immediately
↓
Report it as an unresolved issue
↓
Create a separate task for it later
This prevents a single change from becoming unnecessarily large.
For me:
being able to trace what changed is more important than fixing every discovered problem immediately.
5. Keep a Human Check Before Commit and Push
Even after the code changes are complete and the tests pass, I don't immediately let Codex commit or push.
First, I have it check things like:
pytest -v
git diff --check
git status --short
git diff --stat
git diff
Then I review the results and check:
- Were only the intended files modified?
- Did all pytest tests pass?
- Did unrelated changes get mixed in?
- Were any unexpected files added?
Only after that do I give permission:
"Everything looks good up to this point. You can commit."
Codex can write code for me, but I don't want it to automatically pass through the final Git gate.
After the commit, I also ask it to report:
- commit hash
- commit message
- push destination
- push result
- final
git status
Recently, I've also started thinking that relying only on a written instruction like:
"Do not push."
is not strong enough.
If I forget to include that instruction even once, the AI may interpret pushing as allowed.
So ideally, I want two layers of protection:
Restrict the action in the prompt
+
Restrict the actual permissions
If I also want to account for human mistakes, simply saying:
"Don't do this."
is weaker than creating an environment where the action is difficult or impossible without explicit permission.
6. Let pytest and CI Inspect the Result Too
After pytest passes locally, the relevant pushes and Pull Requests trigger another pytest run through GitHub Actions.
In my current setup, GitHub Actions runs when:
- code is pushed to
main - a Pull Request targeting
mainis created
In another article, I compared this process to truck inspections.
For me, the analogy looks like this:
- pytest: the inspection tool
- pytest tests: the inspection checklist
- local pytest: the pre-departure inspection
- GitHub Actions: the automated inspection at the shipping gate
The task isn't finished just because Codex says:
The fix is complete!
Instead, I try to make every change pass through several checkpoints:
Codex
↓
pytest
↓
git diff
↓
commit
↓
push
↓
GitHub Actions
I don't use the AI's own response as the final source of truth.
Instead:
I use other mechanisms to verify the work performed by the AI.
What Scares Me About Letting Codex "Roam Free"?
Imagine giving Codex a prompt like:
Make this application secure.
That's all.
Now the AI has to guess many things:
- How much is it allowed to change?
- Can it modify the database?
- Can it connect to production?
- Should it fix additional problems it discovers?
- Can it commit?
- Can it push?
- Can it change the specification?
That's a lot of decisions.
What worries me isn't Codex itself.
What worries me more is:
the human defining nothing and then delegating not only the work, but also all of the decisions to the AI.
Delegating work to AI and dumping everything on AI may look similar, but I think they are very different.
It Reminded Me of My Work as a Truck Driver
When I thought about it, this approach felt surprisingly similar to my main job as a truck driver.
Normally, you wouldn't tell a new driver:
Just deliver this cargo somehow.
You would confirm things such as:
- where they are going
- which route they should take
- the delivery time
- what they need to watch out for
- what they must not do
- what to do if something unexpected happens
And when something unusual occurs, there are situations where the right action is not to make an arbitrary decision and continue.
Sometimes the correct action is:
stop and confirm.
While working with Codex, I started thinking:
This is actually similar to assigning work to someone in the real world.
Don't Just Give AI a Task — Define the Area Where It Can Work
I'm still relatively new to programming.
That means I cannot always evaluate 100% of Codex's output by myself.
And that's exactly why I don't want to completely trust the AI and give it unlimited freedom.
Instead, I try to create an environment where:
even if something goes wrong, the damage is less likely to spread.
The same idea applies to the pytest improvements I've been working on.
My goal is not:
Be careful not to cause the same bug again.
My goal is:
If the application ever returns to the same dangerous state, pytest should stop it.
I think the same way about Codex.
I don't let Codex roam free.
- Define the scope
- Create failing tests first
- Keep it away from normal databases and production
- Make it stop when something unexpected appears
- Review changes before commit and push
- Inspect the result again with pytest and CI
I create those guardrails first, and then I let the AI work inside them.
And ideally, I don't want those guardrails to exist only in the prompt.
Where possible, I also want to enforce them through actual permissions and environment restrictions.
That's something I plan to pay more attention to going forward.
Letting Codex roam free still scares me.
But with clear guardrails, I've found it can be extremely useful.
That's what I've learned through my recent personal development work.
Top comments (0)