Hello from Japan 🇯🇵
I'm a truck driver in Japan, teaching myself web application development with Python and Flask while continuing to work full-time.
This article is part of my ongoing record of strengthening pytest in my personal Flask project.
This article was originally published in Japanese on Qiita and has been translated and adapted for DEV Community.
https://github.com/tosane932/sales_data_app
Introduction
Recently, I've been using Codex in VS Code while strengthening pytest in a Flask application I'm building.
I'm currently working on Stage 3.
This stage focuses on authentication, authorization, CSRF protection, and questions such as:
"Who is allowed to perform this operation?"
and:
"Was this action actually intended by the user?"
While working with Codex on this, a strange analogy suddenly came to mind:
Isn't this a bit like a dog eating random things off the ground during a walk? 🦮
In Japanese, there's a word for this: 拾い食い (hiroi-gui).
It describes a dog finding something on the ground and eating it without waiting for its owner.
And the more I worked with Codex, the more similar it started to feel.
Codex Finds Problems I Didn't Ask It to Find
When I ask Codex to investigate code, it sometimes finds problems outside the scope of the current task.
For example, even if I say:
Only investigate authentication this time.
Codex may respond with things like:
There is also a problem in another API.
This process could also be improved.
This part probably needs to be fixed as well.
That isn't a bad thing.
In fact:
having it discover problems that I didn't notice myself is extremely useful.
But the situation changes when it goes one step further and says:
I fixed that while I was at it.
That's where I become cautious.
"While I'm Here, I'll Fix This Too" Started Looking Like a Dog Eating Things Off the Ground
This is the analogy that came to mind while working with Codex.
Imagine a dog out for a walk:
Finds something on the ground
↓
Gets curious
↓
Puts it in its mouth without asking the owner
That's the behavior I'm talking about.
Codex can sometimes behave similarly:
Investigates the code
↓
Finds another problem
↓
Decides it looks worth fixing
↓
Fixes it while it's there
From its perspective, it may feel like:
"I did something helpful! 🐶"
But from the owner's perspective:
Wait, wait!
Where did you get that?
You can't just eat random things!
And with Codex, my reaction becomes:
Wait, wait!
I didn't ask you to touch that this time!
Don't expand the scope on your own!
Finding Problems Is Fine ⭕ — Fixing Them Without Asking Is Not ✖
Recently, I've started making my instructions to Codex much more explicit.
For example:
If you discover an unexpected problem,
do not expand the scope of the current task.
Report the problem only and stop.
The idea is simple:
Finding a problem is fine.
Fixing it without permission is not.
When a new issue is discovered, I want the flow to look like this:
Problem discovered
↓
Report it
↓
Stop the current work
↓
Human reviews it
↓
If necessary, create a separate task
This lets me separate:
discovering a problem
from:
deciding to fix it.
Why Are Unplanned Fixes Risky?
At first glance, you might think:
If you've already found the problem, wouldn't it be faster to fix it at the same time?
I thought the same thing at first.
But after working through several changes, I started seeing some problems with this approach.
The Scope of the Change Becomes Hard to Understand
Imagine that the original task was:
Add authentication
But the final change includes:
Authentication
API fixes
Database changes
Template changes
Logging improvements
Now it becomes much harder to understand:
which change was made for which purpose.
When Something Breaks, Finding the Cause Becomes Harder
If there is only one change, I can reasonably think:
This change is probably what caused the problem.
But if several unrelated fixes are mixed together:
Which change caused it?
becomes much harder to answer.
The Meaning of the Tests Also Becomes Less Clear
Recently, I've been thinking of pytest as an:
"incident prevention log."
When I discover one problem, I try to follow this process:
Reproduce the dangerous state with a test
↓
Confirm that the test fails
↓
Make the smallest necessary fix
↓
Confirm that the test passes
But if I also fix unrelated problems in the middle of that process, it becomes harder to tell:
which code change was actually intended to make which failing test pass.
I Used "Wait 🖐" During pytest Stage 3 Too
In the current Stage 3 of my pytest work, I'm focusing on authentication-related behavior.
First, without changing any production code, I investigated the current state of:
Authentication
Authorization
CSRF
Then I added a future requirement as a failing test:
Reject unauthenticated POST requests.
I also added login behavior as another failing test.
At that point, I confirmed:
5 failed
Only after that did I move on to the smallest authentication implementation needed.
After the fix:
56 passed
During this process, I kept the scope deliberately narrow:
Authentication only this time
Do not touch CSRF yet
Do not touch logout yet
Do not protect the APIs yet
Even when Codex could clearly continue into the next area:
I stopped it.
"It Can Do It" and "It Should Do It Now" Are Different
One thing became especially clear to me during this stage:
what Codex can do and what Codex should do right now are two different things.
For example:
It can implement CSRF protection.
It can add logout.
It can protect the APIs.
But that does not mean:
Do all of them now.
The human still needs to decide:
This is where today's task ends.
I think this applies far beyond programming.
git status and git diff Are Like Checking the Dog's Mouth After a Walk
Imagine your dog comes home from a walk.
You might wonder:
Did you eat anything strange out there?
After Codex finishes working, I do something similar by checking the Git state and diff.
For example:
git status --short
git diff --stat
git diff
git diff --check
I use these to check:
- whether only the requested files were modified
- whether unexpected files appeared
- whether unrelated fixes were mixed in
- whether there are whitespace errors
In my head, it looks something like this:
Codex:
"I finished the walk properly! 🐶"
git diff:
"Okay... let me see what's in your mouth."
Even if the AI reports that there were no changes outside the requested scope, I still check the diff myself.
pytest Is Not a Tool for Checking Whether the Dog Ate Something
I run pytest too.
But pytest is not primarily a tool for checking whether Codex made unauthorized changes.
What pytest checks is:
Did we break behavior that the application was already supposed to preserve?
So these checks have different jobs:
Task scope
↓
git diff
↓
pytest
In my head:
Task scope
= deciding the walking route
Stop on unexpected issues
= don't eat random things
git diff
= checking after the walk
pytest
= checking existing rules and system health
Each one protects a different part of the workflow.
I Don't Let Codex Roam Free
I previously wrote another article about not letting Codex "roam free."
👉 Don't Let Codex Roam Free: 6 Guardrails I Use for AI-Assisted Coding
This idea of preventing it from "eating things off the ground" is really an extension of the same approach.
Codex can work very quickly.
That also means it can move through a sequence like this very quickly:
Investigate
↓
Discover a problem
↓
Fix it
↓
Discover another problem
↓
Fix that too
That's exactly why I think:
the faster an AI can move, the more important it is to define where it must stop.
An Instruction I Use More Often Now
Recently, I've started giving Codex instructions like this:
If you discover an unexpected problem,
do not expand the scope of the current task.
Report it and stop.
Do not continue to the next task.
Do not commit or push.
That may sound overly cautious.
But it isn't really because I distrust Codex.
It's more that:
because Codex can move so quickly, I want its working boundaries to be explicit.
Summary
While working through pytest Stage 3 with Codex, I realized:
Codex making unplanned "while I'm here" fixes feels a little like a dog eating random things off the ground. 🐩
My current rules for preventing that are:
- define the scope of the current task
- report unexpected problems instead of fixing them automatically
- separate discovery from fixing
- don't let the AI move into the next task on its own
- inspect the actual changes with
git diff - use pytest to verify that existing behavior still works
The most important rule is not:
"Don't find problems."
Actually, I want Codex to find them.
What I really mean is:
Find them.
Just don't eat them without asking.
Codex is a very capable dog. 🐾
And for now, I've found that I work best with it when I'm able to say:
Wait.
We're not going there yet.
Next, I plan to continue Stage 3 by working on CSRF protection.
Top comments (0)