Hello from Japan 🇯🇵
I'm a truck driver in Japan, teaching myself web application development mainly with Python while continuing to work full-time.
In my personal Flask application, I've been gradually growing pytest into more than just a tool for checking whether the application works.
I think of it as an "incident prevention log":
a regression suite that helps stop previously discovered failures and near misses from silently returning.
This is Stage 5, the final stage of that series.
This article was originally published in Japanese on Qiita and has been translated and adapted for DEV Community.
https://github.com/tosane932/sales_data_app
87 GREEN Tests Still Had Five Blind Spots
The pytest suite originally contained only:
3 passed
From there, I strengthened it step by step:
Stage 1: 3 → 9
Stage 2: 9 → 51
Stage 3: 51 → 69
Additional demo seed tests: 69 → 71
Stage 4: 71 → 87
Stage 5 took a slightly different direction.
Up through Stages 1–4, the process had mainly been:
There might be a problem
↓
Write a pytest test
↓
RED
↓
Fix the problem
↓
GREEN
But once all 87 tests were GREEN, one question came to mind:
Can I really trust that GREEN?
If the tests themselves have blind spots, broken code may still remain GREEN.
So in Stage 5, I decided to do the opposite:
temporarily break working code on purpose and check whether pytest actually turns RED.
This time, I borrowed the idea of falsification: instead of assuming that GREEN meant the tests were strong, I tried to disprove that assumption with manual mutation testing.
The result was:
Starting point: 87 passed
Mutations performed: 11
Initially KILLED: 6
Initially SURVIVED: 5
Strengthen pytest for the 5 survivors
↓
Apply the same mutations again
↓
Confirm all 5 now turn RED
Final result:
91 passed, 2 known warnings
So even with 87 GREEN tests, five kinds of failures could still slip through.
This article focuses mainly on those five survivors.
pytest Improvement Series
Stage 1
https://qiita.com/tosane932/items/f3de1e190873a90de39fStage 2
https://qiita.com/tosane932/items/b91261e7103df5792f7dStage 3
https://qiita.com/tosane932/items/6d1ca5490979c8cf9d62Stage 4
https://qiita.com/tosane932/items/372270330e73583a227fStage 5
https://qiita.com/tosane932/items/85fd24c7baa6fe7c76a7
In Stage 5, I Started Questioning GREEN
When pytest is GREEN, it means:
the current implementation satisfied the conditions expressed by the tests that currently exist.
It does not mean:
GREEN = completely safe
For example, imagine this safe JavaScript code:
productName.textContent = item[0];
Suppose pytest only checks that:
textContent is used
Then someone adds:
productName.innerHTML = item[0];
directly after it.
The original textContent is still present.
So the test may still pass even though a dangerous HTML sink has been introduced.
In other words:
Safe code exists
and:
Dangerous code does not exist
are two different things.
Instead of only imagining this possibility, I decided to:
actually break the code and see what happened.
What Is Mutation Testing?
Mutation testing intentionally introduces small defects into otherwise working code.
For example, change:
if not product.is_active:
to:
if False:
Or take this query:
DailySales.query.filter_by(
product_id=product.id,
date=sale_date,
).first()
and remove:
date=sale_date
Then run pytest against the mutated code.
The result generally falls into two categories.
KILLED
A mutation is KILLED when introducing the defect causes pytest to turn RED.
Normal code
↓
GREEN
Mutation
↓
RED
That means the test suite can detect that failure.
SURVIVED
A mutation SURVIVES when pytest remains GREEN even after the defect is introduced.
Normal code
↓
GREEN
Mutation
↓
GREEN
That suggests the tests may not be able to distinguish:
the correct implementation from the broken implementation.
I did not introduce a dedicated mutation testing framework in this stage.
Instead, Codex and I selected representative defects and manually applied them:
one mutation at a time.
Because I Was Intentionally Breaking Working Code, I Defined Safety Rules First
Mutation testing means intentionally putting production code into a broken state.
So before starting, I defined strict boundaries:
Do not mutate main
Work only on feature/pytest-stage5
Only one mutation at a time
Do not connect to the production database
Do not connect to the development database
Do not connect to Render
Do not call the real Gemini API
Do not access external networks
Use only isolated pytest SQLite / temporary databases
Do not commit mutations
Do not push mutations
Always restore the original code after verification
If a mutation SURVIVED, I also did not start modifying pytest while the broken production code was still in place.
Every survivor followed this process:
Apply mutation
↓
Run pytest
↓
Confirm SURVIVED
↓
Restore normal code
↓
Add or strengthen pytest formally
↓
Confirm GREEN with normal code
↓
Apply the same mutation again
↓
Confirm RED
↓
Restore again
↓
Confirm GREEN with normal code
All five surviving mutations followed this same sequence.
The 11 Mutations
These are the 11 mutations I selected.
| # | Mutation | Initial Result | After Strengthening |
|---|---|---|---|
| 1 | Add innerHTML next to XSS-safe textContent
|
SURVIVED | KILLED |
| 2 | Disable CSRF protection | KILLED | - |
| 3 | Remove session fingerprint mismatch rejection | KILLED | - |
| 4 | Change year/month validation from or to and
|
KILLED | - |
| 5 | Disable discontinued-product validation | KILLED | - |
| 6 | Change the model UniqueConstraint
|
KILLED | - |
| 7 | Change the migration UniqueConstraint
|
KILLED | - |
| 8 | Remove the year condition from AI advice data retrieval |
SURVIVED | KILLED |
| 9 | Add the Jinja safe filter to the initial ranking |
SURVIVED | KILLED |
| 10 | Treat a session with a missing fingerprint as valid | SURVIVED | KILLED |
| 11 | Remove the date condition from sales lookup |
SURVIVED | KILLED |
The first run produced:
KILLED 6
SURVIVED 5
The six mutations that were KILLED immediately were already detected by existing pytest coverage, so I did not add new test functions for them.
The rest of this article focuses mainly on:
the five mutations that survived.
Six Mutations Were KILLED Immediately
The six mutations detected by the existing suite were:
- #2 Disable CSRF protection
- #3 Remove session fingerprint mismatch rejection
- #4 Break year/month validation
- #5 Disable discontinued-product validation
- #6 Break the model
UniqueConstraint - #7 Break the migration
UniqueConstraint
For example, in Mutation #2, I temporarily disabled CSRF protection.
Requests that should have been rejected with HTTP 400 were then able to reach application logic, and the existing tests turned RED.
For Mutation #5, I changed:
if not product.is_active:
to:
if False:
This allowed discontinued products to pass through sales processing.
The existing pytest suite detected that too.
Mutation #7 was especially interesting.
The migration originally had a unique constraint on:
product_id + date
I temporarily changed that to:
product_id + quantity
The migration itself still completed successfully.
But the pytest test that inspected the resulting schema detected that:
the required UniqueConstraint does not exist
and turned RED.
That confirmed another distinction:
"The migration completed successfully" and "the correct schema was created" are not the same thing.
SURVIVED #1: A Safe textContent Check Missed a Dangerous innerHTML
The first surviving mutation involved XSS protection.
The dynamic ranking displays product names using:
productName.textContent = item[0];
I temporarily added:
productName.innerHTML = item[0];
immediately after it.
So the mutated code looked like this:
productName.textContent = item[0];
productName.innerHTML = item[0];
The safe textContent line still existed.
But now the unsafe innerHTML assignment existed too.
I ran the existing XSS regression tests.
The result was:
GREEN.
The mutation SURVIVED.
Why Did It Survive?
The existing pytest test checked that:
textContent is used
But it did not check that:
innerHTML is not used
The test only proved:
Safe behavior exists
It did not prove:
Dangerous behavior does not also exist
So I strengthened the regression tests to also detect dangerous HTML sinks that could become XSS vectors when used with untrusted data, including:
innerHTMLouterHTMLinsertAdjacentHTML
After confirming GREEN with the normal implementation, I applied the exact same mutation again.
This time, pytest turned RED.
SURVIVED
↓
Strengthen assertion
↓
Apply same mutation again
↓
KILLED
The test count became:
87 → 88 tests
The first lesson was:
sometimes it's not enough to confirm that something safe exists. You also need to confirm that something dangerous does not exist.
The Second Wave Focused Only on Failures Existing pytest Might Miss
You can create almost unlimited mutations if you want to.
Change a condition.
Remove a return.
Disable exception handling.
Remove a database filter.
Add unsafe template behavior.
But the purpose of Stage 5 was not to maximize the number of mutations.
After the first wave, I inspected the code again without changing production code.
I selected four additional mutations where I thought:
The current fixture might not distinguish
correct code from broken code
The current assertion might miss this failure
This mutation might reveal something different
from the existing tests
Those became Mutations #8 through #11.
I decided these would be the final group for Stage 5.
The result:
all four SURVIVED.
This became the most important part of Stage 5 for me.
SURVIVED #8: Removing the year Filter Changed Nothing Because the Fixture Was Too Simple
The AI business-advice feature retrieves sales data for a selected year and month.
The normal code calls:
_get_sales_from_db(target_year, target_month)
I temporarily removed the year condition:
_get_sales_from_db(None, target_month)
So instead of asking for:
August 2026
the mutated code effectively asked for:
August from any year
I ran the existing pytest test.
The result:
GREEN.
The mutation SURVIVED.
The Problem Was the Fixture, Not the Production Code
At the time, the fixture only contained data for:
August 2026
July 2026
So even after removing the year condition:
month = August
still returned only the August 2026 data.
The correct code and the mutated code produced:
the same result with the existing fixture.
So I added:
August 2025
data to the fixture.
The product name was:
Previous-year August product
with quantity:
77
Then, when requesting August 2026, I strengthened the existing test with an assertion equivalent to:
assert "Previous-year August product" not in contents
Normal code:
GREEN.
Same mutation applied again:
RED.
The mutation was now KILLED.
Because I strengthened an existing test function instead of adding a new one, the total remained:
88 tests
What I Learned Here
Test data does not merely need to:
exist.
It needs enough discriminating power to produce different results between:
correct code
and:
broken code
A test can execute a line of code without proving:
that the condition on that line is actually necessary.
SURVIVED #9: Existing XSS Tests Missed Jinja |safe
The next mutation involved the dashboard's initial server-rendered ranking.
Normally, Jinja autoescapes:
{{ name }}
I temporarily changed it to:
{{ name | safe }}
Then I ran the existing XSS tests.
The result:
6 passed
The mutation SURVIVED.
Why Did It Survive Even Though I Already Had XSS Tests?
The existing tests focused mostly on:
- rankings generated later by JavaScript
- AI responses
- unsafe HTML sinks
But they did not directly verify:
the product name rendered by the server in the initial ranking.
So I inserted a product name like:
<em>HTML-like product name</em>
into the database.
If normal Jinja autoescaping is working, <em> should appear as text rather than becoming an actual HTML element.
The new pytest test verifies that:
- escaped text exists in the raw HTML
-
.prod-namedisplays the original product-name text - no
emelement appears inside.prod-name
Normal implementation:
GREEN.
Reapply the |safe mutation:
RED.
SURVIVED
↓
Add server-rendered XSS coverage
↓
KILLED
The test count became:
88 → 89 tests
This taught me that:
having something called an "XSS test" does not mean every rendering path is protected by that test.
SURVIVED #10: A Normal Login Fixture Could Never Produce a Missing-Fingerprint Session
The existing session tests verified whether an old session was rejected after administrator credentials changed.
But what happens if:
the fingerprint itself disappears from the session?
The normal implementation rejects a session if the fingerprint is missing.
So I introduced a mutation that treated a missing fingerprint as though it matched the current fingerprint.
Conceptually:
Fingerprint missing
↓
Pretend it matches the current fingerprint
The existing pytest tests returned:
2 passed
The mutation SURVIVED.
The Problem Was the "Normal Login" Fixture
The existing tests logged in normally.
A successful login naturally stores the fingerprint in the session.
So those tests never created:
a session where only the fingerprint is missing
For the new test, I performed a normal login and then used:
session_transaction()
to remove only the fingerprint from the session.
The Flask-Login _user_id remained.
Now the behavior differed:
Normal implementation:
redirect to login
Mutated implementation:
dashboard → 200
I confirmed GREEN with the normal code.
Then I applied the mutation again.
pytest turned RED.
KILLED.
89 → 90 tests
This showed me that:
fixtures created only through normal user flows may be unable to reproduce abnormal states.
SURVIVED #11: Removing the date Filter Changed Nothing Because There Was No Same-Product, Different-Date Data
This was the final mutation.
DailySales searches for the row to update with:
DailySales.query.filter_by(
product_id=product.id,
date=sale_date,
).first()
So both:
product
+
date
must match.
I temporarily removed the date condition:
DailySales.query.filter_by(
product_id=product.id,
).first()
Then I ran the existing normal sales POST tests.
PASS.
I also ran the database UniqueConstraint tests.
PASS.
The mutation SURVIVED.
Again, the Fixture Was the Problem
The existing fixture did not contain:
sales for the same product on another date.
If a product only has:
August 2 sale
then removing the date filter may still return that exact same row by coincidence.
So I created an existing sales record:
2026-08-01
quantity=5
Then I POSTed:
2026-08-02
quantity=9
With the normal implementation, the expected result is:
August 1 → quantity=5
August 2 → quantity=9
With the mutated implementation:
August 1 → quantity=9
The bug causes:
- the historical August 1 sale to be incorrectly changed from 5 to 9
- the new August 2 sale to never be created
So the new pytest test verifies both:
The August 1 record remains unchanged
and
The August 2 record is created
Normal implementation:
GREEN.
Same mutation reapplied:
RED.
KILLED.
The total became:
90 → 91 tests
All Four Mutations in the Second Wave Initially SURVIVED
The second-wave results were:
| Mutation | Initial Result | Why It Survived | After Strengthening |
|---|---|---|---|
#8 Remove year condition |
SURVIVED | No previous-year same-month fixture | KILLED |
| #9 Add Jinja `\ | safe` | SURVIVED | Initial server rendering wasn't directly tested |
| #10 Missing fingerprint | SURVIVED | No abnormal-session fixture | KILLED |
#11 Remove date condition |
SURVIVED | No same-product, different-date sales data | KILLED |
This really stayed with me.
By the end of Stage 4, the suite had already grown to:
87 passed
Yet all four mutations I deliberately selected because:
"This might still be weak"
passed straight through the suite.
More tests can create a greater sense of confidence.
But:
There are 87 tests
and:
Those 87 tests can distinguish important failures
are not the same statement.
In the End, All 11 Selected Mutations Became Killable
I performed 11 mutations.
The first run produced:
KILLED 6
SURVIVED 5
The five survivors were:
#1
#8
#9
#10
#11
For each of them, I followed the full process:
Restore mutation
↓
Formally add or strengthen pytest
↓
Confirm GREEN with normal code
↓
Apply the same mutation again
↓
Confirm RED
↓
Restore again
↓
Confirm final GREEN
So for the 11 representative mutations I selected, pytest was ultimately able to KILL all of them.
But I want to be careful about what that means.
It does not mean:
Mutation Score for the whole application = 100%
I only mutated:
11 representative cases that I selected manually.
pytest Grew from 87 to 91 Tests
At the beginning of Stage 5:
87 tests
At the end:
91 tests
Only four new test functions were added.
Since I performed 11 mutations, I initially thought the suite might grow more.
But:
- six mutations were already KILLED by existing tests
- #8 only required strengthening the fixture in an existing test function
- new test functions were added for #1, #9, #10, and #11
So:
87 + 4 = 91
I actually think that was a good result.
If my goal had been:
Reach 100 tests
I might have added tests simply to make the number larger.
But the purpose of Stage 5 was not:
to increase the test count.
It was:
to evaluate the failure-detection ability of the GREEN tests I already had.
From the Original Three Tests to About 30 Times as Many
Before I started strengthening pytest:
3 tests
By the end:
Stage 1 9
Stage 2 51
Stage 3 69
Demo seed 71
Stage 4 87
Stage 5 91
Numerically, that is roughly 30 times the original count.
But after Stage 5, what matters more to me is not:
that I have 91 tests.
It is:
that I can explain which failures cause those tests to turn RED.
Stage 5 Made No Formal Production-Code Changes
The only files with formal changes from Stage 5 were:
test_ai_integration.py
test_auth.py
test_sales.py
test_xss_regressions.py
There were no formal production-code changes:
app.py no formal diff
templates no formal diff
models no formal diff
migrations no formal diff
Every temporary mutation was restored.
So Stage 5 did not strengthen the application by permanently changing production code.
Instead:
it strengthened pytest's ability to detect defects.
Five Things I Learned from Stage 5
1. GREEN Can Still Miss Broken Code
Five mutations actually passed through the suite while it remained GREEN.
So:
pytest is GREEN
does not guarantee:
No broken code exists
GREEN means:
the current tests did not detect a violation of the conditions they express.
2. Fixtures Need Discriminating Power
Mutation #8 could not be detected without data from the same month in another year.
Mutation #11 could not be detected without sales for the same product on another date.
So a fixture should not merely be:
data that lets the test run
It may need to be:
data that distinguishes correct behavior
from broken behavior
3. Assertions Sometimes Need to Check Both "Exists" and "Does Not Exist"
Mutation #1 showed this clearly.
I already verified that:
textContent is used
But that assertion still passed after dangerous innerHTML was added next to it.
So I needed to test both directions:
Safe behavior exists
and:
Dangerous behavior does not exist
4. Normal Fixtures Cannot Always Represent Abnormal States
Mutation #10 involved a session where only the fingerprint was missing.
A normal login always creates that fingerprint.
So I had to explicitly construct:
an abnormal session state
Normal user-flow fixtures alone do not always reach the failure condition you actually need to test.
5. RED Does Not Automatically Mean the Mutation Was KILLED
During mutation testing, I did not classify something as:
KILLED
just because pytest became RED.
I checked whether:
the intended mutation caused the intended assertion to fail.
If the test failed for an unrelated reason, that would not prove that the mutation itself had been detected.
The Scariest Part Wasn't Breaking the Code — It Was Accidentally Leaving It Broken
The part I was most careful about was not the mutation itself.
It was:
accidentally leaving intentionally broken code behind.
So every mutation followed the rule:
Do not commit
Do not push
And every cycle included:
Mutation
↓
pytest
↓
restore
↓
git diff
↓
pytest with normal code
Mutation testing is often described as:
"break it and see if the tests catch it."
But for me, reliably restoring the code is part of the same operation.
Final Result
The final pytest result was:
91 passed, 2 warnings
The two warnings are the previously known:
Flask-SQLAlchemy get_engine() DeprecationWarning
from:
migrations/env.py
They were not introduced by Stage 5.
I'm treating them as separate maintenance work, so I did not fix them as part of this stage.
This Still Does Not Mean the Application Is Completely Safe
I performed 11 mutations and eventually strengthened the suite so that all 11 selected failures could be detected.
But I still cannot say:
This application is completely safe.
There are many areas outside the scope of this work, including:
- PostgreSQL-specific behavior
- locking
- concurrency
- race conditions
- simultaneous POST requests
- duplicate submissions
- differences from the Render environment
- production environment variables
- the real Gemini API
- API quotas
- timeouts
- SDK changes
- real-browser DOM behavior
- CSP
- specifications I have not decided yet
- future code
- failures I have not imagined yet
And I did not mutate the entire application exhaustively.
What I verified was narrower:
Can pytest actually detect these 11 representative types of failure that I selected?
It Feels a Little Like Testing a Truck's Safety Equipment
I work as a truck driver.
A truck has many safety systems.
But simply saying:
The safety equipment exists
does not mean accidents can never happen.
When a dangerous situation occurs:
does that equipment actually work the way it is supposed to?
And:
am I becoming overconfident simply because the equipment exists?
I started seeing something similar with pytest.
It is not enough to say:
pytest exists
CI exists
authentication exists
CSRF protection exists
database constraints exist
I also need to ask:
if one of those things breaks, can the system actually detect that something is wrong?
And even excellent safety equipment does not automatically make every human decision safe.
Years of experience do not guarantee perfect judgment either.
Familiarity can sometimes lead to:
This is probably fine.
Software may have a similar trap:
All pytest tests are GREEN,
so it's probably fine.
If I stop thinking there, I may miss something important.
Not "It's Probably Fine," but "What If?"
In Stage 5, I used a mindset from my day job almost directly.
Instead of:
"It's probably fine."
I try to think:
"What if?"
For example:
pytest is GREEN,
but it might still miss a failure
Dangerous behavior might be added
next to safe behavior
The fixture might be too simple
to distinguish broken code
from correct code
A migration might succeed
while producing the wrong schema
I might only be testing normal login
and never checking an abnormal session
I thought through those possibilities and then:
introduced small defects to test them for real.
That was Stage 5.
Summary
Stage 5 moved the suite from:
87 passed
↓
91 passed
But the important part was not the additional four tests.
I introduced 11 representative defects.
The first run produced:
Initially KILLED 6
Initially SURVIVED 5
Investigating the five survivors showed that the problem was not always simply:
Not enough tests
The blind spots included:
Assertions that only checked one direction
Fixtures without enough comparison data
Server-rendered output not being tested
No abnormal-session fixture
No different-date sales data for the same product
For all five survivors, I completed this sequence:
Restore normal code
↓
Add or strengthen pytest
↓
GREEN with normal code
↓
Apply the same mutation again
↓
RED
↓
Restore again
↓
Final GREEN
So in the end:
pytest could KILL all 11 mutations selected for this stage.
Stage 1 began with only three pytest tests and the idea of turning them into a regression suite.
Stage 2 expanded into validation, database updates, and rollback behavior.
Stage 3 expanded into authentication, CSRF protection, and access control.
Stage 4 looked ahead to failures that had not happened yet, including migrations, AI-service failures, and session problems.
And Stage 5:
questioned the GREEN result itself.
The biggest lesson for me was:
GREEN ≠proof of safety
GREEN means:
the current implementation satisfied the conditions expressed by the current tests.
That is why I think there is value in occasionally turning the question around:
Can this test actually detect the failure
it is supposed to detect?
pytest started with three tests.
It ended this series with:
91 tests
But now I care more about:
being able to explain what I can break to make those tests turn RED
than simply saying:
"I have 91 tests."
For me, this marks the end of Stage 5.
What began as a small list of normal behavior checks gradually became an:
"incident prevention log" that I also crash-tested to see whether it could actually detect failures.
Top comments (0)