DEV Community

Cover image for 📝 Growing pytest into an “Incident Prevention Log” — Stage 4: Testing Failure Scenarios (71 ➡ 87 Tests)
tosane932
tosane932

Posted on Originally published at qiita.com

📝 Growing pytest into an “Incident Prevention Log” — Stage 4: Testing Failure Scenarios (71 ➡ 87 Tests)

#python #flask #pytest #security

Hello from Japan 🇯🇵

I'm a truck driver in Japan, teaching myself web application development mainly with Python while continuing to work full-time.

This article is Stage 4 of my ongoing effort to strengthen pytest in a Flask application I'm building.

Instead of only adding more tests for normal behavior, Stage 4 focuses on questions like:

"What if the input is invalid?"

"What if the external AI service goes down?"

"What if authentication settings change after login?"

It is similar to the kind of hazard anticipation I use in my day job as a truck driver: thinking about what might happen before it becomes an accident.

This article was originally published in Japanese on Qiita and has been translated and adapted for DEV Community.

I used Codex to help with investigation and implementation.

Some of the internal behavior became complicated enough that there were points where my own understanding did not fully keep up.

So rather than pretending to explain internal details I don't completely understand, this article focuses on:

what I considered a problem, what I checked, and what the result was.

https://github.com/tosane932/sales_data_app

pytest Improvement Series

Introduction

I've been gradually strengthening pytest in my personal Flask application.

The test suite originally had only three tests.

It grew through the previous stages:

  • Stage 1: 3 → 9 tests
  • Stage 2: 9 → 51 tests
  • Stage 3: 51 → 69 tests
  • additional demo seed tests: 69 → 71 tests

So at the beginning of Stage 4:

71 passed
Enter fullscreen mode Exit fullscreen mode

By this point, I wanted to check more than:

"Does the application work correctly under normal conditions?"

I also wanted to ask:

"When something abnormal happens, does the application fail safely?"

In my work as a truck driver, I constantly think about things that might happen:

  • someone might suddenly step or drive out
  • cargo might shift or collapse
  • a delivery mistake might happen
  • a safety device might not work correctly

The idea is to anticipate danger before an accident occurs.

I approached Stage 4 of pytest in much the same way.

Instead of:

"Fix the accident after it happens."

I wanted to move toward:

"Find the places where an accident might happen before it does."

That became the theme of Stage 4.

Stage 4 Results

By the end of this stage:

71 tests
↓
87 tests
Enter fullscreen mode Exit fullscreen mode

I added 16 tests.

Breakdown of the 16 New Tests

Category Added
Empty-database migration 1
Non-integer dashboard query parameters 6
Gemini failure fallbacks 3
Authenticated AI advice happy path 1
Fail-closed sessions 2
Invalid CSRF tokens 3
Total 16

The final result was:

======================= 87 passed, 2 warnings in 15.65s ========================
Enter fullscreen mode Exit fullscreen mode

1. Can Migrations Build the Database from Nothing?

The first thing I checked was the Alembic migration history.

Instead of testing only against a database that already existed, I wanted to verify:

"Can the current database schema be rebuilt from a completely empty database using only the migration history?"

The test creates an isolated SQLite database under tmp_path and runs:

Empty database
↓
Alembic base
↓
upgrade head
↓
current schema
Enter fullscreen mode Exit fullscreen mode

The test verifies:

  • products
  • daily_sales
  • alembic_version
  • required columns
  • the composite unique constraint on daily_sales(product_id, date)
  • the database revision matches the current Alembic head

I did not hard-code the revision ID into the test.

Instead, the test dynamically checks the current Alembic head.

This test verifies that the migration chain can rebuild the expected schema in an isolated SQLite database. It does not replace PostgreSQL-specific migration testing.

If I compare this to a company, this isn't asking:

"Can the headquarters that already exists keep operating?"

It's asking:

"If we start from an empty lot, can we build a new branch correctly using only the blueprints?"

Result:

1 passed
Enter fullscreen mode Exit fullscreen mode

2. Invalid year / month Should Return 400, Not 500

Next, I investigated dashboard-related query parameters.

The affected routes were:

/dashboard
/api/dashboard-data
/api/ai-advice
Enter fullscreen mode Exit fullscreen mode

At the time, values such as:

?year=abc
Enter fullscreen mode Exit fullscreen mode

or:

?month=abc
Enter fullscreen mode Exit fullscreen mode

were passed directly to int().

That meant a ValueError could occur and potentially result in an HTTP 500 response.

So I defined the expected behavior as:

non-integer year or month values should be rejected with HTTP 400 Bad Request.

I added:

3 routes × year/month
= 6 cases
Enter fullscreen mode Exit fullscreen mode

All six were RED at first:

Expected: 400
Actual:   500
Enter fullscreen mode Exit fullscreen mode

I then added a small helper that converts query parameters to integers and calls abort(400) when the value is invalid.

Result:

6 passed
Enter fullscreen mode Exit fullscreen mode

For the AI advice API, I also verified that invalid query parameters are rejected before the Gemini client is called.

3. Locking Down Gemini Failure Fallbacks

This application uses the Gemini API.

But Gemini is an external service, so failures can happen:

  • 429
  • 503
  • other exceptions

The application already had fallback behavior for these cases.

Stage 4 added regression tests to preserve that behavior.

I tested:

429
→ fallback for rate limiting

503
→ fallback for service unavailable

other exception
→ general fallback
Enter fullscreen mode Exit fullscreen mode

I did not connect to the real Gemini API.

All external calls were mocked.

These tests did not uncover a new bug.

Instead, the goal was:

to record an existing safety mechanism as regression tests so future changes don't accidentally remove it.

Result:

3 passed
Enter fullscreen mode Exit fullscreen mode

In company terms, this is like asking:

"If one of our external business partners goes offline, does our own system stay standing?"

4. Testing the Authenticated AI Advice Happy Path

I also added one test for the normal AI advice flow.

I wanted to verify that the different parts of the system still worked together correctly when nothing was failing.

The flow was:

Authentication
↓
query processing
↓
SQLite database aggregation
↓
prompt generation
↓
mock Gemini
↓
JSON response
Enter fullscreen mode Exit fullscreen mode

For example, suppose the database contains:

August 2026

Product A: 10
Product B: 5
Enter fullscreen mode Exit fullscreen mode

The test verifies that:

  • Product A and Product B for August are included in the prompt
  • products from another month are not included
  • the mocked Gemini response is returned as JSON

Result:

1 passed
Enter fullscreen mode Exit fullscreen mode

5. Fail Closed: Don't Trust an Old Session After Authentication Settings Change

This was the part of Stage 4 where I spent the most time experimenting.

The theme was:

fail-closed behavior.

Fail-closed means:

when something is wrong or uncertain, default to the safer state.

A traffic signal is an easy analogy.

If the signal system breaks, you don't want:

"Something is wrong, so let's assume green."

You want:

"Something is wrong, so stop."

The First Problem I Found

I found that an existing authenticated session might still restore the administrator even after:

ADMIN_PASSWORD_HASH
Enter fullscreen mode Exit fullscreen mode

had been changed to an invalid or different value.

So I created a RED test:

Successful login
↓
change ADMIN_PASSWORD_HASH
↓
GET /dashboard
↓
expect 302 redirect to /login
Enter fullscreen mode Exit fullscreen mode

But then another problem appeared.

The RED Test Wasn't Following the Path I Thought It Was

In the test environment, the application context remained alive longer than I expected.

That meant:

g._login_user
Enter fullscreen mode Exit fullscreen mode

still contained authentication information from the previous request.

I thought I was testing:

session
↓
load_user()
Enter fullscreen mode Exit fullscreen mode

but the request was actually using:

previous request's g._login_user
↓
still authenticated
Enter fullscreen mode Exit fullscreen mode

So I learned something important:

A test being RED does not automatically mean it is RED for the reason you intended.

I changed the test so that the cached authentication state from the previous request was removed.

I also used a spy to confirm that:

load_user("admin")
Enter fullscreen mode Exit fullscreen mode

was actually called again.

Even then:

Expected: 302
Actual:   200
Enter fullscreen mode Exit fullscreen mode

This time, I had confirmed the real RED condition.

One lesson that stood out during this work was:

It's not enough to ask whether GREEN is GREEN for the right reason. RED also needs to be RED for the right reason.

My First Fix Was a Roughly 60-Line Hash Parser

My first implementation tried to validate whether a Werkzeug password hash itself had a valid format.

I wrote a helper that parsed things such as:

  • scrypt
  • pbkdf2
  • digest
  • parameters

The tests passed.

But after reviewing the implementation, another problem became obvious:

it was far too complicated for what I was actually trying to achieve.

Now the application itself would need to understand and maintain Werkzeug's password-hash formats.

If Werkzeug added or changed supported formats in the future, my own parser might also need to change.

So I stopped and redesigned the approach.

Switching to a Fingerprint

The real question I wanted to answer was not:

"Is the current password hash in a valid format?"

It was:

ADMIN_PASSWORD_HASH at login
==
Current ADMIN_PASSWORD_HASH
Enter fullscreen mode Exit fullscreen mode

So instead, I generate a SHA-256 fingerprint from ADMIN_PASSWORD_HASH when login succeeds and store that fingerprint in the session.

Conceptually:

Successful login
↓
record a "fingerprint" of the current ADMIN_PASSWORD_HASH
↓
next request
↓
generate a fingerprint from the current ADMIN_PASSWORD_HASH
↓
match    → continue
mismatch → require login again
Enter fullscreen mode Exit fullscreen mode

Only the fingerprint is stored in the session.

I do not store:

  • the password hash itself
  • the real password
  • a plaintext password

The fingerprint logic was separated into a small helper.

In company terms:

instead of storing the key itself, I added someone who checks the key's fingerprint.

The final two tests verify:

ADMIN_PASSWORD_HASH changed
→ reject existing session

ADMIN_PASSWORD_HASH unchanged
→ restore session normally
Enter fullscreen mode Exit fullscreen mode

Result:

2 passed
Enter fullscreen mode Exit fullscreen mode

6. Can the Application Reject a Tampered CSRF Token?

The final area I checked was CSRF protection.

The existing pytest suite already verified:

No CSRF token
→ reject the POST request
Enter fullscreen mode Exit fullscreen mode

But I had not yet tested:

a token that exists but has been modified or forged.

Using an airport analogy:

No passport
→ rejected
Enter fullscreen mode Exit fullscreen mode

is not the only case worth checking.

I also wanted:

Forged passport
→ rejected
Enter fullscreen mode Exit fullscreen mode

For these tests, I did not generate an entirely fake token from scratch.

Instead:

  1. retrieve a valid CSRF token using the same client
  2. modify only the first character
  3. send the POST request

I added three tests.

Login 

Correct username/password
+
tampered CSRF token
↓
400
↓
authentication not created
Enter fullscreen mode Exit fullscreen mode

Product Registration 

Tampered CSRF token
↓
400
↓
Product unchanged
DailySales unchanged
Enter fullscreen mode Exit fullscreen mode

Sales Registration 

Tampered CSRF token
↓
400
↓
DailySales unchanged
Enter fullscreen mode Exit fullscreen mode

Result:

3 passed
Enter fullscreen mode Exit fullscreen mode

All three tests were GREEN immediately because Flask-WTF was already rejecting the tampered tokens correctly.

So this wasn't a case where I added a new security feature.

Instead:

I added evidence to the regression suite that an existing security checkpoint was actually working.

Final pytest Result

After all Stage 4 changes, I ran:

pytest -v
Enter fullscreen mode Exit fullscreen mode

The result:

collected 87 items

...

======================= 87 passed, 2 warnings in 15.65s ========================
Enter fullscreen mode Exit fullscreen mode

There were no failures or errors.

Stage 4 Commit List

bb8b503 test: cover empty database migrations
a1a5578 fix: reject invalid dashboard query parameters
b36ad7a test: cover Gemini failure fallbacks
757ef6f test: cover authenticated AI advice route
412fcb3 fix: invalidate sessions when admin credentials change
26ae4f8 test: cover invalid CSRF tokens
Enter fullscreen mode Exit fullscreen mode

Verifying Everything Again with GitHub Actions

After confirming all 87 tests were GREEN locally, I used the following flow:

feature/pytest-stage4
↓
Pull Request
↓
GitHub Actions
↓
main
Enter fullscreen mode Exit fullscreen mode

In the Pull Request, I confirmed:

All checks have passed
Enter fullscreen mode Exit fullscreen mode

and verified that there were no conflicts before merging.

After the merge, I updated my local main branch:

git switch main
Enter fullscreen mode Exit fullscreen mode 
git pull
Enter fullscreen mode Exit fullscreen mode

Then, as an additional check, I ran:

git merge-base --is-ancestor feature/pytest-stage4 main \
  && echo "OK: feature/pytest-stage4 is fully included in main" \
  || echo "NG: main still has missing commits"
Enter fullscreen mode Exit fullscreen mode

The result:

OK: feature/pytest-stage4 is fully included in main
Enter fullscreen mode Exit fullscreen mode

The SHA values for main and origin/main also matched.

There were no file differences between the feature branch and main.

Finally, after the merge, I ran:

pytest -v
Enter fullscreen mode Exit fullscreen mode

again on main.

The final result remained:

87 passed, 2 warnings
Enter fullscreen mode Exit fullscreen mode

In company terms, I think of this as:

Build the safety equipment in the test department

↓

Send it through headquarters review

↓

Approve it for official use

↓

Run all 87 safety checks again at headquarters

The Two Remaining Warnings

There are still two warnings during the migration tests:

DeprecationWarning:
'get_engine' is deprecated and will be removed in Flask-SQLAlchemy 3.2.
Enter fullscreen mode Exit fullscreen mode

They originate from:

migrations/env.py
Enter fullscreen mode Exit fullscreen mode

where get_engine() is still used.

The new migration test made these warnings more visible, but they were not introduced by the Stage 4 changes.

I am aware of these warnings, but I deliberately left them outside the scope of Stage 4.

Instead of fixing something just because I happened to notice it, I prefer to treat unrelated changes as separate tasks.

What I Learned from Stage 4

The test suite grew from:

71
↓
87
Enter fullscreen mode Exit fullscreen mode

But once again, the number itself was not the most important part.

The fail-closed work in particular followed this path:

1. Write a RED test
2. Confirm RED
3. Investigate why it is RED
4. Discover a problem in the test fixture
5. Fix the RED test
6. Confirm the real RED condition
7. Implement a fix
8. Decide the implementation is too complicated
9. Throw away the first implementation
10. Redesign it using a simpler fingerprint approach
Enter fullscreen mode Exit fullscreen mode

That experience made me think beyond:

"Does the code work?"

I also needed to ask:

"Do I really want to maintain this implementation in the future?"

And pytest has become easier for me to understand when I think of it not merely as a behavior check, but as:

a mechanism for checking dangerous places before an incident happens.

That feels very similar to hazard anticipation in logistics.

Instead of waiting for danger to appear, you constantly ask:

"What might happen next?"

Next: Stage 5 Will Be a Crash Test

I plan for Stage 5 to be the final stage of this pytest improvement series.

Up to this point, most of the work has been about:

adding tests and building safety mechanisms.

Stage 5 reverses the question:

"Can those safety mechanisms actually detect an accident?"

On a feature branch, I plan to temporarily introduce small intentional defects into the code and check whether pytest correctly turns RED.

Something like:

Intentionally introduce a small defect
↓
Run pytest
↓
Confirm the appropriate test turns RED
↓
Immediately restore the original code
Enter fullscreen mode Exit fullscreen mode

This is similar to a manual form of mutation testing.

The intentionally broken code will not be committed or pushed.

I will always restore the original implementation before moving to the next check.

This is not something I will perform in the production environment.

At the end of Stage 1, the suite had only 9 tests.

At the end of Stage 4:

87 tests
Enter fullscreen mode Exit fullscreen mode

In the final stage, the question will no longer be:

"Do I have 87 tests?"

It will be:

"Can those 87 tests actually detect the failures they are supposed to catch?"

pytest Improvement Series

https://qiita.com/tosane932

https://github.com/tosane932/sales_data_app

Top comments (0)