✅️ A Truck Driver Built a “Pre-Departure Inspection Gate” — Then Found a Hole in His Tests

🍞 What Went Wrong When I Introduced pytest and GitHub Actions into sales_data_app

https://github.com/tosane932/sales_data_app

Introduction

Hello from Japan! 🇯🇵

I am tosane932, a professional truck driver working in logistics while teaching myself Python.

In my previous article, I tested a Docker multi-stage build and measured the actual image-size difference.

At the end of that article, I said that my next topic would be pytest and CI/CD.

This is the practical follow-up.

I will share the conclusion first:

I wrote some tests, felt satisfied, and then discovered that they were allowing an important inconsistency to pass unnoticed.

This article records what happened and what I learned from it.

---

The Current Deployment Flow—and Its Risk

sales_data_app is connected to Render.

Whenever I run:

git push

Render automatically builds and deploys the latest version.

This is convenient.

However, without an automated test stage, buggy code can also travel directly into production.

In logistics terms, it was like sending a truck onto the road without performing a pre-departure inspection.

I decided to add an inspection gate using:

• pytest
• GitHub Actions

The goal was to test the code automatically before treating a push as safe.

---

Writing My First pytest Tests

sales_data_app contains a function called:

build_sales_prompt

This function creates the prompt sent to the Gemini API.

I started with three simple tests.

# test_prompts.py

from prompts import build_sales_prompt


def test_build_sales_prompt_includes_sales_summary():
    sample_data = "Monday: 50 melon buns, 30 red bean buns"

    result = build_sales_prompt(sample_data)

    assert sample_data in result


def test_build_sales_prompt_includes_key_instructions():
    result = build_sales_prompt("Test data")

    assert "3 suggestions" in result
    assert "bullet points" in result


def test_build_sales_prompt_returns_string():
    result = build_sales_prompt("Test data")

    assert isinstance(result, str)

When I ran the tests, all three passed.

test_prompts.py::test_build_sales_prompt_includes_sales_summary PASSED
test_prompts.py::test_build_sales_prompt_includes_key_instructions PASSED
test_prompts.py::test_build_sales_prompt_returns_string PASSED

At first glance, the function appeared to be protected.

---

I Deliberately Broke the Code—and Found a Hole in the Test

The prompt-generation function contained two instructions using the same number.

# prompts.py

def build_sales_prompt(sales_summary: str) -> str:
    return f"""
You are a management consultant with expertise in food-industry
trends and consumer behavior.

Based on the bakery sales data below, provide three specific
suggestions that staff can immediately apply in the workplace.

[Sales Data]
{sales_summary}

[Analysis Points]
- Recent food trends and changes in competing businesses
- Needs of different customer groups
- Potential for promotion on social media
- Hidden demand based on weekdays, seasons, and holidays

[Output Format]
Provide three bullet-point suggestions.
Keep each suggestion to one or two sentences.
"""

The number three appeared in two different contexts:

1. “Provide three specific suggestions”
2. “Provide three bullet-point suggestions”

I wanted to know whether my test was actually protecting both instructions.

So I deliberately changed the prompt.

Pattern 1: Change Only the Lower Instruction

I changed:

Provide three bullet-point suggestions

to:

Provide five bullet-point suggestions

The upper instruction still requested three suggestions.

Result

PASSED

The inconsistency passed through the test.

---

Pattern 2: Change Both Instructions to Five

I changed both instructions from three to five.

Result

FAILED

The test detected the change.

---

Pattern 3: Keep the Upper Instruction at Five and Restore the Lower One to Three

The prompt now contained:

Provide five specific suggestions

and:

Provide three bullet-point suggestions

Result

PASSED

Once again, the inconsistent prompt passed through the test.

---

Why the Original Test Failed to Protect the Prompt

The problem was this assertion:

assert "3 suggestions" in result

This test only asked:

Does the relevant text appear somewhere in the complete prompt?

As long as one matching phrase remained, the assertion passed.

It did not check whether both instructions were correct or whether they contradicted each other.

The test was verifying the existence of a fragment, not the meaning of the prompt structure.

This taught me an important distinction:

Writing a test and protecting the intended behavior are not the same thing.

A green test result may only mean that the assertion was too weak to detect the problem.

---

Improving the Test

Once I understood the problem, I changed the test to check each instruction in its full context.

def test_build_sales_prompt_includes_key_instructions():
    result = build_sales_prompt("Test data")

    assert "provide three specific suggestions" in result.lower()
    assert "provide three bullet-point suggestions" in result.lower()

Now the test protects both locations independently.

When I repeated the same deliberate changes, the inconsistent versions failed correctly.

AssertionError:
assert 'provide three bullet-point suggestions'
in '...provide five bullet-point suggestions...'

The failure message also became more useful.

It clearly showed:

• What the test expected
• What the prompt actually contained
• Which instruction had changed

The test now had a more specific purpose.

---

What Mutation Testing Taught Me

I did not use a formal mutation-testing tool in this experiment.

However, the basic idea was similar:

1. Deliberately change the production code
2. Run the tests
3. Check whether the tests detect the change

If the tests still pass after an important behavior is broken, the test suite may not be protecting that behavior.

This was much more revealing than simply running the tests against correct code.

A test that passes only tells me:

The current code satisfies the current assertion.

A deliberately broken version can reveal:

Whether the assertion would notice the failure I actually care about.

---

Building an Automatic Inspection Gate with GitHub Actions

After improving the test, I created a GitHub Actions workflow so the tests would run automatically on every push and pull request.

I created:

.github/workflows/test.yml

with the following configuration:

name: Run Tests

on:
  push:
    branches:
      - main

  pull_request:
    branches:
      - main

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          pip install -r requirements.txt

      - name: Run pytest
        run: |
          pytest test_prompts.py -v

The intended flow was:

Push code to GitHub
        ↓
GitHub creates a clean virtual environment
        ↓
Install dependencies
        ↓
Run pytest
        ↓
Display success or failure

This was my software version of a pre-departure inspection gate.

---

The First GitHub Actions Run Failed

I pushed the workflow and waited for the result.

The job failed.

The error included:

exit code 127

This usually means that the requested command could not be found.

The cause became clear quickly.

I had installed pytest directly in my local environment:

pip install pytest

However, I had not added it to:

requirements.txt

GitHub Actions starts from a clean environment every time.

It only installs what is explicitly listed in the project configuration.

My local computer knew about pytest.

The CI environment did not.

This was another example of:

It works locally.

not meaning:

It can be reproduced anywhere.

---

Adding pytest to the Dependencies

I added pytest to requirements.txt.

echo "pytest" >> requirements.txt

Then I committed and pushed the change again.

This time, GitHub Actions displayed a green check mark.

Success

The automatic inspection gate was now running successfully.

---

What the Clean CI Environment Revealed

The local environment had accumulated packages from previous development work.

Because of that, missing dependency declarations could remain hidden.

GitHub Actions exposed the problem by creating a clean environment.

The difference looked like this:

Local environment:
pytest was already installed
        ↓
The command worked
        ↓
The missing dependency declaration stayed hidden

GitHub Actions:
Clean environment
        ↓
Install only requirements.txt
        ↓
pytest was missing
        ↓
Command not found

The failure was inconvenient, but useful.

It proved that CI is not only a place to run tests.

It also checks whether the project can be reconstructed from its declared configuration.

---

A Test Gate Is Only as Good as Its Inspection Items

At first, I imagined the GitHub Actions workflow itself as the safety gate.

However, this experiment showed that the gate does not decide what is dangerous.

The test code defines the inspection items.

If the test checks only:

assert "3 suggestions" in result

then the gate only confirms that the phrase appears somewhere.

It does not know that two related instructions must remain consistent.

In logistics terms, it would be like creating a vehicle inspection gate that checks only:

Are there tires on the truck?

while ignoring:

• Tire pressure
• Damage
• Loose wheel nuts
• Uneven wear
• Whether all tires are the correct type

The existence of an inspection process does not guarantee the quality of the inspection.

The checklist itself matters.

---

Why Passing Tests Can Create False Confidence

Before deliberately breaking the prompt, I saw:

3 passed

and felt reassured.

However, the tests were only as strong as their assertions.

A passing test can create false confidence when:

• The assertion is too broad
• Only the happy path is tested
• Important context is ignored
• Multiple conditions are combined into one weak check
• The test confirms output type but not output meaning
• The test suite does not cover production behavior

This does not mean that tests are useless.

It means that tests are executable assumptions.

If the assumption is vague, the protection is vague.

---

More Robust Test Design

For this function, stronger tests could include several independent checks.

from prompts import build_sales_prompt


def test_build_sales_prompt_contains_input_data():
    sample_data = "Monday: 50 melon buns"

    result = build_sales_prompt(sample_data)

    assert sample_data in result


def test_build_sales_prompt_requests_three_suggestions():
    result = build_sales_prompt("Test data")

    assert "provide three specific suggestions" in result.lower()


def test_build_sales_prompt_requires_three_bullet_points():
    result = build_sales_prompt("Test data")

    assert "provide three bullet-point suggestions" in result.lower()


def test_build_sales_prompt_returns_string():
    result = build_sales_prompt("Test data")

    assert isinstance(result, str)


def test_build_sales_prompt_is_not_empty():
    result = build_sales_prompt("Test data")

    assert result.strip()

Separating the assertions makes the purpose of each test clearer.

When one fails, the test name also explains which contract was broken.

---

Production Deployment and the Inspection Gate

There is another important point.

Running GitHub Actions on every push does not automatically guarantee that Render waits for the tests before deploying.

The complete safe flow should ideally be:

Push or pull request
        ↓
Run automated tests
        ↓
Tests pass
        ↓
Merge or deploy

If the deployment platform automatically deploys every push independently of the GitHub Actions result, the application may still be deployed even when the test workflow fails.

That means a real deployment gate may require additional configuration, such as:

• Deploying only from a protected branch
• Requiring status checks before merging
• Using pull requests instead of pushing directly to main
• Triggering deployment only after the test job succeeds
• Configuring branch protection rules

Creating the tests is one layer.

Connecting the test result to deployment permission is another layer.

---

A Warning I Also Noticed

During the workflow run, I also saw a warning related to the Node.js runtime used by a GitHub Action.

It did not affect the pass or fail result of my tests.

However, I recorded it as a future maintenance item.

CI configurations also age.

Even if the application code does not change, workflow actions and runtime environments may eventually require updates.

That is another reason to review automation logs instead of looking only at the final green check mark.

---

Summary

Here is what I learned from this work.

1. Writing a Test Is Not Enough

You cannot know what a test truly protects until you deliberately break the relevant behavior.

2. Weak String Checks Can Miss Contradictions

When the same value appears in multiple contexts, checking only whether the value exists somewhere is not enough.

The surrounding meaning must also be tested.

3. Local and CI Environments Are Different

A package installed only on a developer's machine does not exist in a clean CI environment.

Dependencies must be declared explicitly.

4. CI Failures Are Part of the Learning Process

An error such as:

exit code 127

provides information.

The failure itself becomes a troubleshooting clue.

5. A Green Check Mark Is Not Absolute Proof

It only proves that the configured jobs and assertions passed.

It does not prove that every important behavior has been tested.

6. Tests and Deployment Gates Are Separate Concerns

Running tests on push does not automatically mean a deployment platform will refuse failing code.

The workflow must be connected to branch and deployment rules.

---

Final Thoughts

This was the second project in a row where the result differed from what I expected.

In the previous experiment, I introduced a Docker multi-stage build and reduced the image by only 3 MB.

In this experiment, I introduced tests and discovered that the first version of those tests was weaker than I thought.

That is exactly why I believe these records are worth writing.

Real development rarely follows the perfect story:

Learn a technique
        ↓
Implement it once
        ↓
Everything works

The actual process is more like:

Learn a technique
        ↓
Implement it
        ↓
Misunderstand part of it
        ↓
Observe an unexpected result
        ↓
Investigate
        ↓
Improve the design

The unexpected parts often contain the most useful lessons.

---

This article is based on actual development logs.

sales_data_app Maintenance Series

• https://qiita.com/tosane932/items/ac18b633c8c87b9807bb
• https://qiita.com/tosane932/items/f0aeed574d39c5fa5093
• https://qiita.com/tosane932/items/e19ed4a2ffe27f53faf0
• https://qiita.com/tosane932/items/c1609f17cddf842f1e7c
• https://qiita.com/tosane932/items/b9b6576c1fda3d3a76d2

Pre-Production Inspection Trilogy

• https://qiita.com/tosane932/items/7a15e942745545a37b6a
• https://qiita.com/tosane932/items/3db0e915439048624a40
• https://qiita.com/tosane932/items/b9f32a1b03b99405ad0a

Post-Deployment Maintenance Series

• https://qiita.com/tosane932/items/cb1156c04c0402b7f7ce
• https://qiita.com/tosane932/items/4825452f4bb73fd90ba8

🍞 sales_data_app on GitHub