DEV Community

Cover image for I got a full pytest report in CI without touching my project
Kevi
Kevi

Posted on

I got a full pytest report in CI without touching my project

#programming #cicd #python #pytest

The usual way this goes: you want something in CI, so you set it up locally first. Install the plugin, get it working, tweak the config, make sure it doesn't break anyone else's environment, commit it, and then it shows up in CI as a side effect of all that.

I didn't want to do any of that. I wanted an HTML report in GitHub Actions and I wanted my pyproject.toml, my conftest.py, and everyone's local setup to stay exactly as they were.

I wasn't sure that was possible until I tried it.

What I tried first

The first thing I reached for was pytest-html. Added it to dev dependencies, got the report locally, committed the config. It showed up in CI. But the basic report is pretty bare — if you want screenshots attached to failures you're writing conftest hooks yourself. If you want logs captured alongside test output, more conftest. Running tests in parallel with pytest-xdist? You need pytest-html plus a separate plugin to make the report work correctly across workers. It stopped being a reporting solution and started being a small project of its own inside my project.

Then I looked at allure-pytest. The reports are genuinely beautiful but getting there required installing the Allure commandline tool which is a Java dependency, then allure-pytest in the project, then committing instrumentation. I didn't want Java anywhere near my Python CI pipeline.

I looked at dorny/test-reporter and mikepenz/action-junit-report. These are closer in spirit — they're actions that consume test output rather than project dependencies. But they both expect JUnit XML as input, which means something still has to generate it, which means something still has to be in the project.

A dependency here, a conftest hook there, a Java runtime somewhere else. And none of them gave me the full picture in one place — I was still missing screenshots, or a step summary, or I had to wire up artifact upload myself. It kept being more than I wanted it to be.

What I started with

Standard Poetry project, tests in tests/, CI on GitHub Actions. The workflow was just:

- uses: actions/checkout@v4
- uses: actions/setup-python@v5
  with:
      python-version: "3.12"
- run: poetry install
- run: poetry run pytest tests/
Enter fullscreen mode Exit fullscreen mode

Green or red dot in the UI. If something failed you dug through raw log output. Fine most of the time, annoying when sharing results with someone not living in the terminal.

What I changed

I replaced the bare pytest line with pytest-html-plus-action. The action installs pytest-html-plus inside the runner at runtime — the project itself never sees it.

- uses: reporterplus/pytest-html-plus-action@v3
  with:
    use_poetry: "true"
    test_path: tests/
    git_branch: ${{ github.ref_name }}
    git_commit: ${{ github.sha }}
Enter fullscreen mode Exit fullscreen mode

That was the whole change. pyproject.toml untouched. Local dev environment unaffected. The HTML report shows up as a downloadable artifact on the workflow run.

And the report itself had everything — failure details, captured logs, screenshots on failure, test durations, git branch and commit in the header. No conftest hooks, no extra plugins. I didn't add anything to the project and I didn't have to remember to remove anything either. Everything I had been trying to stitch together from three different tools was just there.

The step summary was the part I didn't expect

After the run, the GitHub step summary tab shows this automatically:

## pytest-html-plus summary
- total: 142
- passed: 139
- failed: 3
- skipped: 0
- duration: 47.3s

### Failed cases
- tests/api/test_auth.py::test_token_expiry: AssertionError: expected 401, got 200
- tests/api/test_users.py::test_create_duplicate: KeyError: 'id'
- tests/integration/test_db.py::test_connection_timeout: TimeoutError
Enter fullscreen mode Exit fullscreen mode

No downloading anything. Just open the workflow run and it's there. For a quick "what broke" check this ended up being more useful than the HTML report most of the time.

My project needed more arguments

We had flaky tests so I needed reruns, we track coverage, and I wanted to exclude a browser test directory that needs a separate environment. So the step grew a bit:

- uses: reporterplus/pytest-html-plus-action@v3
  with:
    use_poetry: "true"
    test_path: tests/
    pytest_args: >-
      --cov=mypackage
      --cov-fail-under=80
      --reruns 1
      --ignore=tests/browser
    git_branch: ${{ github.ref_name }}
    git_commit: ${{ github.sha }}
    post_pr_comment: "true"
    github_token: ${{ secrets.GITHUB_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

The reruns worked exactly as I hoped — the report reflects the final state after retries, not intermediate failures. A test that failed once and passed on the second attempt shows as passed. That was the behavior I wanted.

The PR comment was a nice addition. Every push to a PR posts the summary as a comment, so reviewers see the test state without opening the workflow. We push fairly often so there are a few comment threads on busy PRs — each push adds a new one rather than updating the previous — but that turned out to be useful for seeing how results changed across pushes rather than annoying.

The git branch and commit in the report header also earned their keep. Someone downloaded an artifact two weeks after the fact and wanted to know what commit it was from. It was right there.

That was it

The workflow file is the only thing that changed. The project doesn't know any of this is happening. If I remove the action tomorrow, everything goes back to exactly what it was.

That was the thing I was trying to do.

pytest-html-plus is on PyPI. The action is on the GitHub Marketplace. Docs at pytest-html-plus.readthedocs.io.

Top comments (0)