Stop Digging Through CI Logs: One Test Report for Python, Java, JS, .NET, Go and Robot Framework

Pankaj Kumar Nayak — Wed, 07 Oct 2026 14:08:59 +0000

It is 9:40 in the morning. The nightly build is red. The CI log is 14,000 lines long, and somewhere in it are the four failures you care about. Before you can fix anything, you have to answer the same questions every team asks after every red build:

What broke since yesterday? Not "what is failing", but what is newly failing.
Is it one problem or ten? Forty tests failing with the same ConnectionError is one broken Redis, not forty bugs.
Is this test flaky, or did we really break it?
Can the person who isn't a Python developer read the result? The QA lead, the release manager, the Java team whose service you call.

Most test runners answer none of these out of the box. They print pass/fail to a terminal and leave the rest to you.

I built pyrept to answer all four, for every test framework a real company runs, in one consistent report. This article is a practical, engineering-focused tour: what it produces, how it works inside, and copy-paste recipes for Python, Java, JavaScript/TypeScript, .NET, Go, Robot Framework, Ruby, Rust and shell tests, plus GitHub Actions, GitLab, Jenkins and Azure DevOps.

Try it before reading further: here is the live demo report and the demo history page. Open them on your phone, they work there too.

Installation :

pip install pyrept
pytest --pyrept

Open report.html. That's it for pytest. For everything else:

You have	You run
pytest / pytest-bdd / Playwright / Selenium (Python)	`pytest --pyrept`
unittest (standard library only)	`python -m pyrept.unittest_runner discover -s tests`
nose2	`nose2 --html-report`
behave (Cucumber BDD)	`behave -f pyrept.behave_formatter:PyreptFormatter -o /dev/null`
Java, JS/TS, .NET, Go, Ruby, Rust, PHP, Robot Framework, TAP, Allure, Cucumber	`pyrept convert --from <format> <files>`
GitHub Actions	`uses: pankajnayak1994/pyrept@v1`

Every path produces the same four outputs:

report.html: one self-contained page. No CDN, no external fonts, no server. Open it offline, attach it to a CI run, email it.
report.json: the same data, machine-readable, for dashboards, bots and the next run's comparison.
junit.xml (optional): so GitLab, Jenkins, Azure DevOps and friends show results in their native test tabs.
A Markdown summary (optional): for GitHub job summaries, pull request comments and chat.

The only runtime dependency is jinja2. Python 3.8 to 3.14 are tested in CI, plus the 3.15 pre-releases.

Part 1: The technical spec

Before the recipes, it helps to know the model pyrept builds, because every feature falls out of it.

1.1 The pipeline

  pytest plugin ───────┐
  unittest runner ─────┤
  nose2 plugin ────────┤                         ┌──> report.html  (self-contained)
  behave formatter ────┼──> ReportCollector ─────┼──> report.json  (machine-readable)
  Python API ──────────┤      │                  ├──> junit.xml    (CI test tabs)
                       │      ├─ baseline diff   ├──> summary.md   (job summary / PR)
  pyrept convert ──────┘      ├─ failure groups  └──> Slack / Teams / webhook
   (JUnit, TRX, NUnit,        └─ quality gates
    xUnit, Robot, TAP,
    Allure, Cucumber,
    Playwright, ...)

Every integration, native or imported, feeds one ReportCollector. Everything downstream (HTML, JSON, JUnit, Markdown, comparison, gates, notifications) is written once and behaves identically no matter where the tests came from. That's the whole design: normalise early, render once.

1.2 Four outcomes, mapped the same way everywhere

Every test ends as exactly one of passed, failed, error or skipped. The mapping is deliberate and consistent across all frameworks and importers:

Situation	Reported as
Assertion failure	`failed`
Unexpected exception, setup/teardown failure, collection or import error, undefined BDD step, hook error, timeout, crash	`error`
Skipped test	`skipped`, with the reason
Expected failure (`xfail`, `@expectedFailure`, TAP `# TODO`)	`skipped`, with "Expected failure"
Unexpected success (strict `xfail` that passed, `@expectedFailure` that passed)	`failed`
Failing subtest	its own `failed` / `error` entry
Passed after a rerun	`passed`, with `retries` and `flaky`

Pass rate = passed ÷ (total − skipped).

Engineering note: why skipped tests don't lower the pass rate. If you skip 200 tests because a sandbox is down, your code didn't get 200 tests worse. Counting skips as failures trains teams to delete skip markers instead of fixing environments. Skips are shown prominently, they just don't pollute the quality signal.

Engineering note: why failed and error are different. A failed test means your code produced the wrong answer. An error usually means the test couldn't even run: a fixture blew up, an import failed, a dependency was down. When 30 errors appear at once, you look at infrastructure first. When 3 failures appear, you look at the diff. Keeping them separate saves real triage time.

1.3 The JSON contract

report.json is the stable interface between pyrept and everything else (dashboards, bots, the next run). Abbreviated:

{
  "test_report_title": "Checkout suite",
  "test_summary": {"total": 51, "passed": 46, "failed": 2, "error": 2, "skipped": 1, "percentage": 92.0},
  "timestamp": "2026/10/07 11:58:46 UTC",
  "duration": 13.29,
  "environment": {"Python": "3.13.2", "Framework": "pytest 9.0 + Playwright", "Build": "#1842"},
  "slowest_tests": [{"name": "tests/test_checkout.py::test_pay_with_card", "duration": 3.42}],
  "failure_groups": [{"message": "redis.exceptions.ConnectionError: Error 111 connecting to cache:6379. Connection refused.",
                      "count": 2, "tests": ["tests/test_api.py::test_rate_limit", "tests/test_api.py::test_session_cache"]}],
  "comparison": {
    "percentage_delta": -4.0,
    "new_failures": ["tests/test_checkout.py::test_pay_with_card"],
    "fixed": [], "still_failing": [], "new_tests": [], "removed_tests": [],
    "slower": [{"name": "...", "before": 1.2, "after": 3.42, "ratio": 2.85}]
  },
  "test_results": [
    {"name": "tests/test_checkout.py::test_pay_with_card", "result": "failed",
     "description": "Pays with a saved Visa card.", "traceback": "AssertionError: ...",
     "metadata": {"duration": 3.42, "location": "tests/test_checkout.py:42", "tags": ["smoke", "payments"]}}
  ]
}

test_results lists errors first, then failures, skips and passes. comparison is null when no baseline was given. Because the format is plain JSON, a pyrept report from one job is also a valid input to another: that's how sharded runs and multi-language monorepos get merged (Part 3).

1.4 Regression diff: comparing with the previous run

Give any integration an earlier report.json as a baseline, and every test gets a change label:

Label	Definition
New failure	Failed/errored now, but passed, was skipped, or didn't exist in the baseline
Fixed	Failed/errored in the baseline, passes now
Still failing	Failed/errored in both
New	Not in the baseline, passing
Removed	In the baseline, not in this run
Slower	At least 1.5× as slow and at least 0.5 s slower

Engineering note: why "slower" needs two thresholds. A ratio alone flags a 2 ms test that became 5 ms (2.5×, irrelevant). An absolute delta alone flags a 60 s integration test that jittered to 60.6 s. Requiring both 1.5× and +0.5 s catches real regressions and ignores noise.

Engineering note: the baseline can be the file you're about to overwrite. pyrept reads the baseline before writing, so --pyrept-json=report.json --pyrept-baseline=report.json always compares with the previous run. A missing or corrupt baseline never fails the run; it logs a warning and writes the report without a comparison. That's exactly what you want on the first CI run of a new branch.

Tests are matched by name, so compare reports that come from the same source.

1.5 Common failure causes

Failures that share an error message are grouped. In the demo, two API tests fail with the same redis.exceptions.ConnectionError; the report shows one group of 2, not two separate mysteries. On a bad day this turns "38 failures" into "1 broken Redis + 2 real bugs", which is a very different stand-up.

1.6 What gets captured per test

Description (docstrings, BDD scenario text), tags (pytest markers, NUnit categories, xUnit traits, Robot tags, Allure labels), duration and file:line.
For failures: captured stdout, stderr and logs, the traceback, and its first line as the message.
For browser tests: a full-page screenshot taken while the browser is still open, the browser console output and page errors (Playwright), and links to Playwright traces and videos.
Retries and flakiness (pytest-rerunfailures, Surefire reruns, Playwright retries, Allure history).
Environment: Python, platform, framework, plus anything you add.

1.7 Security and robustness details

These are the boring things that make a report tool safe to run on untrusted CI output:

Everything is HTML-escaped, including tracebacks that contain <script>-like text, and attachment data from imported files. javascript: links in attachment paths are never rendered.
Images up to 5 MB are embedded as data URIs, so the report stays a single file. Larger files stay linked.
JUnit output strips characters XML forbids (such as ANSI colour codes), so CI parsers never reject it.
The Markdown summary stays far below GitHub's 1 MiB job-summary limit, however big the run. It shows the first 20 failures with trimmed errors.
Webhook URLs are treated as secrets: never logged, and a failed notification never fails your build.
Exit codes are predictable: 0 OK, 1 a gate failed, 2 bad input (missing file, wrong format).

Part 2: Python recipes

2.1 pytest in 30 seconds

Here's a tiny shop test suite I'll use for the rest of the Python section:

# tests/test_cart.py
import pytest

@pytest.mark.smoke
def test_add_item():
    """Adding an item updates the cart count."""
    assert 1 + 1 == 2

def test_apply_coupon():
    """AUTUMN15 takes 15% off."""
    assert 85.0 == 85.0

def test_legacy_tax_rules():
    """Known broken since the tax refactor (JIRA-123)."""
    assert round(0.1 + 0.2, 2) == 0.31

@pytest.mark.skip(reason="payment sandbox down")
def test_refund():
    pass

pytest --pyrept

The plugin is installed with the package but stays off until you ask for a report, so it never changes a normal pytest run. Docstrings become descriptions, markers become tags, skip reasons are kept.

All the options:

pytest --pyrept-html=reports/report.html --pyrept-json=reports/report.json
pytest --pyrept --pyrept-title="Nightly regression"
pytest --pyrept --pyrept-junit=reports/junit.xml        # also JUnit XML
pytest --pyrept --pyrept-baseline=reports/report.json   # compare with the previous run
pytest --pyrept --pyrept-markdown=reports/summary.md    # Markdown summary
pytest --pyrept --pyrept-github-summary                 # GitHub job summary

Or set it once in pytest.ini / pyproject.toml:

[pytest]
pyrept = true
pyrept_html = reports/report.html
pyrept_json = reports/report.json
pyrept_junit = reports/junit.xml
pyrept_baseline = reports/report.json
pyrept_github_summary = true

Engineering note: path resolution. Paths on the command line are relative to where you run pytest. Paths in the ini file are relative to the ini file, so the report always lands in the same place no matter which subdirectory a developer runs pytest from. Small thing, saves a lot of "where did my report go?".

2.2 The regression workflow, for real

I ran the suite above twice. On the second run, a coupon calculation broke and the Redis cache was down. pytest's own output tells you 4 things failed. pyrept adds one line that tells you what matters:

------ pyrept: 3 new failures, 0 fixed, 1 still failing (vs report.json) -------
FAILED tests/test_cart.py::test_apply_coupon - assert 90.0 == 85.0
FAILED tests/test_cart.py::test_legacy_tax_rules - assert 0.3 == 0.31
ERROR tests/test_api.py::test_rate_limit - ConnectionError: Error 111 connect...
ERROR tests/test_api.py::test_session_cache - ConnectionError: Error 111 conn...

test_legacy_tax_rules was already broken yesterday. The other three are new, and two of them share one root cause. That's the triage done before you've opened a single traceback.

2.3 Quality gates for legacy suites: the ratchet

Every mature codebase has a few known-broken tests. Teams usually respond by either ignoring the red build (bad) or deleting the tests (worse). pyrept gives you a third option: fail only on new failures, with a floor.

pytest --pyrept-json=reports/report.json --pyrept-baseline=reports/report.json \
       --pyrept-ignore-known-failures --pyrept-fail-under=90

With the known-broken tax test only:

- pyrept: 1 known failure ignored: all of them were already failing in the baseline -
1 failed, 4 passed, 1 skipped in 0.04s
$ echo $?
0

The moment a previously passing test breaks:

--- pyrept: 3 new failures, so --pyrept-ignore-known-failures does not apply ---
$ echo $?
1

The fail-under floor stops known failures from quietly piling up. Over time you raise it: that's the ratchet.

Engineering note: gates never hide other failures. If pytest-cov's --cov-fail-under (or any other plugin) failed the run, pyrept keeps it failed. Collection errors (a test module that doesn't even import) are never ignored. A gate that can accidentally turn a broken build green is worse than no gate.

2.4 Parallel, retries, subtests, BDD

These just work with the same flags:

pytest-xdist (-n auto): workers send results to the controller, which writes one report. Descriptions, tags and screenshots survive the trip.
pytest-rerunfailures: reruns aren't counted as extra tests; the final attempt is recorded with retries, and marked flaky if it passed after a retry.
Subtests (built into pytest 9+, or pytest-subtests): each failing subtest gets its own entry, like test_x (i=1).
pytest-bdd: each scenario shows the feature, the scenario and every step with its status:

Feature: Checkout
Scenario: Pay with card
  Given a cart with 2 items  [passed]
  When I pay with a declined card  [failed]
  Then I see the confirmation page  [skipped]

2.5 Playwright and Selenium (Python)

UI test failures without a screenshot are guesswork. When a test using a pytest-playwright page fixture, or a Selenium driver / selenium / browser fixture, fails, pyrept attaches a full-page screenshot (taken before the browser closes), the browser console output and uncaught page errors, and links to the trace and video:

pip install "pyrept[playwright]" && playwright install chromium
pytest --pyrept --tracing retain-on-failure --video retain-on-failure

Open traces at trace.playwright.dev. Links are relative to the report, so upload reports/ and test-results/ together as one CI artifact.

2.6 Extend it from `conftest.py`

Three hooks let you add environment rows, attachments or anything else without forking anything:

# conftest.py
import os
from pyrept import make_attachment

def pytest_pyrept_environment(config):
    return {"Build": os.environ.get("BUILD_ID", "local"),
            "Commit": os.environ.get("GITHUB_SHA", "")[:7]}

def pytest_pyrept_attachments(item, report):
    # Attach the API server's log to every failing test
    if report.when == "call" and report.failed and os.path.exists("server.log"):
        with open("server.log") as fh:
            return [make_attachment("server.log", "text/plain", text=fh.read()[-20000:])]

def pytest_pyrept_context(config, context):
    context["test_report_title"] += " (nightly)"

pytest_pyrept_attachments runs where the test runs, including xdist workers, so it returns plain data.

2.7 unittest, nose2, behave

unittest, with zero extra dependencies. It accepts all the usual python -m unittest arguments:

python -m pyrept.unittest_runner discover -s tests \
    --pyrept-junit=reports/junit.xml --pyrept-baseline=reports/report.json

Or from code:

import unittest
from pyrept.unittest_runner import PyreptTestRunner

suite = unittest.defaultTestLoader.discover("tests")
result = PyreptTestRunner(html_path="reports/report.html", json_path="reports/report.json",
                          title="Unit tests", verbosity=2).run(suite)
raise SystemExit(not result.wasSuccessful())

nose2 (pip install "pyrept[nose2]"), in nose2.cfg:

[unittest]
plugins = pyrept.html_report

[html-report]
always-on = True
html-report-path = test_results/report.html
json-report-path = test_results/report.json

behave (Cucumber BDD for Python):

pip install "pyrept[behave]"
behave -f pyrept.behave_formatter:PyreptFormatter -o /dev/null -f pretty \
       -D pyrept_html=reports/report.html -D pyrept_json=reports/report.json

Gotcha worth knowing: behave pairs each -o with the -f before it. Put -f pyrept... -o /dev/null first; if you write -f pretty -f pyrept -o /dev/null, your console output disappears into /dev/null. (Windows: -o NUL.)

Part 3: Every other stack

Here's the part I'm most proud of. Your QA team writes Robot Framework, the backend is Java, the frontend is TypeScript, the payment service is .NET, and the CLI is Go. They all produce some report file. pyrept convert turns each of them into the same HTML/JSON/JUnit/Markdown output, and the project you convert from doesn't need to be a Python project. You only need Python on the CI runner, which every hosted runner already has.

3.1 The format matrix

Stack	Produce a report with	Convert with
Java: Maven Surefire / Failsafe	`mvn test`	`pyrept convert --from junit target/surefire-reports/*.xml`
Java / Kotlin: Gradle	`./gradlew test`	`pyrept convert --from junit build/test-results/test/*.xml`
JavaScript / TypeScript: Jest	`jest --ci --reporters=default --reporters=jest-junit`	`pyrept convert --from junit junit.xml`
Cypress	`cypress run --reporter junit --reporter-options "mochaFile=results/junit-[hash].xml"`	`pyrept convert --from junit results/*.xml`
Playwright Test (JS/TS)	`npx playwright test --reporter=json > results.json`	`pyrept convert --from playwright results.json`
Node.js test runner	`node --test --test-reporter=tap > results.tap`	`pyrept convert --from tap results.tap`
.NET (MSTest, NUnit, xUnit via VSTest)	`dotnet test --logger trx --results-directory TestResults`	`pyrept convert --from trx TestResults/*.trx`
NUnit 3 / 2 native XML	NUnit console	`pyrept convert --from nunit TestResult.xml`
xUnit.net v2 XML	`-xml results.xml`	`pyrept convert --from xunit results.xml`
Go	`gotestsum --junitfile go-junit.xml`	`pyrept convert --from junit go-junit.xml`
Rust	`cargo nextest run --profile ci` (with JUnit enabled in `.config/nextest.toml`)	`pyrept convert --from junit target/nextest/ci/junit.xml`
PHP: PHPUnit	`phpunit --log-junit junit.xml`	`pyrept convert --from junit junit.xml`
Ruby: RSpec	`rspec --format RspecJunitFormatter --out rspec.xml`	`pyrept convert --from junit rspec.xml`
Robot Framework 3.x to 7.x	`robot --outputdir results tests/`	`pyrept convert --from robot results/output.xml`
Shell: bats, Perl `prove`, `pg_prove`	`bats --tap tests/ > results.tap`	`pyrept convert --from tap results.tap`
Cucumber (JVM, JS, Ruby, godog)	the `json` formatter (e.g. `--plugin json:target/cucumber.json`)	`pyrept convert --from cucumber target/cucumber.json`
Allure (any language)	an `allure-results` directory	`pyrept convert --from allure allure-results`
pytest-json-report	`pytest --json-report`	`pyrept convert --from pytest-json .report.json`

Each converter does more than read pass/fail:

JUnit XML: properties become tags, Surefire flakyFailure/rerunFailure become retries, [[ATTACHMENT|...]] screenshots are embedded, system-out/err are attached.
TRX: data-driven rows become separate tests, result files are linked or embedded.
Robot Framework: long names like Suite.Sub suite.Test, tags, documentation, durations for both the Robot 7 elapsed attribute and older start/end timestamps.
TAP 12 to 14: plans, SKIP / TODO directives, YAML diagnostics, and Bail out!.
Allure: retries folded into one test with history, labels as tags, attachments kept.

Mapping is consistent across all of them: crashes, timeouts and broken tests become error; ignored, inconclusive and TODO tests become skipped.

3.2 One report for a polyglot release

Here's a real run I did while writing this article: a Java Surefire file, a Jest JUnit file, a .NET TRX file, a Robot Framework output.xml and a Node TAP file, merged into one release report.

pyrept convert --from junit  target/surefire-reports/*.xml --json parts/java.json   --html parts/java.html
pyrept convert --from junit  web/junit.xml                 --json parts/web.json    --html parts/web.html
pyrept convert --from trx    TestResults/*.trx             --json parts/dotnet.json --html parts/dotnet.html
pyrept convert --from robot  results/output.xml            --json parts/robot.json  --html parts/robot.html
pyrept convert --from tap    cli/results.tap               --json parts/cli.json    --html parts/cli.html

# Merge every stack into one report (pyrept JSON is itself an input format)
pyrept convert --from pyrept parts/*.json --title "Release 2026.10 - all stacks" \
               --html all/report.html --json all/report.json --junit all/junit.xml

pyrept: 23 tests (10 passed, 6 failed, 2 errors, 5 skipped)
pyrept HTML report: all/report.html
pyrept JSON report: all/report.json
pyrept JUnit report: all/junit.xml

Your release manager gets one link. Each test keeps its framework metadata (junit, trx, robot, tap), so you can still tell where it came from.

The same trick merges sharded CI jobs: run 8 shards in parallel, have each upload its junit.xml or report.json, then merge in a final job:

pyrept convert --from junit shard-*/junit.xml --html reports/report.html --json reports/report.json

3.3 Gates and baselines work for every stack

Everything from Part 2 applies to converted reports:

# Robot Framework suite: fail the job only on tests that passed last time,
# and never let the pass rate fall below 95%
pyrept convert --from robot output.xml \
    --baseline reports/report.json --json reports/report.json --junit reports/junit.xml \
    --fail-on-failure --ignore-known-failures --fail-under 95

So a Java team gets "3 new failures, 1 fixed" from their Surefire XML exactly like a Python team does from pytest.

Part 4: CI recipes

4.1 GitHub Actions: job summary and a sticky PR comment

The pyrept GitHub Action puts the result on the run's summary page and keeps one pull request comment up to date (it edits its own comment instead of spamming a new one per push).

name: tests
on: [push, pull_request]

permissions:
  contents: read
  pull-requests: write          # for the PR comment

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.13'

      # Keep the previous report so every run is compared with the one before it
      - uses: actions/cache/restore@v4
        with:
          path: reports/report.json
          key: pyrept-${{ github.ref_name }}-${{ github.run_id }}
          restore-keys: pyrept-${{ github.ref_name }}-

      - run: pip install -r requirements.txt pyrept
      - run: >
          pytest --pyrept-html=reports/report.html --pyrept-json=reports/report.json
          --pyrept-baseline=reports/report.json
        continue-on-error: true       # let the action decide whether the job fails

      - uses: pankajnayak1994/pyrept@v1
        with:
          report: reports/report.json
          fail-on-failure: true
          ignore-known-failures: true
          fail-under: 90

      - uses: actions/cache/save@v4
        if: always()
        with:
          path: reports/report.json
          key: pyrept-${{ github.ref_name }}-${{ github.run_id }}

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-report
          path: reports/

The PR comment looks like this (it's plain GitHub Markdown, trimmed here):

### ❌ Test Report: 2 failed, 2 errors

| Total | Passed | Failed | Errors | Skipped | Pass rate | Duration |
|---:|---:|---:|---:|---:|---:|---:|
| 6 | 1 | 2 | 2 | 1 | 20.0% | 0.02s |

**Compared with previous run**: pass rate 80.0% then, -60.00 pts now

3 new failures · 0 fixed · 1 still failing

#### Common failure causes
| Tests | Error |
|---:|---|
| 2 | `ConnectionError: Error 111 connecting to cache:6379. Connection refused.` |

#### Failures
▸ tests/test_api.py::test_rate_limit (error, new failure)
▸ tests/test_cart.py::test_apply_coupon (failed, new failure)
...

Reviewers see "3 new failures, one shared root cause" right in the PR, without opening the Actions tab.

Non-Python repos use convert-from, which runs the conversion inside the action:

      - run: mvn -B test
        continue-on-error: true
      - uses: pankajnayak1994/pyrept@v1
        with:
          convert-from: junit
          convert-inputs: target/surefire-reports/*.xml
          report: reports/report.json
          title: Payments service
          fail-on-failure: true

The action also exposes outputs (total, passed, failed, errors, skipped, pass-rate, new-failures) for later steps, for example to post a badge or block a deploy.

Engineering note: forks. Pull requests from forks get a read-only token, so they can't comment. The action treats the comment as best-effort: it never fails your job because of a permission it was never going to have. The job summary still works.

4.2 GitLab CI

GitLab reads JUnit natively and shows it in the merge request widget; the HTML goes along as an artifact:

test:
  image: python:3.13
  script:
    - pip install -r requirements.txt pyrept
    - pytest --pyrept-html=reports/report.html --pyrept-json=reports/report.json --pyrept-junit=reports/junit.xml
  artifacts:
    when: always
    paths: [reports/]
    reports:
      junit: reports/junit.xml

4.3 Jenkins

stage('Test') {
  steps {
    sh 'pytest --pyrept-html=reports/report.html --pyrept-json=reports/report.json --pyrept-junit=reports/junit.xml'
  }
  post {
    always {
      junit 'reports/junit.xml'
      archiveArtifacts artifacts: 'reports/**', allowEmptyArchive: true
    }
  }
}

4.4 Azure DevOps

- script: |
    pip install pyrept
    dotnet test --logger trx --results-directory TestResults
    pyrept convert --from trx TestResults/*.trx --html reports/report.html --json reports/report.json --junit reports/junit.xml
  displayName: Test and report

- task: PublishTestResults@2
  condition: always()
  inputs:
    testResultsFormat: JUnit
    testResultsFiles: reports/junit.xml

- publish: reports
  artifact: test-report
  condition: always()

4.5 Deep links

The HTML report keeps filters in the URL, so you can link straight to what matters from a chat message or a ticket:

report.html#problems: failures and errors only
report.html#new-failures: only what broke since the baseline
#failed, #error, #skipped, #passed

Part 5: History and flaky tests

A single report tells you about one run. pyrept history reads many JSON reports and tells you about your test suite:

pyrept history history/ --html reports/history.html --json reports/history.json --limit 30

pyrept: 30 runs, 412 tests, 3 flaky, 4 failing now

You get pass-rate trends, outcomes and duration per run, flaky tests, failure streaks ("failing for 5 runs, since …"), the most-failing tests and the slowest tests, each with a run-by-run outcome strip.

Engineering note: a precise definition of "flaky". The naive definition calls a test flaky if it ever failed and ever passed. That's too eager: a test that broke on Tuesday and was fixed on Wednesday is a regression, not flakiness. pyrept counts a test as flaky when it fails in two or more separate stretches with passing runs in between, or when it passed only after a retry. The list stays short and actionable instead of shaming every test that was ever fixed.

To build history in GitHub Actions, keep one report per run in a cached directory:

- uses: actions/cache/restore@v4
  with:
    path: history
    key: pyrept-history-${{ github.run_id }}
    restore-keys: pyrept-history-
- run: |
    pytest --pyrept-json=reports/report.json || true
    mkdir -p history && cp reports/report.json "history/run-${{ github.run_number }}.json"
    ls -t history/*.json | tail -n +31 | xargs -r rm   # keep the newest 30
    pyrept history history --html reports/history.html
- uses: actions/cache/save@v4
  if: always()
  with:
    path: history
    key: pyrept-history-${{ github.run_id }}

Runs are ordered by their timestamp (not file name), and files that aren't pyrept reports are skipped with a message, so a stray file in the cache never breaks the job.

Part 6: Notifications

Set environment variables in CI. Nothing else changes:

Variable	Meaning
`PYREPT_SLACK_WEBHOOK_URL`	Slack incoming webhook
`PYREPT_TEAMS_WEBHOOK_URL`	Microsoft Teams workflow webhook (adaptive card)
`PYREPT_WEBHOOK_URL`	Any endpoint; receives the summary as JSON (counts, comparison, failing tests)
`PYREPT_NOTIFY_ON`	`always` (default), `failure` or `new-failures`
`PYREPT_REPORT_URL`	Link to the published HTML report

❌ Nightly: 2 failed, 1 error of 412 tests (pass rate 99.27%, -0.49 pts) · 2 new failures, 0 fixed · Open the report

PYREPT_NOTIFY_ON=new-failures is the one I recommend for team channels: silence while known issues are being worked on, a ping the moment something new breaks. That's how you keep a channel from becoming noise everyone mutes.

Part 7: The Python API, for anything that isn't a test framework

Not every check is a unit test. Load tests, contract checks, data-quality validations, smoke scripts, migration verifications: if it has a name and a pass/fail, it fits. Here's a performance check reported with the same HTML, JSON and JUnit output as everything else:

from pyrept import ReportCollector, make_attachment

report = ReportCollector(title="Nightly smoke (k6 + custom checks)",
                         environment={"Build": "1842", "Region": "eu-west-1"})

report.add("GET /health", "passed", metadata={"duration": 0.12, "tags": ["smoke"]})
report.add("POST /checkout p95 < 800ms", "failed",
           traceback="p95 was 1240ms (threshold 800ms)",
           metadata={"duration": 60.0, "tags": ["performance"]},
           attachments=[make_attachment("k6 summary", "text/plain",
                                        text="http_req_duration p(95)=1.24s")])
report.add("GET /search", "skipped", traceback="Skipped: search cluster in maintenance")

context = report.write("reports/report.html", "reports/report.json", junit_path="reports/junit.xml")
print(context["test_summary"])
# {'total': 3, 'passed': 1, 'failed': 1, 'error': 0, 'skipped': 1, 'percentage': 50.0}

write() also takes baseline_path, markdown_path, github_summary and report_url, so custom checks get regression diffs and PR summaries for free.

Data engineers: the same pattern works for dbt tests, Great Expectations results or SQL assertions. Platform engineers: wrap your Terraform policy checks or Kubernetes smoke probes. If you can loop over results, you can produce a report your whole org already knows how to read.

Part 8: The HTML report, up close

A few details that come from using reports during real incidents:

Verdict first. A big PASSED / FAILED header with the pass-rate ring and an outcome bar. You know the answer before you scroll.
Problems first. Failures and errors are listed before passes; the comparison panel and failure groups sit above the test list.
Test map. One square per test, coloured by outcome. Click any square to jump to it. On a 2,000-test suite, clusters of red show you which module is on fire.
Search across names, descriptions and errors, filters with counts, sort by status, name or duration, and per-test duration bars.
Keyboard navigation: j / k to move between tests.
Copyable tracebacks and zoomable screenshots.
Dark mode that follows your OS, a print-friendly layout, and it works on a phone.
Branding: PYREPT_LOGO=branding/logo.svg and PYREPT_ACCENT_COLOR="#0f766e" make it look like your company's report. PYREPT_LOGO=none removes the logo.

And again: it's one file. No server, no JavaScript CDN, nothing to host. It opens from an email attachment, from a CI artifact zip, or from an air-gapped network.

How does it compare?

Honest positioning, because good tools exist already:

pytest-html is great if you only use pytest and want a simple results table. pyrept adds the cross-framework model, baseline diffs, failure grouping, history/flakiness, gates, Markdown/PR output and importers for other languages.
Allure is powerful and beautiful, but it needs its own report generator step and a results directory per framework adapter. pyrept produces the finished single-file HTML directly from the test run, and it can import Allure results if you already have them.
Your CI server's test tab (GitLab, Jenkins, Azure) is useful, and pyrept feeds it with JUnit. But those tabs are tied to one CI system and can't be emailed, archived or opened offline.

pyrept isn't trying to replace your test framework or your CI. It's the layer between them that turns raw results into a decision.

Engineering checklist: rolling it out in a team

What I'd do on a real team, in order:

Week 1: add --pyrept (or pyrept convert) and upload reports/ as a CI artifact. Zero risk, instant value.
Week 1: add the GitHub Action (or JUnit for GitLab/Jenkins/Azure) so results show up where reviewers already look.
Week 2: cache report.json and pass it as --pyrept-baseline. Now every run says what's new.
Week 2: turn on fail-on-failure + ignore-known-failures with a fail-under floor at today's pass rate. Raise the floor every sprint.
Week 3: start a history cache and review the flaky list in retro. Fix or quarantine the top 3.
Week 3: set PYREPT_NOTIFY_ON=new-failures for the team channel.
Later: merge the other teams' Java/JS/.NET/Robot results into one release report.

Try it

pip install pyrept
pytest --pyrept

⭐ GitHub: github.com/pankajnayak1994/pyrept
📦 PyPI: pypi.org/project/pyrept
🚀 GitHub Action: marketplace/actions/pyrept
📖 Docs: pankajnayak1994.github.io/pyrept
🖥️ Live demo report: demo/report.html

It's MIT-licensed and tested on Python 3.8 through 3.14. If it saves you one morning of log-scrolling, a star on GitHub helps other engineers find it. Issues, ideas and pull requests are very welcome, especially new importers: if your stack writes a report format pyrept can't read yet, open an issue with a sample file.

I'd love to hear in the comments: what's the first question you ask after a red build?

pankajnayak1994 / pyrept

This Library Generate a HTML and JSON report for your Nose Unit Tests

pyrept: one HTML, JSON and JUnit test report for every test framework

pyrept turns a test run into:

report.html: one self-contained page you can open offline, attach to a CI run or email. It has no CDN or external assets.
report.json: the same data in machine-readable form, for dashboards, bots and pipelines.
junit.xml (optional): JUnit XML, so GitLab, Jenkins, Azure DevOps and other CI servers can show results in their test tabs.
A Markdown summary (optional): for GitHub job summaries, pull request comments and chat.

It works the same way with pytest, unittest, nose2, behave, pytest-bdd and Playwright or Selenium browser tests. It also converts JUnit XML, .NET TRX, NUnit, xUnit.net, Robot Framework, TAP, Allure, Cucumber JSON, Playwright Test JSON and pytest-json-report files, so teams using Java, JavaScript, Go, .NET or Ruby get the same report.

Every report can compare itself with the previous run and highlight new…

View on GitHub

Developed and maintained by Pankaj Kumar Nayak (@pankajnayak1994).

DEV Community: Pankaj Kumar Nayak