Olivier Buitelaar

Posted on Mar 19

Stop Scrolling Through CI Logs: Aggregate All Your Test Results Into One PR Comment

#testing #githubactions #devops #productivity

The Problem Every Team Has

Your CI pipeline runs tests across 3 frameworks. Jest for frontend. pytest for backend. JUnit for integration tests. Each one dumps output in a different format, in different jobs, buried in different log files.

When something fails, you're clicking through job logs, expanding collapsed groups, Control+F-ing for "FAILED", and piecing together what actually broke.

Every. Single. Time.

One Comment. Everything You Need.

I just shipped test-results-reporter — a GitHub Action that collects test results from JUnit XML, Jest JSON, and pytest JSON, then posts a single clean summary as a PR comment.

What You Get

📊 Test Results Summary

✅ Passed: 847  ❌ Failed: 3  ⏭️ Skipped: 12  🔄 Flaky: 1

❌ Failed Tests:
  • auth.test.ts > login > should reject expired tokens
  • test_payments.py > test_refund_partial_amount
  • UserServiceTest.java > testDeleteCascade

🔄 Flaky Tests (passed on retry):
  • integration/api.test.ts > should handle concurrent requests

No more log diving.

Setup: 4 Lines of YAML

- name: Report Test Results
  uses: ollieb89/test-results-reporter@v1
  if: always()
  with:
    paths: |
      **/junit.xml
      **/jest-results.json
      **/pytest-results.json

Drop that after your test steps. Done.

Flaky Test Detection

The thing that makes this different from just parsing JUnit: flaky test tracking.

Tests that fail then pass on retry get flagged separately. You see them, but they don't block your PR. Over time, you build a picture of which tests are unreliable — so you can actually fix them instead of just re-running CI and hoping.

How It Works

Scans your workspace for test result files matching your glob patterns
Parses JUnit XML, Jest JSON, and pytest JSON into a unified format
Detects flaky tests by comparing results across retries
Posts (or updates) a PR comment with the full summary
Optionally fails the workflow if thresholds are exceeded

Everything runs in your own runner. No external APIs. No data leaving your CI.

Pair It With workflow-guardian

I also built workflow-guardian — it lints your GitHub Actions workflow files for security issues, deprecated commands, and config bugs.

Together, they cover two big blind spots:

workflow-guardian catches CI configuration problems
test-results-reporter catches CI output problems

Both free. Both open source.

Links

If you're drowning in CI noise, give these a try. And if you hit edge cases or want features, open an issue — I ship fast.

DEV Community