DEV Community: Kevi

Stop Guessing What Caused Your Flaky Tests Fail or Pass

Kevi — Sun, 05 Apr 2026 17:51:19 +0000

Flaky tests don’t fail when you expect them to.
They fail when you least have time.

One moment everything is green, the next your CI pipeline is red — and then, magically, it passes on rerun.

❌ ❌ ✅ → Passed

So… what just happened?

Was it a network issue?
Timing? State leakage?
The classic DOM detached?
May be, the fixture didnt return the value?

The Problem: We Only See the Final Outcome

Most test reports show you only the final result, or you install a bunch of plugins that would scrap all the xmls for you to show you multiple tests of same title and you click each one of them to see which might have ran first?

If a test fails twice and passes on the third attempt, all you see is:

TestCheckoutFlow → Rerun
TestCheckoutFlow → Rerun
TestCheckoutFlow → Rerun
TestCheckoutFlow → PASSED

That “pass” hides everything that matters:

Why did it fail initially?
What changed between attempts?
Is this a real bug or just instability?

👉 You’re left guessing or too much time scrolling and finding what would have ran first.

And guessing doesn’t scale — especially in CI and neither you wanna write a log scraper on top of it.

What Flaky Tests Actually Do to Your System

Flaky tests aren’t just annoying. They slowly break your engineering system:

You start ignoring failures (“just rerun it”) - coz I dont have time to dig through the logs
Confidence in CI drops
Real bugs(Uncaught ones) get buried under noise
Debugging becomes reactive instead of intentional

Over time, your test suite stops being a safety net and becomes background noise.

The Missing Piece: Attempt-Level Visibility

The root issue is simple:

We don’t see what happened in each retry.

When you use tools like pytest-rerunfailures, retries happen silently. The final result is shown — the journey is lost.

But that journey is where the bug lives.

I had a test that was just consistently failing in re-runs and since I had no control over the data at that point of time, the re-runs simply showed me that there were multiple records created and it failed(But there would not be multiple records in the first one?), so why did it fail in the first place?

A Better Way: Capture Every Attempt

To solve this, we shipped an update to pytest-html-plus that records every attempt of a test, not just the final one.

Instead of:

TestLogin → Passed

You see:

TestLogin  → Passed
 ├── Attempt 1 → Failed (TimeoutError)
 ├── Attempt 2 → Failed (Element not found)
 └── Attempt 3 → Passed

Now you can answer:

Is it a timing issue?
Is the UI not ready?
Is an API inconsistent?

👉 No more guessing. Just evidence.

Real Example: A “Passing” Test That Was Actually Broken

I had a test that looked harmless:

❌ ❌ ❌ → Passed

Normally, I would move on.

But with attempt-level logs, the story changed:

Attempt 1 → API response delayed - but created a partial record
Attempt 2 → UI rendered partially - but did not find that full record

This wasn’t a passing test.

This was a race condition waiting to hit production.

But Won’t This Make Reports Noisy?

Yes — if done poorly.

Logging every attempt can easily clutter reports, especially in large test suites.

So we designed it to stay clean by default:

Attempts are collapsible
Logs are grouped per attempt
You expand only what you need

👉 You get detail without sacrificing readability.

Plug & Play with Pytest

Getting started is simple:

pip install pytest-html-plus
pytest --html=report.html

You get:

HTML reports with attempt breakdown
JSON output for automation
Flaky test detection
Clear visibility into retries

No major setup. No workflow changes. No third party plugins needed

When This Helps the Most

This is especially useful if:

You use retries in CI (pytest-rerunfailures)
Tests pass locally but fail in CI
You suspect timing, async, or state issues
You’ve ever said “it passed on rerun, so it’s fine”

The Mindset Shift

Stop asking:

“Did the test pass?”

Start asking:

“How did the test pass?”

Because that’s where the real bugs hide.

Final Thought

Flaky tests don’t just waste time —
they slowly erode trust in your system.

And once trust is gone,
your tests stop protecting you.

If you can see every attempt,
you can fix the root cause — not just silence the symptom.

If you want to try it out:

👉 [https://github.com/reporterplus/pytest-html-plus / https://pypi.org/project/pytest-html-plus/]

Would love to hear how you’re dealing with flaky tests in your setup.

pytest-html-plus VS Code — A Must-Have Extension If You Run Pytests

Kevi — Sat, 24 Jan 2026 08:54:06 +0000

The problem no one talks about

HTML reports even our own pytest-html-plus are usually opened like this:

Download artifact
Unzip
Open HTML
Scroll
Switch back to code
Switch back to browser
Repeat
It works — but it breaks flow.
When you’re debugging, context switching is the real tax.

That’s exactly where pytest-html-plus-vscode fits in.

Reports should live where you work

_Your reports must be in the same place where you are fixing the bugs, no matter where your tests might have ran

The VS Code extension lets you:

Load reports directly inside the editor
View structured test results without leaving your workspace
Jump between failures and code faster
Treat test reports as part of development, not an external artifact
This is especially powerful when debugging CI failures locally.

Why this extension exists

The extension was not built to “preview HTML”.

It was built to answer one question quickly:

What failed, and what do I need to look at next? and Everything else is secondary.

What you actually gain

1. Faster feedback loops

You no longer lose momentum opening files, switching apps, or searching logs.

Failures (List them in the side bar) → context (Direcly jump to the line of failure) → fix.

All inside VS Code.

2. CI reports, locally

The extension works beautifully with archived reports from CI.

Download the artifact, open it in VS Code, and you’re immediately back in problem-solving mode — not setup mode.

Because the reporter and extension are designed together:

The data structure is predictable
Reports load faster
UI decisions are intentional, not generic
This tight coupling is what makes the experience smooth instead of clunky.

Who should install this?

If you:

Run Pytest regularly
Debug from CI logs
Care about flow state
Live in VS Code

This extension isn’t “nice to have”.

It becomes one of those tools you forget you installed — until you work on a machine without it.

The bigger picture

pytest-html-plus are not separate tools.

They’re two halves of the same idea:

Testing doesn’t end when Pytest finishes.
It ends when you understand the result.

Final note

You don’t need more tools.
You need fewer tools that work together.

That’s the philosophy behind both the reporter and the extension.

From Repeating Boilerplate to One-Click Insights: My Journey with pytest-html-plus

Kevi — Tue, 30 Sep 2025 05:41:56 +0000

When you’re running tests across multiple branches and environments, one question always comes up:

“Okay, but which commit and which env did this report come from?”

At first, we were using pytest-html. It’s a solid plugin — if you have time for writing the basic hooks in conftest, it gives you a clean HTML report out of the box. But even after that, there was one big missing piece: execution metadata.

No branch info.
No commit ID.
No environment label.

Basically, the report looked fine, but without context it was just a wall of green/red.

The Boilerplate Phase

Like most devs, we hacked around it.

Added hooks in conftest.py
Pulled in env vars for branch/commit
Injected them manually into the report header

It worked. The first time, it even felt kind of cool — “hey, now our report shows branch and commit!”

But then came the second project.
And the third.

Every repo needed the same glue code. If we fixed one thing, we had to fix it everywhere. The overhead became annoying. Honestly, it felt like I was solving the same problem again and again instead of writing tests.

After repeating the same boilerplate across projects, I started looking around.

Allure was the first obvious option — but honestly, it felt like way too much setup and bloat just to view something as basic as branch, commit, or env metadata and asking my manager to setup java in their system to open the report. I didn’t want dashboards, databases, and 10 extra moving parts. I just wanted a simple HTML report with context.

Thanks to Gemini, It suggested me pytest-html-plus and their screenshot contained the relevant execution metadata I needed which I first thought would have to be configured but

It just… had the metadata baked in.

Branch name?
Commit ID
Environment pulled from cli args

All sitting neatly at the top of the same single-page HTML I was already used to. No extra config, no repetitive hooks, no “copy this snippet to every repo.”

We Also removed all of our report generation hooks in the conftest

The best part of it was, it was easily shareable being a self contained html and no setups needed to view by our business.

Why This Changed Things

It sounds small, but it completely changed how we shared reports:

No more digging through CI logs to see which run this was.
No more maintaining copy-pasted conftest logic.

Just open the HTML, and all the context is right there.

For me, it turned the report into something I could actually trust at a glance.

If you’re already on pytest-html and tired of wiring up metadata by hand, give pytest-html-plus a look. For us, it was the difference between “yet another report” and “a report with real context.”

I stopped using plugins to merge JUnit XMLs in Pytest (pytest-html-plus)

Kevi — Tue, 16 Sep 2025 05:09:23 +0000

For the longest time, I thought generating XML reports from Pytest was just… annoying.

Run tests with -n for parallel workers → I’d get 4 different XML files or more with increasing number of tests.
Retry a flaky test → extra XMLs scattered around.
Then I’d have to either write my own merge script or dig up some plugin to stitch them back together.(Not to mention the xml paths relative and absolute and file not found stuff or generating the blobs)
Finally, I’d spend time debugging why my CI job wasn’t uploading the “right” XML to TestRail.

All of this felt like extra work just to give my test management tool a single file and devops guys who are already loaded and this would be a P3 for them to look at.

While searching for something that would automatically generate and merge, I stumbled upon this underrated tool called pytest-html-plus, and I was surprised that it quietly solved this problem for me.

All I had to do was:

pytest --generate-xml --xml-report=testrail.xml

And it gave me one combined XML, even with parallel runs and retries. No extra merge step, no custom code.

The XML already included logs, stdout/stderr, flaky test information, screenshots etc. Uploading to my test management tool was finally a single step instead of three, supposedly made my sync with devops team much lesser as they only needed to use the test rail command to upload the xml.

If you are using pytest and trying to link your tests with an test management tool, try this before going for other CI plugins

https://github.com/reporterplus/pytest-html-plus

Phew!

Discovered how easy can traceability in test become with smart(not just good) reporting tools. pytest-html-plus

Kevi — Wed, 03 Sep 2025 08:51:50 +0000

One of the challenges in test automation is keeping track of where each test belongs:

Which JIRA ticket does it cover?
Is there a linked test case ID in Testmo or Notion?
Can I see bug reports linked directly from failing tests?

Normally, this means a lot of manual mapping, plugins, or custom scripts. Also, with bloated reporting tools its more of navigation than instantly doing it. plus the way to enable is to configure their own custom markers.

I was surprised to see that in pytest-html-plus, it’s already built in and accommodated very much in their single page reporter.

How It Works

The HTML report comes with search bar. Just type(either partial or full) and tests are instantly filtered by:

Test names
Linked external IDs (JIRA-123, DOC-456, etc.)
Custom URLs or keywords

That means if you tagged your tests with a JIRA issue or a test case ID, you can just type that ID in the report and immediately see all related tests

You don’t need any custom logic. Just use standard pytest.mark decorators:

import pytest

@pytest.mark.jira("https://jira.example.com/browse/PROJ-123")
@pytest.mark.testcase("https://testcases.example.com/case/5678")
def test_login():
    assert True

That’s it.
When you generate the report, these links show up next to your test case, clickable, so you can jump directly to JIRA, Notion, or your test case system.

Tracing Coverage

Searching for a quoted ID (like PROJ-123) shows you all tests linked to that reference.

Before I could think what about the cases where we may have missed tracking? I saw the filter called, filter for unlinked tests (using the “Show Untracked” filter) to see which ones aren’t mapped to any external system yet and this was a game changer, I could easily and tag all the test cases with the relevant JIRA and bug ids and this has become our new metric in my company.

This also makes it easy to:

Shows how many test cases we have automated
Check test automation coverage during release cycles
Group results by feature or ticket
Spot untracked cases that need linking

Why I Like This

No extra setup. No additional plugins. No scripts to maintain.
It’s just pytest.mark + run tests → open report → search and navigate. We did not need a huge test management tool to do this job.

Traceability is one of those things that’s usually painful, but here it feels natural.

That’s all — clean, built-in traceability without extra effort.

You can checkout the report by installing and running your pytest

pip install pytest-html-plus

Frankly, installation is all what matters to generate this wonderful report, so I did not mention any commands about pytest as you can use them as usual.

Check out the their plugin and the repo here:

https://github.com/reporterplus/pytest-html-plus

https://pypi.org/project/pytest-html-plus/

No More “How to Create Pytest HTML Reports or how to email test reports"

Kevi — Sat, 16 Aug 2025 09:40:15 +0000

Today I ran into something funny — I needed a decent HTML report for my pytest run, and my first thought was:
another half-hour lost to figuring out pytest-html configs.

Usually, that means:

Installing pytest-html or allure-pytest
Setting up .ini or passing a long list of CLI flags (or even installing Java in allure’s case)
Making sure people on other machines can actually open the report
Writing a little conftest.py hook to add metadata
Trying not to break CI/CD in the process

Time spent? Way too much, just for a simple report.

And in my case, there was another catch — I was running tests in parallel with pytest-xdist.

If you’ve done that before, you will need another merger plugin to merge all the xmls

So I looked around for that plugin but found pytest-html-plus.

I didn’t even read the docs at first — I just installed it:

pip install pytest-html-plus pytest -n auto

Done.

That’s it.
…and it just worked.
Full HTML report, already pretty, self-contained, no setup, no config.

What surprised me:

It handled parallel runs without breaking the report (even with xdist)
Added metadata like branch, commit, environment, and timestamp automatically

I could copy logs and errors with one click

The report was shareable as-is without worrying about dependencies

I didn’t have to touch a single test file — no extra hooks or decorators

Honestly, the whole “report” part of my workflow went from “ugh, I’ll do it later” to “done in 5 seconds”, even in CI.

If you’ve been wrestling with HTML reports in pytest — especially with parallel runs — this was a nice surprise.

Even better part is that I could email the test report by just triggering --plus-email

🔗 https://pypi.org/project/pytest-html-plus/