DEV Community

Harshad Lambate
Harshad Lambate

Posted on

Your test passed on retry. Your report should say so.

Sample Report

Most Java test suites have a retry somewhere: an IRetryAnalyzer in TestNG, @RetryingTest from JUnit Pioneer, or Surefire’s rerunFailingTestsCount. Retries keep pipelines green. They also hide information.

When a test fails, gets retried and passes, most reports show one word: Passed. The failure that happened two minutes earlier, along with its stack trace, its API calls and the screenshot of what the browser showed, is gone. The test is flaky, and the report hides it.

The second problem is quieter. To get a useful report, teams end up writing reporting code inside their tests:

`java
test.log(Status.INFO, "Calling POST /users");
test.log(Status.INFO, "Response: " + response.asString());
Allure.addAttachment("Screenshot", new ByteArrayInputStream(png));`
Enter fullscreen mode Exit fullscreen mode

That code isn’t testing anything. People forget to add it, it drifts out of date, and it lands in every test for the rest of the project’s life.

I built Ace Reports to fix both problems. Here’s what it does in Java.

One dependency, no reporting code
xml

`<dependency>
    <groupId>io.github.acereports</groupId>
    <artifactId>acereports-junit5</artifactId>  <!-- or acereports-testng / acereports-cucumber -->
    <version>1.1.0</version>
    <scope>test</scope>
</dependency>
Enter fullscreen mode Exit fullscreen mode

`

There are no listeners to register, no base class and no annotations. Run mvn test as usual, and a single self-contained HTML file appears in TestResults/.

Every attempt keeps its own history

A test that fails and then passes on a retry is shown as flaky, with one tab per attempt. Each attempt keeps:

its own log lines,
its own error and stack trace,
its own API calls,
its own screenshots.

So when attempt 1 timed out and attempt 2 passed, you can see exactly what was different. Pass, fail, flaky and retried counts sit at the top of the report.

API calls are recorded automatically

This is a typical API test with no reporting code in it:

java

`
class UsersApiTest {
private final HttpClient client = HttpClient.newHttpClient();

@Test
void createsUser() throws Exception {
    HttpResponse<String> response = client.send(
        HttpRequest.newBuilder(URI.create("https://reqres.in/api/users"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString("{\"name\":\"John\"}"))
            .build(),
        HttpResponse.BodyHandlers.ofString());
    assertEquals(201, response.statusCode());
}
Enter fullscreen mode Exit fullscreen mode

}
`

The report’s API tab for this test shows the method, URL, request body, response body (JSON indented), status code and duration.

REST Assured calls are captured the same way, with no code.
Authorization and cookie headers are never stored.
One limitation: a client created inside the test method itself can’t be reached automatically, so you wrap it once with AceHttpClient.wrap(...).
Screenshots before teardown

The failure screenshot is taken before @AfterEach / @AfterMethod runs, so the browser is still open. Ace Reports finds a Playwright Page or Selenium WebDriver in a field of the test class, a base class or a ThreadLocal by itself. A page that never navigated anywhere (about:blank) is skipped, so a failed API test doesn’t get an empty screenshot.

Cucumber: steps are the log

With Cucumber (JUnit 5 or TestNG runner), each scenario is a row and every step is a line in the log:


GIVEN the login page is open
WHEN I log in as "InvalidUser" with password "admin123"
FAIL Step failed. TimeoutError: Timeout 3000ms exceeded.
SKIPPED Then the dashboard is shown
Scenario Outline rows carry their example values.
Tags become categories.
Images you attach with scenario.attach(...) show up as screenshot tabs.
Writing to the report, when you want to

You never have to, but you can:

java


Ace.info("Submitting payment");
Ace.takeScreenshot("After login");

Each line attaches to whichever test is running on the current thread, so parallel runs keep their logs apart.

Same report in .NET, Java and Python

Ace Reports started in .NET (xUnit, NUnit, MSTest, Reqnroll) and Python (pytest, pytest-bdd). The Java version produces the identical report, which helps when one team works across more than one stack.

Try it
Live demo report: **https://harshadlambate.github.io/AceReportsWebsite/demo.html
**Website and docs:
https://harshadlambate.github.io/AceReportsWebsite/
Maven Central: https://central.sonatype.com/namespace/io.github.acereports

I’d like to hear how your team handles flaky tests: do you keep the history of each attempt, or only the final result?

Top comments (0)