Allure Report Hands-on Guide

Allure Report is an open-source, multi-language test reporting tool. It builds a detailed representation of what has been tested end extracts maximum of from the everyday tests execution.

In this guide, we’ll take a journey through the main steps to creating your first Allure report and discover all the fancy features that it brings to routine automated testing reports.

As far as Allure Report has various integrations with various testing frameworks on different programming languages, there is a chance that some steps will vary for each reader, so feel free to jump into the official documentation page for details.

Installing Allure Report

As always, the first step is to install Allure library. The exact steps vary depending on your OS:

Linux

For debian-based repositories, a PPA is provided:

sudo apt-add-repository ppa:qameta/allure
sudo apt-get update
sudo apt-get install allure

Mac OS X

For Maс OS, automated installation is available via Homebrew:

brew install allure

Windows

For Windows, Allure is available from the Scoop commandline-installer.

To install Allure, download and install Scoop, and then execute in the Powershell:

scoop install allure

Manual installation

Download the latest version as a zip archive from GitHub releases.

• Unpack the archive to the allure-commandline directory.
• Navigate to bin directory.
• Use allure.bat for Windows or allure for Unix platforms.
• Add allure to system PATH.
• Allure CLI requires Java Runtime Environment to be installed.

Plugging Allure into the code

Next step is to provide all the necessary dependencies within the configuration file, so the build tool would be able to use Allure.

Each framework and build tool has its own configuration settings, so the best way to get a clue on adding dependencies is to look for an example at the documentation page or get an example at GitHub.

Good news, everyone! Qameta Software, the Allure Report maintainer, is developing a GUI-based template projects generator. The templates will be standalone examples with a number of mock tests and all the infrastructure presets! Stay tuned and subscribe to our blog to get your hands on the tool among the first.

Adding annotations to the Allure Report

After plugging the Allure Report into the codebase, we can run it. However, the report will be empty.

In order to provide all the necessary data to Allure, we need to annotate the tests. There are several types of annotations:

• Descriptive annotations that provide as much information about the test case and its context
• Step annotation, the one that allows Allure to build nice and detailed test scenarios
• Parameterizing annotations that allow you to specify

Allure Report Descriptive annotations

@epic, @feature, @story

A set of annotations designed to make test-case tree grouping more flexible and informative. The annotations follow Agile approach for tasks definition. These annotations may be implemented on the class or on the method level.

Epic defines the highest-level task that will be decomposed into features. Features will group specific stories, providing an easily-readable structure.

As story is the lowest part of epic-feature-story hierarchy, the class-level story adds data to all methods of the class.

@Description

An annotation that provides a detailed description of a test method/class to be displayed in the Allure Report.

@Owner

Simple annotation to highlight a person behind the exact test case, so everyone would know whom to ask for a fix in case of a broken/failed test. Quite useful for large teams.

@Severity

In Allure, any @test can be defined with a @Severity annotation with any of these values like BLOCKER, CRITICAL, NORMAL, MINOR, TRIVIAL.

The severity level will be displayed in the Report, so the tester understands the severity level of a test if Failed.

Sample Tests

You may find the example of all these annotation usage in Java code, annotations for any other technology will look quite similar:

public class AllureExampleTest {

    @Test
        @Epic("Sign In flow")
        @Feature("Login form")
        @Story("User enters wrong password")
        @Owner("Nicola Tesla")
        @Severity(SeverityLevel.BLOCKER)
    @Description("Test that verifies a user cannot enter the page without logging in")
    public void annotationDescriptionTest() {
    }

    /**
     * JavaDoc description
     */
    @Test
    @Description(useJavaDoc = true)
    public void javadocDescriptionTest() {
    }
}

Allure Report Step Annotation

Detailed reporting with steps is one of the features people love about Allure Report. It is the @step annotation that makes it possible by providing a human-readable description of any action within a test. Steps can be used in various testing scenarios. They can: be parametrized, make checks, have nested steps, and create attachments. Each step has a name.

In order to define steps in code, each method should have a @step annotation with String description; otherwise the step name is equal to the annotated method name.

Note that steps mechanics were revised and now support smart fields analysis. In Allure 1, users had to specify indexes to refer to which args’ values they wanted to inject into a step. Allure 2 uses a reflection-based approach, which provides deep fields’ extraction by their names.

In code, the Step annotation will look like this:

package io.qameta.allure.examples.junit5;

import io.qameta.allure.Allure;
import io.qameta.allure.Step;
import org.junit.jupiter.api.Test;

public class AllureStepTest {

    private static final String GLOBAL_PARAMETER = "global value";

    @Test
    public void annotatedStepTest() {
        annotatedStep("local value");
    }

    @Test
    public void lambdaStepTest() {
        final String localParameter = "parameter value";
        Allure.step(String.format("Parent lambda step with parameter [%s]", localParameter), (step) -> {
            step.parameter("parameter", localParameter);
            Allure.step(String.format("Nested lambda step with global parameter [%s]", GLOBAL_PARAMETER));
        });
    }

    @Step("Parent annotated step with parameter [{parameter}]")
    public void annotatedStep(final String parameter) {
        nestedAnnotatedStep();
    }

    @Step("Nested annotated step with global parameter [{this.GLOBAL_PARAMETER}]")
    public void nestedAnnotatedStep() {

    }

Allure Report Parameters Annotations

Attachment

The annotation allows attaching a String or Byte array to the report. The annotation is used to attach a screenshot or a failure stack trace to the result.

Link / Links

Well, that’s simple. If you need to add a link to the test, be it a reference or

It takes a number of parameters:

name: link text
url: an actual link
type: type of link
value: similar to name

Muted

An annotation that excludes a test from a report.

TmsLink

A way to link a result with a TMS object, if you use any. Allows entering of just test case id that will be added to the pre-configured (via allure.link.tms.pattern) URL. The annotation takes a String value, which is the link to the management system. For example, if our test case on TMS link is https://tms.yourcompany.com/browse/tc-12, then we can use tc-12 as value.

Allure Report Launch

Local

A great way to get started with Allure Report is to run it locally. Local execution does not provide execution and results history, as well as the trend graphs.

The best way to give Allure Report a try is to open an example with an IDE. Check all the dependencies and build the project with Gradle.

After that, we need to run the tests with the ./gradlew test command. When the tests are executed, Gradle will store the results in the target directory.

Let’s take the data and build a report! With the allure serve /path/to/the/project/allure-example/build/allure-results command, we start the Allure Report instance and build a local web-report which automatically opens as a page.

CI (Jenkins, TeamCity and Bamboo)

Allure Report has several neat integrations with different CI-systems. Each system setup has its specificities, so we won’t cover them in this post. Follow the Documentation page for steps to create a report with a CI-System.

Allure Report Features

Parameterized tests

Allure knows how to work with parameterized automated tests. Let’s take a JUnit test as an example. First, let’s create a test class with parameterized tests of the following kind:

@Layer("rest")
@Owner("baev")
@Feature("Issues")
public class IssuesRestTest {

    private static final String OWNER = "allure-framework";
    private static final String REPO = "allure2";

    private final RestSteps steps = new RestSteps();

    @TM4J("AE-T1")
    @Story("Create new issue")
    @Microservice("Billing")
    @Tags({@Tag("api"), @Tag("smoke")})

    **@ParameterizedTest(name = "Create issue via api")**

    @ValueSource(strings = {"First Note", "Second Note"})
    public void shouldCreateUserNote(String title) {
        parameter("owner", OWNER);
        parameter("repo", REPO);
        parameter("title", title);

        steps.createIssueWithTitle(OWNER, REPO, title);
        steps.shouldSeeIssueWithTitle(OWNER, REPO, title);
    }

As you can see, after execution, Allure provides the parameterized test run results as a set of tests. If any of the tests fails, Allure provides detailed information about that particular case.

Categories

Categories is one of the most time-saving features of Allure Report. It provides simple automation for fail resolution. There are two categories of defects by default:

• Product defects (failed tests)
• Test defects (broken tests)

It is fully customizable via simple json configuration. To create custom defects classification add categories.json file to allure-results directory before report generation.

Open json template [ { "name": "Ignored tests", "matchedStatuses": ["skipped"] }, { "name": "Infrastructure problems", "matchedStatuses": ["broken", "failed"], "messageRegex": ".*bye-bye.*" }, { "name": "Outdated tests", "matchedStatuses": ["broken"], "traceRegex": ".*FileNotFoundException.*" }, { "name": "Product defects", "matchedStatuses": ["failed"] }, { "name": "Test defects", "matchedStatuses": ["broken"] } ]

The json includes the following data:

• (mandatory) category name
• (optional) list of suitable test statuses. Default [“failed”, “broken”, “passed”, “skipped”, “unknown”]
• (optional) regex pattern to check test error message. Default “.“
• (optional) regex pattern to check stack trace. Default “.”
• Test result falls into the category if its status is in the list and both error message and stack trace match the pattern.

In case of using allure-maven or allure-gradle plugins, categories.json file can be stored in test resources directory.

Retries

Retires are executions of the same test cases (signature is calculated based on test method name and parameters as well) within one test suite execution, e.g. when we are using TestNG IRetryAnalyzer or JUnit retry Rules. Not supported for local runs.

Test History

Allure 2 supports history for tests in the report. At each report generation during the build, Jenkins, Allure Plugin will try to access the working directory of the previous build and copy contents of allure-report\\history folder to the current report.

At the moment history entry for test case stores information about up to 5 previous results. Not supported for local runs.

Report Structure and Dashboards

Overview

The default page would be the ‘Overview’ page with dashboards and widgets. The page has several default widgets representing basic characteristics of your project and test environment:

• Statistics – overall report statistics.
• Launches – if this report represents several test launches, statistics per launch will be shown here.
• Behaviors – information on results aggregated according to stories and features.
• Executors – information on test executors that were used to run the tests.
• History Trend – if tests accumulate some historical data, its trend will be calculated and shown on the graph.
• Environment – information on test environment.

Home page widgets are draggable and configurable. Also, Allure supports its own plugin system, so quite different widget layouts are possible.

Categories

The page shows total defects. If the test is an assertion failure, it reports as ‘Product defect’ and if a test has failed because of some exception, it shows the report as ‘Test defect’.

Suites, Behaviors and Packages

Three tabs that show the test case tree built on different criteria:

• Suites display test cases based on the suite executed.
• Behaviors tab builds the tree based on stories and features annotations.
• Packages tab shows the test cases grouped by package names.

Graphs

The tab for testing results visualization with charts. The default configurations provide:

• General execution results pie chart.
• Test Case execution duration and duration trend. Nice feature to investigate which tests take more time and need optimization.
• Retries and Categories trends

Timeline

The view displays a timeline of executed tests, quite a self-explaining graph.

That's it!

Qameta Software focuses on developing amazing tools that help software testers. Learn more about Allure Framework, a lightweight automation reporting tool, and Allure TestOps, the all-in-one DevOps-ready testing platform.

Follow us on LinkedIn or Twitter, or ask for assistance on GitHub Discussions.

Comments

[Pritesh Usadadiya]

[[..PingBack..]]
This article is curated as a part of #60th Issue of Software Testing Notes Newsletter.

[Sakshi Singla]

Hi, you mentioned that Home Page widgets are configurable. What is the way to add a new widget on HomePage? Or add a new tab to the report? @ruslan_testops