When REST API is tested, one has to take into account a number of low level details. This makes test execution reports verbose, difficult for reading and complicates the maintenance of the test scripts. For instance, the backend of an application consists of about 15 services and a separate service, that provides API for the mobile client. The minimal set of integration tests contains about 400 tests, which create more than 70000 rows of log files. To sort out such size of the text file, generated by only one run of the test is not feasible, even with joint effort of the entire QA department.

In this article we will examine how to make tests execution reports more comprehensive, using a specific tool – Allure, created by Yandex company. Yandex Allure is one of the solutions to the task for the design of the testing reports.

In the first part of this article we demonstrate how to connect Allure to the corresponding project, create and view the report. In the second part we briefly examine the contents of the generated report and how the results of the tests execution are presented. Further, we describe features, which facilitate everyday work with tests: attaching extra files to reports, splitting test cases into steps, adding links to the third party sites, such as a bug tracking system, outputting the data about environment.

In the examples, provided in this article, we use Java 8, Maven 3 + Surefire plugin, JUnit 4, Jenkins.

How to generate and see the report:

As the first step, connect Allure framework to your project, launch tests and view the report locally in Jenkins.

How to connect JUnit adapter via Maven

In order to connect Allure to your project, add Allure adapter to JUnit4 into section project->dependencies of the pom.xml file:

<dependency>
  <groupId>io.qameta.allure</groupId>
  <artifactId>allure-junit4</artifactId>
  <version>$LATEST_VERSION</version>
</dependency>

There also exist adapters for other popular testing frameworks, for example JUnit5 and TestNG. The integrations with popular libraries are being added and developed. A complete list of integrations you can see in the repository, following the link.

Further, you need to tune a Surefire plugin for collecting results after the tests run. In the pom.xml file the reference to this plugin should be written in the following way in the project->build->plugins section:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>${surefire.version}</version>
  <configuration>
    <argLine>
      -javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
    </argLine>
    <properties>
      <property>
        <name>listener</name>
        <value>io.qameta.allure.junit4.AllureJunit4</value>
      </property>
    </properties>
  </configuration>
  <dependencies>
    <dependency>
      <groupId>org.aspectj</groupId>
      <artifactId>aspectjweaver</artifactId>
      <version>${aspectj.version}</version>
    </dependency>
  </dependencies>
</plugin>

Listener is a class that processes such events as testStarted, testFinished, testFailure and other. Standard AllureJunit4 listener stores results of the test run in the files of the JSON format. AspectJ Weaver library allows Allure to collect information about methods, including private ones. Without connecting this library, the report will be created, but there won't be any information about the steps of the test case in it. It should be noted, that custom settings of Allure features are passed through settings of the Surefire plugin, not impacting other parts of the project.

Tests launch

In order to launch tests, using Maven+Surefire it's necessary to use the same command, that is used without the plug in Allure.

mvn clean test

Tests results in JSON format are located in the project folder target/surefire-reports by default.

This directory can be specified in the Surefire settings section of the pom.xml file project->build->plugins:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  ...
  <configuration>
    <systemProperties>
      <property>
        <name>allure.results.directory</name>
        <value>
         ${project.build.directory}/${customResultFolder}
        </value>
      </property>
    </systemProperties>
  </configuration>
</plugin>

Building and viewing reports

You can build and view the prepared report either via console application or via the employed project comprehension tool, which is Maven in our case. We won't go into details in the first case, noting that this application is available for Linux, MacOS and Windows operating systems. Details about the console application can be found following this link.

In order to launch the report building via Maven, first, add a plug-in to pom.xml file:

<plugin>
  <groupId>io.qameta.allure</groupId>
  <artifactId>allure-maven</artifactId>
   <version>$PLUGIN_LATEST_VERSION</version>
</plugin>

Then, in order to build a report, execute the following command in the console:

mvn allure:serve

If everything has been set and configured correctly, the finished report will be opened in the web-browser on the local HTTP server.

One may also build a report, without running it in a browser. For this purpose, execute in the console:

mvn allure:report

The finished report will be placed to target/site/allure-maven-plugin directory. And one can view it, deploying it on the local server.

Example: In Intellij Idea . Open context menu for the file target/site/allure-maven-plugin/index.html Select menu entry Open in Browser -> then select browser you need -> and report opens up.

Integration with CI server

Besides running tests locally and viewing reports, Allure can also be used in the continuous integration (CI). For the quicker and cheaper defects discovering in CI, the frequent project builds and consequently test runs are used.

Example: Tests are launched automatically:

• every time the master branch of the definite service is updated – sanity tests for this service.
• everyday: complete set of tests (for all services) on the dedicated SQA environment.
• every week: after new versions of services are deployed – full set of tests on the environment for the manual testing.

Results of these test runs can be published for the entire development team in the form of the Allure reports on the CI-server. Currently Jenkins plugin is used to include Allure reports to CI process. There are plug-ins for other CI tools, such as TeamCity, Bamboo. Details can be found in the Allure official documentation.

Report templates:

Practical guide for making tests executions

Finished reports is a web page with a few tabs. On each tab, the results are presented in a specific way. The report is published for the entire development team and each member of the team can select one of the several tests executions report templates.

Overview

Overview is the start page of the report. It contains a few widgets, some of which allow you to navigate to other tabs and some of which provide additional data: the information about environment, total number of tests, percentage of passed / failed tests, graph that displays information of the recent results of the tests run.

Suits and Packages

Under the Suits and Packages tabs, the results are grouped by classes, which are in the code of tests. In addition to this representation, the display on the packages tab repeats the structure of your tests, like a packages tree. This method is handy for developers and QA engineers who work with the tests code.

Behaviour

The behaviour tab is handy for other members of the team. Here the results are grouped, according to the behaviour of the system under the test. Here are three levels of hierarchy: Epics, Feature, Story. While the results displayed under the Suites and Packages tab are grouped automatically, the results on this tab in the corresponding branch will be formed only if a test method or a class in the test code is entitled by the annotation @Epic, @Feature, @Story correspondingly. You needn't use all three levels. Unmarked methods will be displayed here just as a list.

Categories

The defects are displayed under this tab. There are two standard defect categories: product defects – defects of the fallen tests and tests defects – tests that have to be fixed. You also have an opportunity to add custom test categories.

Graphs

Under this tab the results of the tests are displayed in the form of graphs: passed / failed tests, tests distribution, according to the duration and severity.

Timeline

Under this tab tests are shown on the timeline, according to the order they were launched and how long they were executed. Here you can see the results of the tests, that were launched simultaneously. Tests being executed in different threads are shown separately.

Useful Allure features

There are several features below, which facilitate in everyday work with tests:

• Environment widget, that displays the data about the test environment.
• Attachments feature is used for the attachment of any file types to the test report.
• Links feature adds to the test case links to the external sites.
• Steps feature allows test cases structuring. All raised exceptions, screenshots and attached files will be displayed in the test report inside the corresponding test step. It helps to determine the step, from which the test failed, not looking through the code of the test.

Below, there are examples of usage and settings of the above mentioned features.

@Steps

Add this annotation for methods, which you want to mark as separate steps of the scenario. These steps will be displayed in the test report in the form of a tree. There is a demo-report, which shows how it looks. If the method is marked as a step, then all error messages, attached files, included steps will be shown in the report inside of this step.

Mark methods of the client API as steps:

• Only important variables should be located in the step header.
• Stick to one format for all client methods.
• Indicate to which service or API group a particular method relates.

Example of the method for the update of the user profile.

@Step("User: Update profile [user id: {userId}]")
    public Response updateProfile(
        final String accessToken,
        final String userId,
        final JSONObject body
    ) {
        Response response = given(accessToken)
            .body(body.toString())
            .when()
            .put(PREFIX + "/profile/{userId}", userId);
        response.then().contentType(ContentType.JSON);
        return response;
    }

The screenshot below shows how a step in the report is displayed.

Check the methods of the data preparation and removal as steps

In particular:

• If you use JUnit ExternalResource method, mark 'before' and 'after' methods;
• If you mark JUnit method with annotation @Before or @After, mark it with the annotation @Step;
• In the current version Allure for JUnit static methods marked as @BeforeClass and @AfterClass and their analogues in ExternalResource used as @ClassRule are not displayed in the report.

@Attachment

Use this annotation to add HTTP requests, responses and other files to the report.

Screenshots attachment

@Attachment(value = "Page screenshot", type = "image/png")
public byte[] saveScreenshot(byte[] screenshot) {
    return screenShot;
}

HTTP requests attachment

@Attachment(value = "{method} {url}", type = "text/plain")
private static String saveRequest(
   final String url,
   final String method,
   final String body,
   final Map<String, Object> headers
) {
   StringBuilder result = formatRequestSomehow(url, method, body, headers);
   return result.toString();
}

*HTTP responses attachment

@Attachment(value = "{code} {reasonPhrase}", type = "text/plain")
    private static String saveResponse(
        final String body,
        final int code,
        final String reasonPhrase
    ) {
        StringBuilder result = formatResponseSomehow(code, reasonPhrase, body);
        return result.toString();
    }

There are other ready for use solutions for other HTTP clients, with the help of which, using only one string of code one can attach all requests and responses to the report.

Attachment requests and responses for Rest Assured

Add library allure-rest-assured to your project:

<dependency>
   <groupId>io.qameta.allure</groupId>
   <artifactId>allure-rest-assured</artifactId>
   <version>$LATEST_VERSION</version>
</dependency>

While configuring the client, add filter AllureRestAssured

RequestSpecification spec = prepareSpecSomehow();
spec.filter(new AllureRestAssured());

@Link

This feature allows you to attach links to the test method. By default, 3 types of links are available: @Link, @TmsLink, @Issue.

Configure basic paths:

For @TmsLink this is the path to the test management system, for @Issue – this is the path to the bug tracking system.

Example: settings for Maven + Jira. As it was mentioned above, these settings are for the Surefire plugin:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>${surefire.version}</version>
  <configuration>
    … 
    <systemProperties>
      …
      <property>
        <name>allure.link.issue.pattern</name>
        <value>
          https://myjiranamespace1.atlassian.net/browse/{}
        </value>
      </property>
      <property>
        <name>allure.link.tms.pattern</name>
        <value>
          https://myjiranamespace2.atlassian.net/browse/{}
        </value>
      </property>
    </systemProperties>
  </configuration>
  … 
</plugin>

Add links to the methods:

@Test
    @TmsLink("MyTMSNamespace-1999")
    @Issue("MyBugTrackerSpace-2999")
    public void makeFriendsAndGetFriendlist() {
	… 
}

Here is the result:

First link (with bug image) navigates to: https://myjiranamespace1.atlassian.net/browse/MyBugTrackerSpace-2999

second one navigates to: https://myjiranamespace2.atlassian.net/browse/MyTMSNamespace-1999

Environment widget

It is located on the Overview tab. Here the status of the testing environment can be displayed.

In order to display the information here, add file enviroment.properties to the folder, which keeps test run results. By default it is the folder target/surefire-reports.

Example: versions and statuses for two services in enviroment.properties

Social           0.12.0-20180521-1 (master) [healthy]
User             0.15.0-20180516-1 (master) [healthy]

Result:

Conclusion

This way, Allure presents results in a structure, allowing one to work with more comfortably, than with raw data. It's especially important if we mean a huge number of tests, unstructured tests, which you need to make uniform, complex cases, consisting of many steps. The article has covered the experience of REST API testing, but tools, described in the article will be successfully used in other types of testing. So, for UI tests it will be useful to attach screenshots to the report and store the testing environment configuration, for instance browsers versions. Besides, this tool supports many popular software programming languages and test frameworks.

Links mentioned in this blog post:

Allure repository on GitHub: https://github.com/allure-framework/allure2 Java integrations for Allure: https://github.com/allure-framework/allure-java Official Allure documentation: https://docs.qameta.io/allure/ Jenkins plugin for Allure: https://github.com/jenkinsci/allure-plugin Demo-report: https://demo.qameta.io/allure/