Integrating Allure with TestRail
Allure (opens in a new tab) is popular language and framework agnostic reporting Framework. Railflow CLI and CI plugins can process Allure reports and map test results back to TestRail.
Railflow
provides a convenient JAVA annotations package for JUnit framework which enables users to integrate their JUnit project with TestRail in a variety of flexible ways. Railflow JUNIT Annotations package provides the following capabilities
Test Mapping: Map JUnit tests existing tests in TestRailAttach Screenshots: Attach screenshots and arbitrary attachments to the test case results in TestRail.Custom Fields: Support for TestRail custom fields.TestRail Entities: Update / Create Test Plans, Runs, Milestone, Tests, ResultsCI Application: Support for Jenkins, TeamCity, Gitlab, Github, etc.
Getting Started
When using the Railflow annotation package, JUnit produces an enriched xml test report railflow_report.xml. This report is then processed by Railflow CI Plugins or the CLI to upload the tests and test results to TestRail.
Maven Setup
This is example is using Maven as the build tool. You can use any other build tool like Gradle, Ivy, etc.
:::info Railflow (opens in a new tab) platform provides support for many popular test frameworks including Allure (opens in a new tab). It allows users to integrate their Allure tests with TestRail easily and provides the following capabilities:
- Map Allure tests into one or several test cases in TestRail by providing test case IDs.
- Attach screenshots to the test case results in TestRail. :::
How do Allure and Railflow works together
Each time users run Allure tests with their favorite testing framework, Allure generates a bunch of JSON report files with different format (depending on the particular test framework).
Those report files are needed to generate Allure HTML report and also unified JSON report files which can be then fed to Railflow for creating/updating required entities in TestRail.
To generate those reports Allure provides a special generate command (opens in a new tab). This command generates the allure-report folder with the following structure:
JSON report files are located in allure-report/data/test-cases folder.
These JSON report files need to be fed to Railflow in order to export results into TestRail.
Railflow creates or updates the following entities in TestRail:
- Sections and subsections
- Test cases
- Test plan and/or test run and test results
- Milestones and run configurations (optional)
Having written above, there are three required steps to integrate Allure reports with TestRail:
- Run the tests and create "intermediate" test reports
- Run
allure generatecommand to generate HTML and "unified" JSON report files - Run Railflow CLI or use CI Plugins and feed "unified" JSON report files to it.
Allure with TestRail integration example
Allure supports variety of test frameworks like Cucumber (opens in a new tab), JUnit (opens in a new tab), TestNG (opens in a new tab), PyTest (opens in a new tab) and many more.
For the example below we will be using JUnit 5 together with selenium and Apache Maven, but the configuration and steps are quite similar to any other test framework.
Prerequisites
- Maven 3
- JUnit 5
- Allure Framework
Create a maven project and modify pom.xml by adding JUnit 5, Allure, and Selenium libraries
Test Class : AllureDemoTest.java
LoginPage.java
WebdriverTestWatcher.java
Run tests and generate JUnit 5 report
mvn clean testThis command will generate different intermediate allure report files in $project_directory/target/allure-results directory.
Generate Allure JSON reports
Run allure generate command to generate unified report from the intermediate report files located in the $project_directory/target/allure-results directory:
allure generate $project_directory/target/allure-results --clean -o $project_directory/target/allure-report:::note
You can also run mvn allure:report to generate the report. In such a case report files will be located in $project_directory/target/site directory.
:::
This will generate HTML and unified allure JSON report files in $project_directory/target/allure-report directory:
Note that JSON files are located in $project_directory/target/allure-report/data/test-cases directory. Those files should be provided to Railflow.
Install Railflow CLI
npm install railflowRun Railflow CLI and upload test results into TestRail
npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path section1/section2 -f allure -r $project_directory/target/allure-report/data/test-cases/*.json -sm pathWhere:
| Key | Description | Example |
|---|---|---|
| -k, --key | (online activation) License key. Can be set with RAILFLOW_LICENSE environment variable | -k XXXXX-XXXXX-XXXXX-XXXXX |
| -url, --url | TestRail instance URL | -url https://example.testrail.io (opens in a new tab) |
| -u, --username | TestRail username. Can be set with RAILFLOW_TR_USER environment variable | -u test-username |
| -p, --password | TestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable | -p XtpHXiPLEODyhF |
| -pr, --project | TestRail project name | -pr "example project" |
| -path, --test-path | TestRail test cases path | -path "Section1/subsection2/ShoppingCart |
| -f, --format | Report format: 'pytest-railflow' (case insensitive) | -f junit |
| -r, --report-files | The file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/reports/*.xml can be used.E.g. use **target/reports/\*.xml** to capture all XML files in target/reports directory. | -r target/surefire-reports/*.xml target/failsafe-reports/*.xml |
| -sm, --search-mode | Specifies the test case lookup algorithm. name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path path: search for test case matching the name within the specified -path. If test case found, update the test case. If test case not found, create a new test case within specified -path | -sm path |
Please see Railflow NPM documentation (opens in a new tab) for the all the details about Railflow CLI options.
View results in TestRail
Test run details
Test result details
Test result with attachment
