Docs
Test Frameworks
JUnit

Integrating JUnit with TestRail

JUnit (opens in a new tab) is a popular JAVA testing framework which is used by many teams for both unit and end-to-end testing using tools such as REST Assured, Selenium WebDriver, Playwright, etc.

Railflow provides a convenient JAVA annotations package for JUnit framework which enables users to integrate their JUnit project with TestRail in a variety of flexible ways. Railflow JUNIT Annotations package provides the following capabilities

  • Test Mapping: Map JUnit tests existing tests in TestRail
  • Attach Screenshots: Attach screenshots and arbitrary attachments to the test case results in TestRail.
  • Custom Fields: Support for TestRail custom fields.
  • TestRail Entities: Update / Create Test Plans, Runs, Milestone, Tests, Results
  • CI Application: Support for Jenkins, TeamCity, Gitlab, Github, etc.

Getting Started

When using the Railflow annotation package, JUnit produces an enriched xml test report railflow_report.xml. This report is then processed by Railflow CI Plugins or the CLI to upload the tests and test results to TestRail.

Maven Setup

This is example is using Maven as the build tool. You can use any other build tool like Gradle, Ivy, etc.

Add the Railflow JUnit dependency into pom.xml:

<dependency>
 <groupId>io.railflow.annotations</groupId>
 <artifactId>railflow-junit-annotations</artifactId>
 <version>0.6</version>
 <scope>test</scope>
</dependency>

For JUnit 5, there is no additional configuration needed. railflow-junit-annotations annoations automatically adds required JUnit 5 listener as soon as it's added into the project classpath.

Configure Surefire Plugin (JUnit 4 only)

Configuring the surefire plugin and adding the Railflow dependency will enable the junit project to generate an enriched junit schema compliant report during mvn test. This report contains all the test metadata added via annotation and will be consumed by Railflow CI plugins or CLI.

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.22.2</version>
	<dependencies>
		<dependency>
			<groupId>org.apache.maven.surefire</groupId>
			<artifactId>surefire-junit47</artifactId>
			<version>2.22.2</version>
		</dependency>
	</dependencies>
	<configuration>
		<properties>
			<property>
				<name>listener</name>
				<value>io.railflow.annotations.junit.RailflowJunit4Listener</value>
			</property>
		</properties>
	</configuration>
</plugin>

Register Railflow listener (JUnit 4 only)

Add io.railflow.annotations.junit.RailflowJunit4Listener Junit4 listener in JUnitCore object in your test runner:

import org.junit.runner.JUnitCore;
import io.railflow.annotations.junit.RailflowJunit4Listener;
 
public class MyTestRunner {
  public static void main(String args[]) {
    JUnitCore runner = new JUnitCore();
    runner.addListener(new RailflowJunit4Listener());
    runner.run(YourTestClass.class);
  }
}

You can change the default locatio of railflow_report.xml file using the raillfow report_dir property.

mvn clean test -Dio.railflow.report_dir=target\surefire-reports

Railflow Annotations

The Java Annotation package provides io.railflow.annotations.Railflow annotation which can be applied to test class or test method.

The io.railflow.annotations.Railflow annotation has a number of attributes which have slightly different meaning based on class level or method level applicaiton.

Attributes of the Railflow annotation on method level override similar attributes of the Railflow annotation on class level.

Class Level Annotations

NameDescription
titlename for the subsection which contains the test in TestRail, if not defined
testrailIdsignored
caseFieldsarray of test case field label/value pairs, e.g.@CustomField(name="My custom case field", value="Railflow rocks")
resultFieldsarray of test result field label/value pairs, e.g.@CustomField(name="My custom result field", value="Railflow rocks")
caseTypename of the case type for a test in TestRail, e.g.Automated
casePriorityname of the case priority for a test in TestRail, e.g.High
jiraIdsarray of Jira issue IDs, which will be set as refs in TestRail
smartFailureAssignmentarray of user email IDs, each failed test in the class will be assigned to one of these users in a round-robin fashion

Method Level Annotations

NameDescription
titletest title, if not specified, test method name is used
testrailIdsarray of TestRail test IDs into which the result of this method will be added
caseFieldsarray of test case field label/value pairs, e.g.@CustomField(name="My custom case field", value="Railflow rocks")
resultFieldsarray of test result field label/value pairs, e.g.@CustomField(name="My custom result field", value="Railflow rocks")
caseTypename of the case type for a test in TestRail, e.g.Automated
casePriorityname of the case priority for a test in TestRail, e.g.High
jiraIdsarray of Jira issue IDs, which will be set as refs in TestRail
smartFailureAssignmentuser email ID to assign failed test result to

Screenshots and attachments

When using Selenium WebDriver, it is common to take screenshots of the browser window when a test fails. Railflow provides utility classes and several static methods for taking automatic screenshot so that they can also be uploaded to TestRail along with the failure exception message

  • addAttachment(File attachmentFile) : Adds arbitrary attachment file to the XML report.
  • addAttachment(final String name, final InputStream inputStream) : Adds an attachment file with given name and contents of inputStream
  • addAttachment(final String name, final byte[] data): Adds an attachment file with given name and content specified by the given byte array

Railflow CLI / CI Plugins

The resulting railflow_report.xml report file can be processed and the tests and test results uploaded to TestRail using Railflow CLI or CI plugins.

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/railflow_report.xml

Jenkins plugin

TeamCity plugin

Annotations Examples

Class Level

package io.railflow.annotations.demo;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
 
import org.junit.Assert;
import org.junit.Test;
 
import io.railflow.annotations.CustomField;
import io.railflow.annotations.Railflow;
import io.railflow.annotations.junit.CurrentTest;
 
@Railflow(title = "Railflow on the class level", jiraIds = {"ISSUE-42", "ISSUE-43"}, caseType = "Automated", casePriority = "Critical",
        caseFields = {@CustomField(name = "required text field", value = "class value"), @CustomField(name = "estimate", value = "42s")},
		resultFields = {@CustomField(name = "Custom field", value = "result from class annotations")},
		smartFailureAssignment = { "[email protected]", "[email protected]" })
public class RailflowDemoTest {
 
	@Test
	public void test() throws IOException {
		CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
	}
 
	@Test
	public void test_failed() throws IOException {
		CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
		Assert.fail("oops");
	}
}
 

Results in TestRail after uploading generated railflow_report.xml

Method Level

package io.railflow.annotations;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
 
import org.junit.Assert;
import org.junit.Test;
 
import io.railflow.annotations.CustomField;
import io.railflow.annotations.junit.CurrentTest;
 
public class RailflowDemoTest {
 
    @Railflow(title = "Railflow on the method level", jiraIds = {"ISSUE-44", "ISSUE-45"}, caseType = "Performance", casePriority = "High",
            caseFields = {@CustomField(name = "required text field", value = "method value"), @CustomField(name = "estimate", value = "24s")},
            resultFields = {@CustomField(name = "Custom field", value = "result from annotation on method")}, smartFailureAssignment = {"[email protected]"})
    @Test
    public void test() throws IOException {
        CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
    }
 
    @Test
    public void test_failed() throws IOException {
        CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
        Assert.fail("oops");
    }
}

Results in TestRail after uploading generated railflow_report.xml

CLI Reference

Railflow CLI will automatically create tests, runs, plans, milestones, etc. if they do not exist.

Environment Variables: To avoid exposing sensitive information like TestRail URL, License, username, and password in CLI arguments, users can set the following environment variables: RAILFLOW_LICENSE, RAILFLOW_TR_URL, RAILFLOW_TR_USER, RAILFLOW_TR_PASSWORD. Railflow CLI will always check these environment variables at run time.

⚠️

Use double quotes for argument values with spaces. Example: --project "demo project"

KeyRequiredDescriptionExample
-v, --versionNoOutputs Railflow version number-v
-k, --key-k or -l(online activation) License key. Can be set with RAILFLOW_LICENSE environment variable-k XXXXX-XXXXX-XXXXX-XXXXX
-l, --license-file-k or -l(offline activation) File path or remote url license file-l /files/ActivationFile.skm
-url, --urlYesTestRail instance URL. Can be set with RAILFLOW_TR_URL environment variable-url https://example.testrail.io (opens in a new tab)
-u, --usernameYesTestRail username. Can be set with RAILFLOW_TR_USER environment variable-u test-username
-p, --passwordYesTestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable-p XtpHXiPLEODyhF
-pr, --projectYesTestRail project name-pr "example project"
-path, --test-pathYesTestRail test cases path-path "Section1/subsection2/ShoppingCart
-f, --formatYesReport format: JUnit, JUnit-Steps, TestNg, TestNg-Steps, Cucumber, NUnit, NUnit-SpecFlow, Allure, Robot, TRX, xUnit, PyTest-Railflow, Playwright (case insensitive)-f junit
-r, --report-filesYesThe file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/surefire-reports/*.xml can be used.
E.g. use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports directory.
-r target/surefire-reports/*.xml target/failsafe-reports/*.xml
-sm, --search-modeYesSpecifies the test case lookup algorithm.
name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path
path: search for test case matching the name within the specified -path. If test case found, update the test case. If test case not found, create a new test case within specified -path
-sm path
-px, --proxyNoHTTP or SOCKS proxy configuration
E.g. socks://username:[email protected]:1080
-px socks://username:[email protected]:1080
-t, --timeoutNoUpload timeout (seconds)-t 10
-tr, --test-runNoTestRail test run name-tr "Chrome Regression Run"
-tp, --test-planNoTestRail test plan Name-tp "Shopping Cart Test Plan"
-mp, --milestone-pathNoTestRail milestone path-mp Milestone1/Milestone2
-cf, --case-fieldsNoTestRail test case custom fields.
The format is [Field label]=[value] pairs, separated with space.
E.g. "Case Field 1=value 1" "Case Field 2=value 2" ...
-cf "Case Field 1=value 1" "Case Field 2=value 2"
-rf, --result-fieldsNoTestRail test result custom fields.
The format is [Field label]=[value] pairs, separated with space.
E.g. "Result Field 1=value 1" "Result Field 2=value 2" ...
-rf "Result Field 1=value 1" "Result Field 2=value 2"
-a, --assignNoSmart Test Failure Assignment. Comma-separated list of TestRail users (email addresses). Railflow will assign failures based on a round robin algorithm.-a [email protected],[email protected]
-af, --assign-fileNoSmart Test Failure Assignment. File path containing list of TestRail users (email addresses).
Note: One user per line
-af /assignees.txt
-cn, --config-namesNoTestRail test plan configuration options.
Configuration format is: [config_group_name]/[config_name].
E.g. "Operating Systems/Linux" "Browsers/Chrome" ...
-cn "Operating Systems/Linux" "Browsers/Chrome"
-cr, --close-runNoIf Railflow should close the corresponding run after uploading test results-cr
-cp, --close-planNoIf Railflow should close the corresponding plan after uploading test results-cp
-dg, --disable-groupingNoIf Railflow should ignore report structure and just upload all tests into a folder which is set by test-path parameter-dg
-tn, --template-nameNoThe name of a template to use in TestRail. If it is not set, 'Test Case (Steps)' or the default one will be used-tn "Test Case (Steps)"
-cst, --case-typeNoThe name of a type for cases-cst other
-csp, --case-priorityNoThe name of a priority for cases-csp medium
-th, --thread-numberNoThe number of concurrent threads for exporting data. Default is 4-th 8
-um, --upload-modeNoUpload mode. Available values are:
0 (default) - create new test cases and do not overwrite existing ones;
1 - create new cases and overwrite existing ones;
2 - do not create new cases and overwrite existing ones;
3 - do not create new cases and do not overwrite existing ones
-um 1
-csf, --case-search-fieldNoThe name of the case field in TestRail which will be using for searching for existing test cases instead of test case title-csf "Custom field"
-ds, --disable-statsNoIf Railflow should disable collecting usage and error logs-ds
-fqtn, --fully-qualified-test-nameNoIf checked, Railflow will use fully qualified test names from the report files for test names in TestRail-fqtn
-us, --untested-statusNoThe name of the status to use in TestRail for untested/skipped tests-us Skipped
-ams, --attachment-max-sizeNoMaximum size of an attachment. Can be set in Gb, Mb, Kb or b. If the unit is not provided considered as Mb.
E.g. 1Gb, 2, 20Kb, 3Mb
-ams 1Gb
-atw, --attachment-type-whitelistNoThe comma-separated list of file extensions which are allowed to upload. E.g. json, .html, .xml-atw "json, .html, .xml"
-atb, --attachment-type-blacklistNoThe comma-separated list of file extensions which are not allowed to upload. E.g. json, .html, .xml-atb "json, .html, .xml"
-nr, --no-runNoDo not create or update Test Run in TestRail-nr
-tfn, --tags-field-nameNoThe name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags-tfn "Cucumber Tags"
-h, --helpNoShow the help information-h