Integrating TestNG with TestRail

TestNG (opens in a new tab) is a popular JAVA testing framework which is used by many teams for both unit and end-to-end testing using tools such as REST Assured, Selenium WebDriver, Playwright, etc.

Railflow provides a convenient JAVA annotations package for TestNG framework which enables users to integrate their TestNG project with TestRail in a variety of flexible ways. Railflow TestNG Annotations package provides the following capabilities

• Test Mapping: Map TestNG tests existing tests in TestRail
• Attach Screenshots: Attach screenshots and arbitrary attachments to the test case results in TestRail.
• Custom Fields: Support for TestRail custom fields.
• TestRail Entities: Update / Create Test Plans, Runs, Milestone, Tests, Results
• CI Application: Support for Jenkins, TeamCity, Gitlab, Github, etc.

Getting Started

Maven Setup

Add the Railflow TestNG dependency into pom.xml:

<dependency>
    <groupId>io.railflow.annotations</groupId>
    <artifactId>railflow-testng-annotations</artifactId>
    <version>0.5</version>
    <scope>test</scope>
</dependency>

Configure Surefire Plugin

Then, configure Maven Surefire plugin. This together with the Railflow dependency will generate an enriched report railflow_report.xml in the target/surefire-reports directory during mvn test. This file contains additional test metadata that will be consumed by Railflow CLI or Railflow CI plugins for uploading tests and test results to TestRail.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.1</version>
        <configuration>
            <properties>
                <property>
                    <name>listener</name>
                    <value>io.railflow.annotations.testng.RailflowReporterio.railflow.annotations.testng.RailflowReporter</value>
                </property>
            </properties>
        </configuration>
</plugin>

Railflow Annotations

Railflow annotations package allow flexible mapping of TestNG tests to TestRail, attach screenshots, and configure TestRail custom fields. The package provides a number of annotations that can be applied at the class or method level.

Before using the Railflow annotations, you should think about how you want to map your TestNG tests to TestRail. Railflow provides two mapping options TESTNG and TESTNG_STEPS which treat each test method as a single test case or as a test step within a test case respectively.

TestNG Mapping

The TESTNG option treats each test method as a single testcase. This means that when Railflow exports tests results to TestRail, each test method will be mapped to a separate test case in TestRail. The Railflow annoations pacakge io.railflow.annotations.Railflow has a number of attributes which can be applied at the class or method level.

Class Level

When TestNG classes use Railflow Class level attributes, the class level attributes are applied to the test results of methods inside the class.

Code Example: Using the attributes above, the class level attributes will be passed to each of the test methods in the class.

package io.railflow.annotations.demo;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
 
import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
 
import io.railflow.annotations.testng.CurrentTest;
import io.railflow.annotations.testng.RailflowReporter;
 
@Listeners(value = { RailflowReporter.class })
//class level annotation
@Railflow(title = "Railflow on the class level", jiraIds = { "ISSUE-42", "ISSUE-43" }, caseType = "Automated", casePriority = "Critical",
		caseFields = {@CustomField(name = "required text field", value = "class value"), @CustomField(name = "estimate", value = "42s") },
		resultFields = {@CustomField(name = "Custom field", value = "result from class annotations")},
		smartFailureAssignment = { "[email protected]", "[email protected]" })
public class RailflowDemo {
 
	@Test
	public void test() throws IOException {
		CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
	}
 
	@Test
	public void test_failed() throws IOException {
		CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
		Assert.fail("oops");
	}
}

How Test Results Appear in TestRail

Method Level

Sometimes you do not want to apply the same attributes to all the test methods in a class. In such cases, you can use the Railflow annotation at the method level. Method level can be used to override the class level attributes and can be used together.

Code Example: Using the attributes above, the method level attributes can be applied to any of the tests methods.

package io.railflow.annotations;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
 
import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
 
import io.railflow.annotations.testng.CurrentTest;
import io.railflow.annotations.testng.RailflowReporter;
 
@Listeners(value = { RailflowReporter.class })
public class RailflowDemo {
 
    //method level annotation
    @Railflow(title = "Railflow on the method level", jiraIds = {"ISSUE-44", "ISSUE-45"}, caseType = "Performance", casePriority = "High",
            caseFields = { @CustomField(name = "required text field", value = "method value"), @CustomField(name = "estimate", value = "24s") },
            resultFields = { @CustomField(name = "Custom field", value = "result from annotation on method")}, smartFailureAssignment = { "[email protected]"})
    @Test
    public void test() throws IOException {
        CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
    }
 
    @Test
    public void test_failed() throws IOException {
        CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
        Assert.fail("oops");
    }
}

How Test Results Appear in TestRail

TESTNG_STEPS Mapping

The TESTNG_STEPS option treats each test class as a single testcase. Methods inside the test class are treated as test steps within TestRail. This means that when results are exported to TestRail, each test class will be a separate test case in TestRail and each method inside the test class will be a test step in TestRail

Class Level

When TestNG classes use Railflow Class level attributes, the class level attributes are applied to the test results of methods inside the class.

Example: Using the attributes above, these class level attributes will be passed to each of the test methods in the class.

package io.railflow.annotations.demo;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
 
import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
 
import io.railflow.annotations.testng.CurrentTest;
import io.railflow.annotations.testng.RailflowReporter;
 
@Listeners(value = { RailflowReporter.class })
@Railflow(title = "Railflow on the class level", jiraIds = { "ISSUE-42", "ISSUE-43" }, caseType = "Automated", casePriority = "Critical",
		caseFields = {@CustomField(name = "required text field", value = "class value"), @CustomField(name = "estimate", value = "42s") },
		resultFields = {@CustomField(name = "Custom field", value = "result from class annotations")},
		smartFailureAssignment = { "[email protected]", "[email protected]" })
public class RailflowDemo {
 
	@Test
	public void test() throws IOException {
		CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
	}
 
	@Test
	public void test_failed() throws IOException {
		CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
		Assert.fail("oops");
	}
}

How Test Results Appear in TestRail

Method Level

Screenshots and attachments

When using TestNG with Selenium WebDriver, it can be very useful to take screenshots of the browser window when a test fails. Railflow provides utility classes and several static methods for taking automatic screenshot so that they can also be uploaded to TestRail along with the failure exception message.

• addAttachment(File attachmentFile) : Adds arbitrary attachment file to the XML report.
• addAttachment(final String name, final InputStream inputStream) : Adds an attachment file with given name and contents of inputStream
• addAttachment(final String name, final byte[] data): Adds an attachment file with given name and content specified by the given byte array

Railflow CLI / CI Plugins

The resulting railflow_report.xml report file can be processed and the tests and test results uploaded to TestRail using Railflow CLI or CI plugins.

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/railflow_report.xml

Jenkins plugin

TeamCity plugin

CLI Reference