Integrating JUNIT with TestRail

Pytest (opens in a new tab) is a popular Python testing framework which is used by many teams for both unit and end-to-end testing using tools such as REST Assured, Selenium WebDriver, Playwright, etc.

Railflow provides a convenient Pytest plugin which enables users to integrate their Pytest project with TestRail in a variety of flexible ways. Railflow Pytest plugin provides the following capabilities

• Test Mapping: Map Pytest tests existing tests in TestRail
• Attach Screenshots: Attach screenshots and arbitrary attachments to the test case results in TestRail.
• Custom Fields: Support for TestRail custom fields.
• TestRail Entities: Update / Create Test Plans, Runs, Milestone, Tests, Results
• CI Application: Support for Jenkins, TeamCity, Gitlab, Github, etc.

Getting Started

Railflow pytest plugin (opens in a new tab) is available on PyPI and can be easily installed using pip and can also be added to the project's requirements.txt file.

pip install pytest-railflow-testrail-reporter

Railflow Markers

The Railflow Pytest plugins provides a number of markers at the class and function level. Class level markers are inherited by all the test functions in the class. Function level markers can be used to override the class level markers. This behavior provides a great deal of flexibility in mapping Pytest tests to TestRail test cases.

Screenshot and Attachments

Adding Attachments The testrail_add_screenshot fixture can be used to add arbitrary attachments into the test report:

def test_add(testrail_add_screenshot):
    a = 3
    b = 2
    c = a + b
    assert c == 5
    testrail_add_screenshot("image.png")

Automatic Screenshot for Selenium Tests Railflow plugin supports Pytest Splinter (opens in a new tab) and if your selenium/appium tests use pytest-splinter plugin, then Railflow plugin automatically takes screenshots of failed tests and upload them to TestRail (yes, really)

import pytest
 
@pytest.mark.railflow(
        testrail_ids=["C134"],
        case_type="Automated"
)
def test_google(browser): 
	browser.visit("https://www.google.com")
	browser.fill('q', 'splinter - python acceptance testing for web applications')
	# Find and click the 'search' button
	button = browser.find_by_name('btnK')
	# Interact with elements
	button.click()
	assert browser.is_text_present('splinter.cobrateam.info'), "splinter.cobrateam.info wasn't found."

Screenshots and attachments

When using Selenium WebDriver, it is common to take screenshots of the browser window when a test fails. Railflow provides utility classes and several static methods for taking automatic screenshot so that they can also be uploaded to TestRail along with the failure exception message

• addAttachment(File attachmentFile) : Adds arbitrary attachment file to the XML report.
• addAttachment(final String name, final InputStream inputStream) : Adds an attachment file with given name and contents of inputStream
• addAttachment(final String name, final byte[] data): Adds an attachment file with given name and content specified by the given byte array

Class Level Annotations

Annotation package provides io.railflow.annotations.Railflow annotation which can be applied to test class or test method.

The io.railflow.annotations.Railflow annotation has a number of attributes which have slightly different meaning based on class level or method level applicaiton.

Class Level Annotations

Configure Surefire Plugin (JUnit 4 only)

Configuring the surefire plugin and adding the Railflow dependency will enable the junit project to generate an enriched junit schema compliant report during mvn test. This report contains all the test metadata added via annotation and will be consumed by Railflow CI plugins or CLI.

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.22.2</version>
	<dependencies>
		<dependency>
			<groupId>org.apache.maven.surefire</groupId>
			<artifactId>surefire-junit47</artifactId>
			<version>2.22.2</version>
		</dependency>
	</dependencies>
	<configuration>
		<properties>
			<property>
				<name>listener</name>
				<value>io.railflow.annotations.junit.RailflowJunit4Listener</value>
			</property>
		</properties>
	</configuration>
</plugin>

Register Railflow listener (JUnit 4 only)

Add io.railflow.annotations.junit.RailflowJunit4Listener Junit4 listener in JUnitCore object in your test runner:

import org.junit.runner.JUnitCore;
import io.railflow.annotations.junit.RailflowJunit4Listener;
 
public class MyTestRunner {
  public static void main(String args[]) {
    JUnitCore runner = new JUnitCore();
    runner.addListener(new RailflowJunit4Listener());
    runner.run(YourTestClass.class);
  }
}

mvn clean test -Dio.railflow.report_dir=target\surefire-reports

Railflow Annotations

The Java Annotation package provides io.railflow.annotations.Railflow annotation which can be applied to test class or test method.

The io.railflow.annotations.Railflow annotation has a number of attributes which have slightly different meaning based on class level or method level applicaiton.

Class Level Annotations

Method Level Annotations

Screenshots and attachments

When using Selenium WebDriver, it is common to take screenshots of the browser window when a test fails. Railflow provides utility classes and several static methods for taking automatic screenshot so that they can also be uploaded to TestRail along with the failure exception message

• addAttachment(File attachmentFile) : Adds arbitrary attachment file to the XML report.
• addAttachment(final String name, final InputStream inputStream) : Adds an attachment file with given name and contents of inputStream
• addAttachment(final String name, final byte[] data): Adds an attachment file with given name and content specified by the given byte array

Railflow CLI / CI Plugins

The resulting railflow_report.xml report file can be processed and the tests and test results uploaded to TestRail using Railflow CLI or CI plugins.

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/railflow_report.xml

Jenkins plugin

TeamCity plugin

Annotations Examples

Class Level

package io.railflow.annotations.demo;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
 
import org.junit.Assert;
import org.junit.Test;
 
import io.railflow.annotations.CustomField;
import io.railflow.annotations.Railflow;
import io.railflow.annotations.junit.CurrentTest;
 
@Railflow(title = "Railflow on the class level", jiraIds = {"ISSUE-42", "ISSUE-43"}, caseType = "Automated", casePriority = "Critical",
        caseFields = {@CustomField(name = "required text field", value = "class value"), @CustomField(name = "estimate", value = "42s")},
		resultFields = {@CustomField(name = "Custom field", value = "result from class annotations")},
		smartFailureAssignment = { "[email protected]", "[email protected]" })
public class RailflowDemoTest {
 
	@Test
	public void test() throws IOException {
		CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
	}
 
	@Test
	public void test_failed() throws IOException {
		CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
		Assert.fail("oops");
	}
}
 

Results in TestRail after uploading generated railflow_report.xml

Method Level

package io.railflow.annotations;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
 
import org.junit.Assert;
import org.junit.Test;
 
import io.railflow.annotations.CustomField;
import io.railflow.annotations.junit.CurrentTest;
 
public class RailflowDemoTest {
 
    @Railflow(title = "Railflow on the method level", jiraIds = {"ISSUE-44", "ISSUE-45"}, caseType = "Performance", casePriority = "High",
            caseFields = {@CustomField(name = "required text field", value = "method value"), @CustomField(name = "estimate", value = "24s")},
            resultFields = {@CustomField(name = "Custom field", value = "result from annotation on method")}, smartFailureAssignment = {"[email protected]"})
    @Test
    public void test() throws IOException {
        CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
    }
 
    @Test
    public void test_failed() throws IOException {
        CurrentTest.addAttachment("my log.txt", "something".getBytes(StandardCharsets.UTF_8));
        Assert.fail("oops");
    }
}

Results in TestRail after uploading generated railflow_report.xml

CLI Reference