Integrating Jenkins with TestRail
Jenkins is one of the most popular CI applications and is used by thousands of companies to define annd run development and QA pipelines across a variety of testing frameworks and tools like JUnit, Pytest, Selenium, Playwright, etc. While Jenkins is great at running generic tasks, it is not designed to track, triage, or report on test results across a large number of Jenkins jobs and servers.
Railflow Jenkins Plugin and Command Line (CLI) has been designed to seamlessly export a variety of test results reports from Jenkins Builds to TestRail. Once integrated, TestRail users can use TestRail's powerful dashboard and reporting capabilities, Test Results workflows, and Test Cases Inventory to triage test results that would otherwise be locked up in Jenkins.
Railflow for Jenkins provides the following capabilities for TestRail users
Railflow Jenkins Plugin
for plugin based rich UX controlsRailflow CLI
for a non-plugin based dockerized or NPM experienceSmart Failure Assignment
: Ability to auto assign test failure to team membersBDD Reports
: Support for Cucumber, Serenity, and other popular BDD frameworksFramework Reports
: Support for TestNG, JUnit, Pytest, Allure, MSTest, Robot, and others.
Railflow can be setup with Jenkins in three ways.
1. Jenkins Plugin
Jenkins version 2.204.1 or newer
Railflow is a commercial product. As such, it is not available in the Jenkins plugin directory. Railflow Jenkins plugin (hpi file) can be downloaded from our website's download section (opens in a new tab).
Jenkins plugins are the standard way of adding new capabilities to a Jenkins instance. Railflow Jenkins plugin exposes a new Post Builds step for easily integrating Jenkins with TestRail.
Install
Download Railflow Jenkins Plugin (opens in a new tab)
Upload the plugin via Manage Jenkins
→ Manage Plugins
→ Advanced
→ Upload Plugin
Upload and restart Jenkins for the plugin to take effect.
Configure
To configure the plugin and license, navigate to Manage Jenkins
→ Configure System
page of the Jenkins UI and scroll down to the Railflow Global Settings
section:
License Key (online activation): Select License Key
option and copy/paste the license key from your welcome email and click Activate License
License File (offline activation): If your Jenkins instance does not have outbound internet access, you may use this option to upload the license activation from your email. The license file has a .skm extension and part of your welcome email.
Railflow plugin configuration section allows you to define one or more TestRail servers.
Freestyle Job
If you are using Jenkins UI based (freestyle job) jobs to run some test frameworks/tools, Railflow plugin's rich controls offer a variety of ways to to integrate with TestRail.
Add and configure the Jenkins job's post-build action: Railflow Plugin: TestRail Test Results Processor
. Railflow's post-build action allows you to specify the test framework, test results location, and various other TestRail configurations.
You can add multiple Raillflow post-build actions using Add More
button. This may be needed if you have multiple reports to process and want to map each report to different places in TestRail.
For additional TestRail export functionality, check out the Advanced
section.
Jenkinsfile Job
Railflow plugin can also be configured using a Jenkinsfile. The table below describes all the available options within a Jenkinsfile
Key | Required | Description |
---|---|---|
testRailServerName | Yes | One of the server names configured in "Global settings configuration" |
testRailProjectName | Yes | The name of a project in TestRail to which results should be exported |
failIfUploadFailed | No | If true, the build will be marked as failed if for any reason the plugin could not upload the results. This could be due to Railflow issues, TestRail server issues, network issues, etc. |
overriddenUserName | No | If specified, it overrides TestRail user name defined in Global Configuration |
overriddenPassword | No | If specified, it overrides TestRail password defined in Global Configuration |
jobConfigurations | Yes | This can contain job configuration parameters defined below |
resultFilePattern | Yes | The file path to the test report file(s) generated during the build. Ant-style patterns such as **/surefire-reports/*.xml can be used.Example: Use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports directory. |
searchMode | Yes | Specifies the test case lookup algorithm. Name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified Test Case Path Path : search for test case matching the name within the specified Test Case Path. If test case found, update the test case. If test case not found, create a new test case within specified Test Case Path |
testCasePath | No | Path to where Railflow should upload test case definitions, must contain suite name in the beginning (for single-suite project, suite name is always 'Master'), e.g. Master/Section1/Section2 |
testPlanName | No | Name of a test plan in TestRail to which test results will be added |
testRunName | No | Name of a test run in TestRail to which test results will be added |
milestonePath | No | Path to a milestone in TestRail to which test run/plan will be added. E.g. Milestone1/Milestone2 |
smartTestFailureAssignment | No | A comma separated list of user emails for smart failure assignment. Each failed result will be assigned to a person from this list in a round robin fashion |
testCaseType | No | Name of a case type in TestRail, e.g. Automated |
testCasePriority | No | Name of a case priority in TestRail, e.g. High |
testCaseTemplate | No | Name of a TestRail template. If it is blank, Test Case (Steps) will be used |
testCaseCustomFields | No | Values for case fields in TestRail can be specified in this field. The format is [TestRail field label]=[value] and each field name\value pair should start with the new line. E.g.: Custom Field One=foo Custom Field Two=bar |
testResultCustomFields | No | Values for result fields in TestRail can be specified in this field. The format is [TestRail field label]=[value] and each field name\value pair should start with the new line. E.g.: Custom Result Field One=foo Custom Result Field Two=bar |
configurationNames | No | A list of configuration names in TestRail. The format is [Config Group Name]/[Config Name]. Each entry must start with the new line. E.g.: Operating Systems/Linux Browsers/Chrome |
caseSearchField | No | The name of the case field in TestRail which will be using for searching for existing test cases instead of test case title |
tagsFieldName | No | The name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags |
uploadMode | No | Test case upload mode:
|
disableGrouping | No | If true, Railflow will ignore structure in report files and upload all test cases into one Section, defined by the Test Path parameter. |
closeRun | No | If true, Railflow will close the test run in TestRail and archive its tests and results |
closePlan | No | If true, Railflow will close the test plan in TestRail and archive its tests and results |
fullCaseNamesAllowed | No | If true, Railflow will use fully qualified test names from the report files for test names in TestRail |
pipeline {
agent {
label "linux"
}
stages {
stage('Build') {
steps {
echo '********BUILD STEP**********'
git branch: 'master', credentialsId: '*****', url: 'https://...junit-demo.git'
sh 'mvn test'
}
}
}
post {
always {
echo '********UPLOAD RESULT INTO TESTRAIL**********'
railflow(
testRailServerName: 'TestRail',
testRailProjectName: 'Railflow Demo',
failIfUploadFailed: true,
overriddenUserName: 'userName',
overriddenPassword: 'password',
jobConfigurations: [[configurationNames: '''OS/Linux
Browser/Chrome''',
milestonePath: 'M1/M2',
resultFilePattern: '**/surefire-reports/*.xml',
uploadMode: 'CREATE_NO_UPDATE',
searchMode: 'Path',
testCaseCustomFields: '''Required text field=Hello from Jenkins
estimate=30s''',
testCasePath: 'Railflow/Website',
testCasePriority: 'High',
testCaseType: 'Automated',
testPlanName: 'Test plan',
testReportFormat: 'JUNIT',
testResultCustomFields: '''Custom field=Results from Jenkins
version=2.0''',
testRunName: '${JOB_NAME}-${BUILD_NUMBER}'
smartTestFailureAssignment: '[email protected], [email protected]',
testCaseTemplate: 'template1',
caseSearchField: 'case search field name',
tagsFieldName: 'tags field name',
uploadMode: 'CREATE_NO_UPDATE',
disableGrouping: true,
closeRun: true,
closePlan: true,
fullCaseNamesAllowed: true ]])
}
}
}
2. Command Line Interface
Jenkins version: 2.204.1 or newer
Node version: 20.x or newer
Jenkins plugins can be a pain to manage and can occassionally cause issues on the Jenkins server. In addition, any updates to plugins need to be carefully managed and planned with Jenkins admins. This is true of all Jenkins plugins.
Railflow Command Line (opens in a new tab) is a NPM based utility that can be run on any Jenkins setup (Linux, Windows, OSX, and Railflow CLI Docker Image (opens in a new tab)) without the need for any Jenkins plugins.
CLI Reference
Railflow CLI will automatically create tests, runs, plans, milestones, etc. if they do not exist.
Environment Variables:
To avoid exposing sensitive information like TestRail URL, License, username, and password in CLI arguments, users can set the following environment variables: RAILFLOW_LICENSE
, RAILFLOW_TR_URL
, RAILFLOW_TR_USER
, RAILFLOW_TR_PASSWORD
. Railflow CLI will always check these environment variables at run time.
Use double quotes for argument values with spaces. Example: --project "demo project"
Key | Required | Description | Example |
---|---|---|---|
-v, --version | No | Outputs Railflow version number | -v |
-k, --key | -k or -l | (online activation) License key. Can be set with RAILFLOW_LICENSE environment variable | -k XXXXX-XXXXX-XXXXX-XXXXX |
-l, --license-file | -k or -l | (offline activation) File path or remote url license file | -l /files/ActivationFile.skm |
-url, --url | Yes | TestRail instance URL. Can be set with RAILFLOW_TR_URL environment variable | -url https://example.testrail.io (opens in a new tab) |
-u, --username | Yes | TestRail username. Can be set with RAILFLOW_TR_USER environment variable | -u test-username |
-p, --password | Yes | TestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable | -p XtpHXiPLEODyhF |
-pr, --project | Yes | TestRail project name | -pr "example project" |
-path, --test-path | Yes | TestRail test cases path | -path "Section1/subsection2/ShoppingCart |
-f, --format | Yes | Report format: JUnit, JUnit-Steps, TestNg, TestNg-Steps, Cucumber, NUnit, NUnit-SpecFlow, Allure, Robot, TRX, xUnit, PyTest-Railflow, Playwright (case insensitive) | -f junit |
-r, --report-files | Yes | The file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/surefire-reports/*.xml can be used. E.g. use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports directory. | -r target/surefire-reports/*.xml target/failsafe-reports/*.xml |
-sm, --search-mode | Yes | Specifies the test case lookup algorithm. name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path path: search for test case matching the name within the specified -path . If test case found, update the test case. If test case not found, create a new test case within specified -path | -sm path |
-px, --proxy | No | HTTP or SOCKS proxy configuration E.g. socks://username:[email protected]:1080 | -px socks://username:[email protected]:1080 |
-t, --timeout | No | Upload timeout (seconds) | -t 10 |
-tr, --test-run | No | TestRail test run name | -tr "Chrome Regression Run" |
-tp, --test-plan | No | TestRail test plan Name | -tp "Shopping Cart Test Plan" |
-mp, --milestone-path | No | TestRail milestone path | -mp Milestone1/Milestone2 |
-cf, --case-fields | No | TestRail test case custom fields. The format is [Field label]=[value] pairs, separated with space. E.g. "Case Field 1=value 1" "Case Field 2=value 2" ... | -cf "Case Field 1=value 1" "Case Field 2=value 2" |
-rf, --result-fields | No | TestRail test result custom fields. The format is [Field label]=[value] pairs, separated with space. E.g. "Result Field 1=value 1" "Result Field 2=value 2" ... | -rf "Result Field 1=value 1" "Result Field 2=value 2" |
-a, --assign | No | Smart Test Failure Assignment. Comma-separated list of TestRail users (email addresses). Railflow will assign failures based on a round robin algorithm. | -a [email protected],[email protected] |
-af, --assign-file | No | Smart Test Failure Assignment. File path containing list of TestRail users (email addresses). Note: One user per line | -af /assignees.txt |
-cn, --config-names | No | TestRail test plan configuration options. Configuration format is: [config_group_name]/[config_name]. E.g. "Operating Systems/Linux" "Browsers/Chrome" ... | -cn "Operating Systems/Linux" "Browsers/Chrome" |
-cr, --close-run | No | If Railflow should close the corresponding run after uploading test results | -cr |
-cp, --close-plan | No | If Railflow should close the corresponding plan after uploading test results | -cp |
-dg, --disable-grouping | No | If Railflow should ignore report structure and just upload all tests into a folder which is set by test-path parameter | -dg |
-tn, --template-name | No | The name of a template to use in TestRail. If it is not set, 'Test Case (Steps)' or the default one will be used | -tn "Test Case (Steps)" |
-cst, --case-type | No | The name of a type for cases | -cst other |
-csp, --case-priority | No | The name of a priority for cases | -csp medium |
-th, --thread-number | No | The number of concurrent threads for exporting data. Default is 4 | -th 8 |
-um, --upload-mode | No | Upload mode. Available values are: 0 (default) - create new test cases and do not overwrite existing ones; 1 - create new cases and overwrite existing ones; 2 - do not create new cases and overwrite existing ones; 3 - do not create new cases and do not overwrite existing ones | -um 1 |
-csf, --case-search-field | No | The name of the case field in TestRail which will be using for searching for existing test cases instead of test case title | -csf "Custom field" |
-ds, --disable-stats | No | If Railflow should disable collecting usage and error logs | -ds |
-fqtn, --fully-qualified-test-name | No | If checked, Railflow will use fully qualified test names from the report files for test names in TestRail | -fqtn |
-us, --untested-status | No | The name of the status to use in TestRail for untested/skipped tests | -us Skipped |
-ams, --attachment-max-size | No | Maximum size of an attachment. Can be set in Gb, Mb, Kb or b. If the unit is not provided considered as Mb. E.g. 1Gb, 2, 20Kb, 3Mb | -ams 1Gb |
-atw, --attachment-type-whitelist | No | The comma-separated list of file extensions which are allowed to upload. E.g. json, .html, .xml | -atw "json, .html, .xml" |
-atb, --attachment-type-blacklist | No | The comma-separated list of file extensions which are not allowed to upload. E.g. json, .html, .xml | -atb "json, .html, .xml" |
-nr, --no-run | No | Do not create or update Test Run in TestRail | -nr |
-tfn, --tags-field-name | No | The name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags | -tfn "Cucumber Tags" |
-h, --help | No | Show the help information | -h |
npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]
npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt
npx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt
Jenkinsfile Example
pipeline {
agent any
environment {
RAILFLOW_KEY = credentials('railflow-key')
TESTRAIL_CREDS = credentials('testrail-creds')
}
tools {
jdk 'default'
maven 'default'
nodejs 'default'
}
stages {
stage('Checkout'){
steps{
checkout([$class: 'GitSCM', branches: [[name: '*/development']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'git-creds', url: 'https://gitrepo.com/workspace/project.git']]])
}
}
stage('Run unit tests') {
steps {
sh 'mvn clean test'
}
}
}
post{
always{
echo "Begin exporting to TestRail"
sh "npx railflow -k ${RAILFLOW_KEY} -url https://testrail_server/ -u ${TESTRAIL_CREDS_USR} -p ${TESTRAIL_CREDS_PSW} -pr \\\"Railflow Demo\\\" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tp TestPlan12345 -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cf \\\"Required text field=123\\\""
}
}
}
3. Docker
There are a variety of ways to use Docker within Jenkins. For this example, we will be using the Jenkins Docker Plugin (opens in a new tab) with Railflow CLI Docker Image (opens in a new tab).
In this example below, we are building and publishing the test results from a TestNG project.
pipeline {
agent any
environment {
RAILFLOW_KEY = credentials('railflow-key') // see https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#handling-credentials
TESTRAIL_CREDS = credentials('testrail-credentials') // see https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#usernames-and-passwords
TESTRAIL_URL = "https://testrail_server/"
}
tools {
jdk 'default'
maven 'default'
}
stages {
stage('Checkout') {
script {
steps {
git branch: 'master', url: 'https://github.com/railflow/testng_example.git'
}
}
}
stage('Run unit tests') {
steps {
script {
sh 'mvn clean test'
}
}
}
}
post {
always {
sh 'docker pull railflow/railflow:latest'
withDockerContainer('railflow/railflow:latest') {
sh 'railflow -k ${RAILFLOW_KEY} -url ${TESTRAIL_URL} -u ${TESTRAIL_CREDS_USR} -p ${TESTRAIL_CREDS_PSW} -pr \"Railflow Demo\" -path Demo/Pizza -f testng -r "**/surefire-reports/testng-results.xml" -sm path -tp "Test Plan" -cf \"Required text field=something\"'
}
}
}
}
Railflow Output
Railflow Jenkins Plugin and CLI both generate rich log output for viewing TestRail integration activity.
Jenkins Build Logs
Automatic Test Creation
Automatic Test Plan and Runs
Test Results Details
Automatic Milestones
Smart Failure Assignment
Smart Failure assignment is a very powerful feature of Railflow and allows teams to efficiently and strategically assign test failures to specified team members. Doing this automatically as part of the CI process means that teams don't waste valuable time during the test triage process.
To use Smart Failure Assignment feature, the users need to have Global Role
under Project Access
.
Example
Consider a Jenkins Selenium Webdriver job build is failing with 5 test failures, and 2 users configured in the Smart Test Failure Assignment
field.
Jenkins Build Logs
In this example above, Railflow's Smart Assignment feature filtered all the test failures and distributed them equally across the specified users. This nifty feature greatly eliminates the manual test triage process and saves teams valuable time.