Docs
CICD
Jenkins

Integrating Jenkins with TestRail

Jenkins is one of the most popular CI applications and is used by thousands of companies to define annd run development and QA pipelines across a variety of testing frameworks and tools like JUnit, Pytest, Selenium, Playwright, etc. While Jenkins is great at running generic tasks, it is not designed to track, triage, or report on test results across a large number of Jenkins jobs and servers.

Railflow Jenkins Plugin and Command Line (CLI) has been designed to seamlessly export a variety of test results reports from Jenkins Builds to TestRail. Once integrated, TestRail users can use TestRail's powerful dashboard and reporting capabilities, Test Results workflows, and Test Cases Inventory to triage test results that would otherwise be locked up in Jenkins.

Railflow for Jenkins provides the following capabilities for TestRail users

  • Railflow Jenkins Plugin for plugin based rich UX controls
  • Railflow CLI for a non-plugin based dockerized or NPM experience
  • Smart Failure Assignment: Ability to auto assign test failure to team members
  • BDD Reports: Support for Cucumber, Serenity, and other popular BDD frameworks
  • Framework Reports: Support for TestNG, JUnit, Pytest, Allure, MSTest, Robot, and others.

Railflow can be setup with Jenkins in three ways.

1. Jenkins Plugin

Jenkins version 2.204.1 or newer

Railflow is a commercial product. As such, it is not available in the Jenkins plugin directory. Railflow Jenkins plugin (hpi file) can be downloaded from our website's download section (opens in a new tab).

Jenkins plugins are the standard way of adding new capabilities to a Jenkins instance. Railflow Jenkins plugin exposes a new Post Builds step for easily integrating Jenkins with TestRail.

Install

Download Railflow Jenkins Plugin (opens in a new tab)

Upload the plugin via Manage JenkinsManage PluginsAdvancedUpload Plugin

Jenkins plugin installation

Upload and restart Jenkins for the plugin to take effect.

Configure

To configure the plugin and license, navigate to Manage JenkinsConfigure System page of the Jenkins UI and scroll down to the Railflow Global Settings section:

Jenkins plugin config

License Key (online activation): Select License Key option and copy/paste the license key from your welcome email and click Activate License

Jenkins plugin online activation

License File (offline activation): If your Jenkins instance does not have outbound internet access, you may use this option to upload the license activation from your email. The license file has a .skm extension and part of your welcome email.

Jenkins plugin offline activation

Railflow plugin configuration section allows you to define one or more TestRail servers.

Jenkins plugin config

Freestyle Job

If you are using Jenkins UI based (freestyle job) jobs to run some test frameworks/tools, Railflow plugin's rich controls offer a variety of ways to to integrate with TestRail.

Add and configure the Jenkins job's post-build action: Railflow Plugin: TestRail Test Results Processor. Railflow's post-build action allows you to specify the test framework, test results location, and various other TestRail configurations.

You can add multiple Raillflow post-build actions using Add More button. This may be needed if you have multiple reports to process and want to map each report to different places in TestRail.

Jenkins plugin

For additional TestRail export functionality, check out the Advanced section.

Jenkins plugin

Jenkinsfile Job

Railflow plugin can also be configured using a Jenkinsfile. The table below describes all the available options within a Jenkinsfile

KeyRequiredDescription
testRailServerNameYesOne of the server names configured in "Global settings configuration"
testRailProjectNameYesThe name of a project in TestRail to which results should be exported
failIfUploadFailedNoIf true, the build will be marked as failed if for any reason the plugin could not upload the results. This could be due to Railflow issues, TestRail server issues, network issues, etc.
overriddenUserNameNoIf specified, it overrides TestRail user name defined in Global Configuration
overriddenPasswordNoIf specified, it overrides TestRail password defined in Global Configuration
jobConfigurationsYesThis can contain job configuration parameters defined below
resultFilePatternYesThe file path to the test report file(s) generated during the build. Ant-style patterns such as **/surefire-reports/*.xml can be used.

Example: Use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports directory.
searchModeYesSpecifies the test case lookup algorithm.
Name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified Test Case Path

Path: search for test case matching the name within the specified Test Case Path. If test case found, update the test case. If test case not found, create a new test case within specified Test Case Path
testCasePathNoPath to where Railflow should upload test case definitions, must contain suite name in the beginning (for single-suite project, suite name is always 'Master'), e.g. Master/Section1/Section2
testPlanNameNoName of a test plan in TestRail to which test results will be added
testRunNameNoName of a test run in TestRail to which test results will be added
milestonePathNoPath to a milestone in TestRail to which test run/plan will be added.
E.g. Milestone1/Milestone2
smartTestFailureAssignmentNoA comma separated list of user emails for smart failure assignment. Each failed result will be assigned to a person from this list in a round robin fashion
testCaseTypeNoName of a case type in TestRail, e.g. Automated
testCasePriorityNoName of a case priority in TestRail, e.g. High
testCaseTemplateNoName of a TestRail template. If it is blank, Test Case (Steps) will be used
testCaseCustomFieldsNoValues for case fields in TestRail can be specified in this field. The format is [TestRail field label]=[value] and each field name\value pair should start with the new line.
E.g.:
Custom Field One=foo
Custom Field Two=bar
testResultCustomFieldsNoValues for result fields in TestRail can be specified in this field. The format is [TestRail field label]=[value] and each field name\value pair should start with the new line.
E.g.:
Custom Result Field One=foo
Custom Result Field Two=bar
configurationNamesNoA list of configuration names in TestRail. The format is [Config Group Name]/[Config Name]. Each entry must start with the new line.
E.g.:
Operating Systems/Linux
Browsers/Chrome
caseSearchFieldNoThe name of the case field in TestRail which will be using for searching for existing test cases instead of test case title
tagsFieldNameNoThe name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags
uploadModeNo
Test case upload mode:
  • Create new test cases and do not overwrite existing ones:
    If test case not found, create a new test case within specified Test Case Path.
    If test case found, do not update the test case.
    Value for pipeline: CREATE_NO_UPDATE
  • Create new cases and overwrite existing ones:
    If test case not found, create a new test case within specified Test Case Path.
    If test case found, update the test case.
    Value for pipeline: CREATE_UPDATE
  • Do not create new cases and overwrite existing ones:
    If test case not found, do not create a new test case and the corresponding test result will not be uploaded into TestRail.
    If test case found, update the test case.
    Value for pipeline: NO_CREATE_UPDATE
  • Do not create new cases and do not overwrite existing ones:
    If test case not found, do not create a new test case and the corresponding test result will not be uploaded into TestRail.
    If test case found, do not update the test case.
    Value for pipeline: NO_CREATE_NO_UPDATE
disableGroupingNoIf true, Railflow will ignore structure in report files and upload all test cases into one Section, defined by the Test Path parameter.
closeRunNoIf true, Railflow will close the test run in TestRail and archive its tests and results
closePlanNoIf true, Railflow will close the test plan in TestRail and archive its tests and results
fullCaseNamesAllowedNoIf true, Railflow will use fully qualified test names from the report files for test names in TestRail
Exammple Jenkins Pipeline Example running JUnit test project
pipeline {
    agent {
        label "linux"
    }
    stages {
       stage('Build') {
          steps {
             echo '********BUILD STEP**********'
             git branch: 'master', credentialsId: '*****', url: 'https://...junit-demo.git'
             sh 'mvn test'
          }
       }
    }
    post {
     always {
     echo '********UPLOAD RESULT INTO TESTRAIL**********'
 
    railflow(
        testRailServerName: 'TestRail',
        testRailProjectName: 'Railflow Demo',
        failIfUploadFailed: true, 
        overriddenUserName: 'userName', 
        overriddenPassword: 'password',
        jobConfigurations: [[configurationNames: '''OS/Linux
            Browser/Chrome''', 
        milestonePath: 'M1/M2', 
        resultFilePattern: '**/surefire-reports/*.xml',
        uploadMode: 'CREATE_NO_UPDATE',
        searchMode: 'Path',
        testCaseCustomFields: '''Required text field=Hello from Jenkins
            estimate=30s''', 
        testCasePath: 'Railflow/Website', 
        testCasePriority: 'High', 
        testCaseType: 'Automated', 
        testPlanName: 'Test plan', 
        testReportFormat: 'JUNIT', 
        testResultCustomFields: '''Custom field=Results from Jenkins
            version=2.0''', 
        testRunName: '${JOB_NAME}-${BUILD_NUMBER}'
        smartTestFailureAssignment: '[email protected], [email protected]', 
        testCaseTemplate: 'template1', 
        caseSearchField: 'case search field name', 
        tagsFieldName: 'tags field name',
        uploadMode: 'CREATE_NO_UPDATE', 
        disableGrouping: true, 
        closeRun: true, 
        closePlan: true, 
        fullCaseNamesAllowed: true ]])
      }
    }
}

2. Command Line Interface

Jenkins version: 2.204.1 or newer

Node version: 20.x or newer

Jenkins plugins can be a pain to manage and can occassionally cause issues on the Jenkins server. In addition, any updates to plugins need to be carefully managed and planned with Jenkins admins. This is true of all Jenkins plugins.

Railflow Command Line (opens in a new tab) is a NPM based utility that can be run on any Jenkins setup (Linux, Windows, OSX, and Railflow CLI Docker Image (opens in a new tab)) without the need for any Jenkins plugins.

CLI Reference

Railflow CLI will automatically create tests, runs, plans, milestones, etc. if they do not exist.

Environment Variables: To avoid exposing sensitive information like TestRail URL, License, username, and password in CLI arguments, users can set the following environment variables: RAILFLOW_LICENSE, RAILFLOW_TR_URL, RAILFLOW_TR_USER, RAILFLOW_TR_PASSWORD. Railflow CLI will always check these environment variables at run time.

⚠️

Use double quotes for argument values with spaces. Example: --project "demo project"

KeyRequiredDescriptionExample
-v, --versionNoOutputs Railflow version number-v
-k, --key-k or -l(online activation) License key. Can be set with RAILFLOW_LICENSE environment variable-k XXXXX-XXXXX-XXXXX-XXXXX
-l, --license-file-k or -l(offline activation) File path or remote url license file-l /files/ActivationFile.skm
-url, --urlYesTestRail instance URL. Can be set with RAILFLOW_TR_URL environment variable-url https://example.testrail.io (opens in a new tab)
-u, --usernameYesTestRail username. Can be set with RAILFLOW_TR_USER environment variable-u test-username
-p, --passwordYesTestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable-p XtpHXiPLEODyhF
-pr, --projectYesTestRail project name-pr "example project"
-path, --test-pathYesTestRail test cases path-path "Section1/subsection2/ShoppingCart
-f, --formatYesReport format: JUnit, JUnit-Steps, TestNg, TestNg-Steps, Cucumber, NUnit, NUnit-SpecFlow, Allure, Robot, TRX, xUnit, PyTest-Railflow, Playwright (case insensitive)-f junit
-r, --report-filesYesThe file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/surefire-reports/*.xml can be used.
E.g. use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports directory.
-r target/surefire-reports/*.xml target/failsafe-reports/*.xml
-sm, --search-modeYesSpecifies the test case lookup algorithm.
name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path
path: search for test case matching the name within the specified -path. If test case found, update the test case. If test case not found, create a new test case within specified -path
-sm path
-px, --proxyNoHTTP or SOCKS proxy configuration
E.g. socks://username:[email protected]:1080
-px socks://username:[email protected]:1080
-t, --timeoutNoUpload timeout (seconds)-t 10
-tr, --test-runNoTestRail test run name-tr "Chrome Regression Run"
-tp, --test-planNoTestRail test plan Name-tp "Shopping Cart Test Plan"
-mp, --milestone-pathNoTestRail milestone path-mp Milestone1/Milestone2
-cf, --case-fieldsNoTestRail test case custom fields.
The format is [Field label]=[value] pairs, separated with space.
E.g. "Case Field 1=value 1" "Case Field 2=value 2" ...
-cf "Case Field 1=value 1" "Case Field 2=value 2"
-rf, --result-fieldsNoTestRail test result custom fields.
The format is [Field label]=[value] pairs, separated with space.
E.g. "Result Field 1=value 1" "Result Field 2=value 2" ...
-rf "Result Field 1=value 1" "Result Field 2=value 2"
-a, --assignNoSmart Test Failure Assignment. Comma-separated list of TestRail users (email addresses). Railflow will assign failures based on a round robin algorithm.-a [email protected],[email protected]
-af, --assign-fileNoSmart Test Failure Assignment. File path containing list of TestRail users (email addresses).
Note: One user per line
-af /assignees.txt
-cn, --config-namesNoTestRail test plan configuration options.
Configuration format is: [config_group_name]/[config_name].
E.g. "Operating Systems/Linux" "Browsers/Chrome" ...
-cn "Operating Systems/Linux" "Browsers/Chrome"
-cr, --close-runNoIf Railflow should close the corresponding run after uploading test results-cr
-cp, --close-planNoIf Railflow should close the corresponding plan after uploading test results-cp
-dg, --disable-groupingNoIf Railflow should ignore report structure and just upload all tests into a folder which is set by test-path parameter-dg
-tn, --template-nameNoThe name of a template to use in TestRail. If it is not set, 'Test Case (Steps)' or the default one will be used-tn "Test Case (Steps)"
-cst, --case-typeNoThe name of a type for cases-cst other
-csp, --case-priorityNoThe name of a priority for cases-csp medium
-th, --thread-numberNoThe number of concurrent threads for exporting data. Default is 4-th 8
-um, --upload-modeNoUpload mode. Available values are:
0 (default) - create new test cases and do not overwrite existing ones;
1 - create new cases and overwrite existing ones;
2 - do not create new cases and overwrite existing ones;
3 - do not create new cases and do not overwrite existing ones
-um 1
-csf, --case-search-fieldNoThe name of the case field in TestRail which will be using for searching for existing test cases instead of test case title-csf "Custom field"
-ds, --disable-statsNoIf Railflow should disable collecting usage and error logs-ds
-fqtn, --fully-qualified-test-nameNoIf checked, Railflow will use fully qualified test names from the report files for test names in TestRail-fqtn
-us, --untested-statusNoThe name of the status to use in TestRail for untested/skipped tests-us Skipped
-ams, --attachment-max-sizeNoMaximum size of an attachment. Can be set in Gb, Mb, Kb or b. If the unit is not provided considered as Mb.
E.g. 1Gb, 2, 20Kb, 3Mb
-ams 1Gb
-atw, --attachment-type-whitelistNoThe comma-separated list of file extensions which are allowed to upload. E.g. json, .html, .xml-atw "json, .html, .xml"
-atb, --attachment-type-blacklistNoThe comma-separated list of file extensions which are not allowed to upload. E.g. json, .html, .xml-atb "json, .html, .xml"
-nr, --no-runNoDo not create or update Test Run in TestRail-nr
-tfn, --tags-field-nameNoThe name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags-tfn "Cucumber Tags"
-h, --helpNoShow the help information-h
Railflow CLI Example
npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]
Export using License Key
npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt
Export using License File
npx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

Jenkinsfile Example

Jenkinsfile Example using Railflow CLI
pipeline {
    agent any
    environment {
        RAILFLOW_KEY = credentials('railflow-key')
        TESTRAIL_CREDS = credentials('testrail-creds')
    }
    tools {
        jdk 'default'
        maven 'default'
        nodejs 'default'
    }
    stages {
        stage('Checkout'){
            steps{
                checkout([$class: 'GitSCM', branches: [[name: '*/development']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'git-creds', url: 'https://gitrepo.com/workspace/project.git']]])
            }
        }
        stage('Run unit tests') {
            steps {
                sh 'mvn clean test'
            }
        }
    }
    post{
        always{
            echo "Begin exporting to TestRail"
            sh "npx railflow -k ${RAILFLOW_KEY} -url https://testrail_server/ -u ${TESTRAIL_CREDS_USR} -p ${TESTRAIL_CREDS_PSW} -pr \\\"Railflow Demo\\\" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tp TestPlan12345 -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cf \\\"Required text field=123\\\""
        }
    }
}

3. Docker

There are a variety of ways to use Docker within Jenkins. For this example, we will be using the Jenkins Docker Plugin (opens in a new tab) with Railflow CLI Docker Image (opens in a new tab).

In this example below, we are building and publishing the test results from a TestNG project.

Jenkins Pipeline Example
pipeline {
   agent any
   environment {
       RAILFLOW_KEY = credentials('railflow-key') // see https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#handling-credentials
       TESTRAIL_CREDS = credentials('testrail-credentials') // see https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#usernames-and-passwords      
       TESTRAIL_URL = "https://testrail_server/"       
   }
   tools {
       jdk 'default'
       maven 'default'
   }
   stages {
       stage('Checkout') {
           script {
               steps {
                    git branch: 'master', url: 'https://github.com/railflow/testng_example.git'
               }
           }    
       }
       stage('Run unit tests') {
           steps {
               script {
                   sh 'mvn clean test'
               }
           }
       }
   }
   post {
       always {
           sh 'docker pull railflow/railflow:latest'
           withDockerContainer('railflow/railflow:latest') {
               sh 'railflow -k ${RAILFLOW_KEY} -url ${TESTRAIL_URL} -u ${TESTRAIL_CREDS_USR} -p ${TESTRAIL_CREDS_PSW} -pr \"Railflow Demo\" -path Demo/Pizza -f testng -r "**/surefire-reports/testng-results.xml" -sm path -tp "Test Plan" -cf \"Required text field=something\"'
           }
       }
   }
}

Railflow Output

Railflow Jenkins Plugin and CLI both generate rich log output for viewing TestRail integration activity.

Jenkins Build Logs Jenkins plugin

Automatic Test Creation Jenkins plugin

Automatic Test Plan and Runs Jenkins plugin

Jenkins plugin

Test Results Details Jenkins plugin

Automatic Milestones Jenkins plugin

Smart Failure Assignment

Smart Failure assignment is a very powerful feature of Railflow and allows teams to efficiently and strategically assign test failures to specified team members. Doing this automatically as part of the CI process means that teams don't waste valuable time during the test triage process.

⚠️

To use Smart Failure Assignment feature, the users need to have Global Role under Project Access.

Jenkins plugin

Example

Consider a Jenkins Selenium Webdriver job build is failing with 5 test failures, and 2 users configured in the Smart Test Failure Assignment field.

Jenkins plugin

Jenkins Build Logs Jenkins plugin

In this example above, Railflow's Smart Assignment feature filtered all the test failures and distributed them equally across the specified users. This nifty feature greatly eliminates the manual test triage process and saves teams valuable time.

Jenkins plugin Jenkins plugin