Docs
CICD
Travis

Integrating Travis with TestRail

Travis is a very popular commerical SAAS CI applications and is used by thousands of companies to define annd run development and QA pipelines across a variety of testing frameworks and tools like JUnit, Pytest, Selenium, Playwright, etc. While Travis is great at running generic tasks, it is not designed to track, triage, or report on test results across a large number of Travis builds.

Railflow Command Line (CLI) has been designed to seamlessly export a variety of test results reports from Travis Builds to TestRail. Once integrated, TestRail users can use TestRail's powerful dashboard and reporting capabilities, Test Results workflows, and Test Cases Inventory to triage test results that would otherwise be locked up in Travis.

Railflow for Travis provides the following capabilities for TestRail users

  • Railflow CLI: To easily integrate Travis workflows with TestRail
  • Smart Failure Assignment: Ability to auto assign test failure to team members
  • BDD Reports: Support for Cucumber, Serenity, and other popular BDD frameworks
  • Framework Reports: Support for TestNG, JUnit, Pytest, Allure, MSTest, Robot, and others.

Command Line Interface

Node version: 20.x or newer

Railflow Command Line (opens in a new tab) is a NPM based utility that can be run on any Travis setup (Linux, Windows, OSX, and Railflow CLI Docker Image (opens in a new tab))

CLI Reference

Railflow CLI will automatically create tests, runs, plans, milestones, etc. if they do not exist.

Environment Variables: To avoid exposing sensitive information like TestRail URL, License, username, and password in CLI arguments, users can set the following environment variables: RAILFLOW_LICENSE, RAILFLOW_TR_URL, RAILFLOW_TR_USER, RAILFLOW_TR_PASSWORD. Railflow CLI will always check these environment variables at run time.

⚠️

Use double quotes for argument values with spaces. Example: --project "demo project"

KeyRequiredDescriptionExample
-v, --versionNoOutputs Railflow version number-v
-k, --key-k or -l(online activation) License key. Can be set with RAILFLOW_LICENSE environment variable-k XXXXX-XXXXX-XXXXX-XXXXX
-l, --license-file-k or -l(offline activation) File path or remote url license file-l /files/ActivationFile.skm
-url, --urlYesTestRail instance URL. Can be set with RAILFLOW_TR_URL environment variable-url https://example.testrail.io (opens in a new tab)
-u, --usernameYesTestRail username. Can be set with RAILFLOW_TR_USER environment variable-u test-username
-p, --passwordYesTestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable-p XtpHXiPLEODyhF
-pr, --projectYesTestRail project name-pr "example project"
-path, --test-pathYesTestRail test cases path-path "Section1/subsection2/ShoppingCart
-f, --formatYesReport format: JUnit, JUnit-Steps, TestNg, TestNg-Steps, Cucumber, NUnit, NUnit-SpecFlow, Allure, Robot, TRX, xUnit, PyTest-Railflow, Playwright (case insensitive)-f junit
-r, --report-filesYesThe file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/surefire-reports/*.xml can be used.
E.g. use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports directory.
-r target/surefire-reports/*.xml target/failsafe-reports/*.xml
-sm, --search-modeYesSpecifies the test case lookup algorithm.
name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path
path: search for test case matching the name within the specified -path. If test case found, update the test case. If test case not found, create a new test case within specified -path
-sm path
-px, --proxyNoHTTP or SOCKS proxy configuration
E.g. socks://username:[email protected]:1080
-px socks://username:[email protected]:1080
-t, --timeoutNoUpload timeout (seconds)-t 10
-tr, --test-runNoTestRail test run name-tr "Chrome Regression Run"
-tp, --test-planNoTestRail test plan Name-tp "Shopping Cart Test Plan"
-mp, --milestone-pathNoTestRail milestone path-mp Milestone1/Milestone2
-cf, --case-fieldsNoTestRail test case custom fields.
The format is [Field label]=[value] pairs, separated with space.
E.g. "Case Field 1=value 1" "Case Field 2=value 2" ...
-cf "Case Field 1=value 1" "Case Field 2=value 2"
-rf, --result-fieldsNoTestRail test result custom fields.
The format is [Field label]=[value] pairs, separated with space.
E.g. "Result Field 1=value 1" "Result Field 2=value 2" ...
-rf "Result Field 1=value 1" "Result Field 2=value 2"
-a, --assignNoSmart Test Failure Assignment. Comma-separated list of TestRail users (email addresses). Railflow will assign failures based on a round robin algorithm.-a [email protected],[email protected]
-af, --assign-fileNoSmart Test Failure Assignment. File path containing list of TestRail users (email addresses).
Note: One user per line
-af /assignees.txt
-cn, --config-namesNoTestRail test plan configuration options.
Configuration format is: [config_group_name]/[config_name].
E.g. "Operating Systems/Linux" "Browsers/Chrome" ...
-cn "Operating Systems/Linux" "Browsers/Chrome"
-cr, --close-runNoIf Railflow should close the corresponding run after uploading test results-cr
-cp, --close-planNoIf Railflow should close the corresponding plan after uploading test results-cp
-dg, --disable-groupingNoIf Railflow should ignore report structure and just upload all tests into a folder which is set by test-path parameter-dg
-tn, --template-nameNoThe name of a template to use in TestRail. If it is not set, 'Test Case (Steps)' or the default one will be used-tn "Test Case (Steps)"
-cst, --case-typeNoThe name of a type for cases-cst other
-csp, --case-priorityNoThe name of a priority for cases-csp medium
-th, --thread-numberNoThe number of concurrent threads for exporting data. Default is 4-th 8
-um, --upload-modeNoUpload mode. Available values are:
0 (default) - create new test cases and do not overwrite existing ones;
1 - create new cases and overwrite existing ones;
2 - do not create new cases and overwrite existing ones;
3 - do not create new cases and do not overwrite existing ones
-um 1
-csf, --case-search-fieldNoThe name of the case field in TestRail which will be using for searching for existing test cases instead of test case title-csf "Custom field"
-ds, --disable-statsNoIf Railflow should disable collecting usage and error logs-ds
-fqtn, --fully-qualified-test-nameNoIf checked, Railflow will use fully qualified test names from the report files for test names in TestRail-fqtn
-us, --untested-statusNoThe name of the status to use in TestRail for untested/skipped tests-us Skipped
-ams, --attachment-max-sizeNoMaximum size of an attachment. Can be set in Gb, Mb, Kb or b. If the unit is not provided considered as Mb.
E.g. 1Gb, 2, 20Kb, 3Mb
-ams 1Gb
-atw, --attachment-type-whitelistNoThe comma-separated list of file extensions which are allowed to upload. E.g. json, .html, .xml-atw "json, .html, .xml"
-atb, --attachment-type-blacklistNoThe comma-separated list of file extensions which are not allowed to upload. E.g. json, .html, .xml-atb "json, .html, .xml"
-nr, --no-runNoDo not create or update Test Run in TestRail-nr
-tfn, --tags-field-nameNoThe name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags-tfn "Cucumber Tags"
-h, --helpNoShow the help information-h
Railflow CLI Example
npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]
Export using License Key
npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt
Export using License File
npx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

Travis Example

Travis Pipeline Example
language: java
 
git:
  depth: 3
 
services:
  - docker
 
install: true
 
before_install:
- docker pull railflow/railflow:latest
- docker run -d railflow/railflow:latest /bin/sh -c "cd /usr/railflow"
 
script:
- mvn clean test || true
- npx railflow -k $RAILFLOW_KEY -url $TESTRAIL_URL -u $RAILFLOW_USERNAME -p $RAILFLOW_PASSWORD -pr "Github-Demo" -path Demo/TestNG -f testng -r ./target/surefire-reports/testng-results.xml -sm path -a john@foo.com, jane@foo.com -tp TestPlanName 
 

Railflow Output

Railflow creates a very rich and flexible integration between Circle CI and TestRail. Based on Railflow configuration, TestRail entities can be created or updated automatically as part of your CICD process. The screenshots below shows the output of processing a typical JUnit test framework report.

Travis Build Logs Travis CI

Automatic Test Creation automatic test creation

Automatic Test Plan and Runs automatic testplan

automatic testrun

Test Results Details test results

Automatic Milestones automatic milestones

Smart Failure Assignment

Smart Failure assignment is a very powerful feature of Railflow and allows teams to efficiently and strategically assign test failures to specified team members. Doing this automatically as part of the CI process means that teams don't waste valuable time during the test triage process.

⚠️

To use Smart Failure Assignment feature, the users need to have Global Role under Project Access.

smart failure

Example

Consider a Travis Selenium Webdriver build is failing with 5 test failures, and 2 users configured in the Smart Test Failure Assignment field.

Travis CI

Travis Build Logs Travis CI

In this example above, Railflow's Smart Assignment feature filtered all the test failures and distributed them equally across the specified users. This nifty feature greatly eliminates the manual test triage process and saves teams valuable time.

smart failure smart failure