Integrating Travis with TestRail

Travis is a very popular commerical SAAS CI applications and is used by thousands of companies to define annd run development and QA pipelines across a variety of testing frameworks and tools like JUnit, Pytest, Selenium, Playwright, etc. While Travis is great at running generic tasks, it is not designed to track, triage, or report on test results across a large number of Travis builds.

Railflow Command Line (CLI) has been designed to seamlessly export a variety of test results reports from Travis Builds to TestRail. Once integrated, TestRail users can use TestRail's powerful dashboard and reporting capabilities, Test Results workflows, and Test Cases Inventory to triage test results that would otherwise be locked up in Travis.

Railflow for Travis provides the following capabilities for TestRail users

• Railflow CLI: To easily integrate Travis workflows with TestRail
• Smart Failure Assignment: Ability to auto assign test failure to team members
• BDD Reports: Support for Cucumber, Serenity, and other popular BDD frameworks
• Framework Reports: Support for TestNG, JUnit, Pytest, Allure, MSTest, Robot, and others.

Command Line Interface

Railflow Command Line (opens in a new tab) is a NPM based utility that can be run on any Travis setup (Linux, Windows, OSX, and Railflow CLI Docker Image (opens in a new tab))

CLI Reference

npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

npx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

Travis Example

language: java
 
git:
  depth: 3
 
services:
  - docker
 
install: true
 
before_install:
- docker pull railflow/railflow:latest
- docker run -d railflow/railflow:latest /bin/sh -c "cd /usr/railflow"
 
script:
- mvn clean test || true
- npx railflow -k $RAILFLOW_KEY -url $TESTRAIL_URL -u $RAILFLOW_USERNAME -p $RAILFLOW_PASSWORD -pr "Github-Demo" -path Demo/TestNG -f testng -r ./target/surefire-reports/testng-results.xml -sm path -a john@foo.com, jane@foo.com -tp TestPlanName 
 

Railflow Output

Railflow creates a very rich and flexible integration between Circle CI and TestRail. Based on Railflow configuration, TestRail entities can be created or updated automatically as part of your CICD process. The screenshots below shows the output of processing a typical JUnit test framework report.

Travis Build Logs

Automatic Test Creation

Automatic Test Plan and Runs

Test Results Details

Automatic Milestones

Smart Failure Assignment

Smart Failure assignment is a very powerful feature of Railflow and allows teams to efficiently and strategically assign test failures to specified team members. Doing this automatically as part of the CI process means that teams don't waste valuable time during the test triage process.

Example

Consider a Travis Selenium Webdriver build is failing with 5 test failures, and 2 users configured in the Smart Test Failure Assignment field.

Travis Build Logs

In this example above, Railflow's Smart Assignment feature filtered all the test failures and distributed them equally across the specified users. This nifty feature greatly eliminates the manual test triage process and saves teams valuable time.