Integrating Gitlab with TestRail

Gitlab is one of the most popular commerical SAAS CI applications and is used by thousands of companies to define annd run development and QA pipelines across a variety of testing frameworks and tools like JUnit, Pytest, Selenium, Playwright, etc. While Gitlab is great at running generic tasks, it is not designed to track, triage, or report on test results across a large number of Gitlab builds.

Railflow Command Line (CLI) has been designed to seamlessly export a variety of test results reports from Gitlab Builds to TestRail. Once integrated, TestRail users can use TestRail's powerful dashboard and reporting capabilities, Test Results workflows, and Test Cases Inventory to triage test results that would otherwise be locked up in Gitlab.

Railflow for Gitlab provides the following capabilities for TestRail users

• Railflow CLI: To easily integrate Gitlab workflows with TestRail
• Smart Failure Assignment: Ability to auto assign test failure to team members
• BDD Reports: Support for Cucumber, Serenity, and other popular BDD frameworks
• Framework Reports: Support for TestNG, JUnit, Pytest, Allure, MSTest, Robot, and others.

Command Line Interface

Railflow Command Line (opens in a new tab) is a NPM based utility that can be run on any CircleCI setup (Linux, Windows, OSX, and Railflow CLI Docker Image (opens in a new tab))

CLI Reference

npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

npx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

Gitlab Example

maven-build:
  stage: build
  image:
    name: maven
  variables:
    GIT_DEPTH: 10
  script:
    - mvn clean test || true
  artifacts:
    untracked: true
    paths:
      - ./target/surefire-reports/testng-results.xml
  only:
    - master
 
railflow-test:
  image:
    name: railflow/railflow:latest
    entrypoint: ["/usr/bin/env"]
  stage: test
  variables:
    RAILFLOW_KEY: $RAILFLOW_KEY
    RAILFLOW_USERNAME: $RAILFLOW_USERNAME
    RAILFLOW_PASSWORD: $RAILFLOW_PASSWORD
    TESTRAIL_URL: $TESTRAIL_URL
  dependencies:
    - maven-build
  script:
    - cp ./target/surefire-reports/testng-results.xml /usr/railflow
    - cd /usr/railflow
    - npx railflow -k $RAILFLOW_KEY -url $TESTRAIL_URL -u $RAILFLOW_USERNAME -p $RAILFLOW_PASSWORD -pr "Gitlab-Demo" -path Demo/TestNG -f testng -r ./testng-results.xml -sm path -tp TestPlanName -a john@foo.com, jane@foo.com
  only:
    - master

Railflow Output

Railflow creates a very rich and flexible integration between Gitlab and TestRail. Based on Railflow configuration, TestRail entities can be created or updated automatically as part of your CICD process. The screenshots below shows the output of processing a typical JUnit test framework report.

Gitlab Build Log

Automatic Test Creation

Automatic Test Plan and Runs

Test Results Details

Automatic Milestones

Smart Failure Assignment

Smart Failure assignment is a very powerful feature of Railflow and allows teams to efficiently and strategically assign test failures to specified team members. Doing this automatically as part of the CI process means that teams don't waste valuable time during the test triage process.

Example

Consider a Gitlab Selenium Webdriver build is failing with 5 test failures, and 2 users configured in the Smart Test Failure Assignment field.

Github Actions Build Logs

In this example above, Railflow's Smart Assignment feature filtered all the test failures and distributed them equally across the specified users. This nifty feature greatly eliminates the manual test triage process and saves teams valuable time.