Integrating CircleCI with TestRail

CircleCI is one of the most popular commerical SAAS CI applications and is used by thousands of companies to define annd run development and QA pipelines across a variety of testing frameworks and tools like JUnit, Pytest, Selenium, Playwright, etc. While CircleCI is great at running generic tasks, it is not designed to track, triage, or report on test results across a large number of CircleCI builds.

Railflow Command Line (CLI) has been designed to seamlessly export a variety of test results reports from CircleCI Builds to TestRail. Once integrated, TestRail users can use TestRail's powerful dashboard and reporting capabilities, Test Results workflows, and Test Cases Inventory to triage test results that would otherwise be locked up in CircleCI.

Railflow for CircleCI provides the following capabilities for TestRail users

• Railflow CLI: To easily integrate CircleCI workflows with TestRail
• Smart Failure Assignment: Ability to auto assign test failure to team members
• BDD Reports: Support for Cucumber, Serenity, and other popular BDD frameworks
• Framework Reports: Support for TestNG, JUnit, Pytest, Allure, MSTest, Robot, and others.

Command Line Interface

Railflow Command Line (opens in a new tab) is a NPM based utility that can be run on any CircleCI setup (Linux, Windows, OSX, and Railflow CLI Docker Image (opens in a new tab))

CLI Reference

npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

npx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

CircleCI Example

version: 2
jobs:
  build:
    docker:
      - image: circleci/openjdk:8-jdk
 
    working_directory: ~/repo
 
    environment:
      # Customize the JVM maximum heap limit
      JVM_OPTS: -Xmx3200m
      TERM: dumb
 
    steps:
      - checkout
      # run tests!
      - run: mvn clean test || true
      - run: ls .
      - store_artifacts:
          path: ./target/surefire-reports/testng-results.xml
 
  railflow_test:
    docker:
      - image: railflow/railflow:latest
 
    working_directory: /usr/railflow
    steps:
      - run:
          name: download artifacts
          command: |
            curl -H "Circle-Token: $CIRCLE_TOKEN" https://circleci.com/api/v1.1/project/github/railflow/testng_example/10/artifacts \
            | grep -o 'https://[^"]*' \
            | wget --verbose --header "Circle-Token: $CIRCLE_TOKEN" --input-file -
      - run:
          name: Run railflow testing
          command: npx railflow -k $RAILFLOW_KEY -url $TESTRAIL_URL -u $RAILFLOW_USERNAME -p $RAILFLOW_PASSWORD -pr "CircleCI-Demo" -path Demo/TestNG -f testng -a john@foo.com, jane@foo.com -r /usr/railflow/testng-results.xml -sm path -tp TestPlanName
 
workflows:
  version: 2
  build_and_test:
    jobs:
      - build
      - railflow_test:
          requires:
            - build
 

Railflow Output

Railflow creates a very rich and flexible integration between Circle CI and TestRail. Based on Railflow configuration, TestRail entities can be created or updated automatically as part of your CICD process. The screenshots below shows the output of processing a typical JUnit test framework report.

Circle CI Build Logs

Automatic Test Creation

Automatic Test Plan and Runs

Test Results Details

Automatic Milestones

Smart Failure Assignment

Smart Failure assignment is a very powerful feature of Railflow and allows teams to efficiently and strategically assign test failures to specified team members. Doing this automatically as part of the CI process means that teams don't waste valuable time during the test triage process.

Example

Consider a CircleCI Selenium Webdriver build is failing with 5 test failures, and 2 users configured in the Smart Test Failure Assignment field.

Circle CI Build Logs

In this example above, Railflow's Smart Assignment feature filtered all the test failures and distributed them equally across the specified users. This nifty feature greatly eliminates the manual test triage process and saves teams valuable time.