Integrating Cucumber BDD with TestRail

Cucumber BDD framework (opens in a new tab) is a popular functional testing framework which is used by many software development and QA teams. Railflow (opens in a new tab) integration solutions helps users integrate Cucumber BDD tests with TestRail with basic configurations and without any hassle. Railflow consumes the native Cucumber BDD results report, extracts all the results information, and uploads it to TestRail using the TestRail REST API.

Railflow for Cucumber BDD provides the following capabilities

• Test Mapping: Map Cucumber scenarios to existing tests in TestRail
• Auto Extraction: Automatically process Cucumber reports
• Custom Fields: Support for TestRail custom fields.
• TestRail Entities: Update / Create Test Plans, Runs, Milestone, Tests, Results
• Smart Failure Assignment: Auto assign test failures to TestRail user
• CI Application: Support for Jenkins, TeamCity, Gitlab, Github, etc.

Sample Project

To demonstrate this integration, we have created a sample project using Cucumber BDD using Selenium (opens in a new tab)

Feature File

Lets say we have a feature file containing two simple scenarios

Feature: Pizza tests
  Some tests for pizza search
 
  Background:
    Given user navigates to https://google.com
 
  Scenario: There are results for pizza
    When the user enters "pizza" and hits Enter
    Then number of results more than zero
 
  Scenario: There are no bad pizzas
    When the user enters "bad pizza" and hits Enter
    Then no results are shown

Test Mapping

Many teams first define tests in TestRail and then write Cucumber BDD tests. Railflow provies a way to map the Cucumber tests to the existing TestRail tests by using Cucumber tag @testrail.id=C123 in the feature file where C123 is the ID of the existing test case in TestRail. This tag and value is automatically extracted from the cucumber reports by Railflow CLI and used to update the test results in TestRail.

Feature: Pizza tests
  Some tests for pizza search
 
  Background:
    Given user navigates to https://google.com
 
  @testrail.id=C1234
  Scenario: There are results for pizza
    When the user enters "pizza" and hits Enter
    Then number of results more than zero
 
  @testrail.id=C928
  Scenario: There are no bad pizzas
    When the user enters "bad pizza" and hits Enter
    Then no results are shown

Cucumber Test Runner

In order to run JUnit tests together with Cucumber we need to configure Cucumber Test Runner. To do this, just add io.railflow.demo.cucumber.TestRunner class. This code also generates the JSON report in target/test-report.json. This report will be consumed by TestRail CLI or CI Plugins.

package io.railflow.demo.cucumber;
 
import org.junit.runner.RunWith;
 
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
 
 
@RunWith(Cucumber.class)
@CucumberOptions(plugin = { "json:target/test-report.json" })
public class TestRunner {
 
}

mvn clean test

Railflow CLI

Once the Cucumber test-report.json is generated, it can be uploaded to TestRail using the Railflow CLI (opens in a new tab) or CI plugins (opens in a new tab).

npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f cucumber -r test-report.json -sm <search mode> -tp [test plan name] -mp [milestone path]

CLI Reference