Integrating ROBOT Framework with TestRail

ROBOT framework (opens in a new tab) is a Keywork driven frameworks used as a generic test automation frameworks and for robot process automation (RPA). Railflow (opens in a new tab) integration solutions helps users integrate ROBOT tests with TestRail with basic configurations and without any hassle. Railflow consumes the native ROBOT results report, extracts all the results information, and uploads it to TestRail using the TestRail REST API.

Railflow for ROBOT Framework provides the following capabilities

Test Mapping: Map ROBOT tests with existing tests in TestRail
Auto Extraction: Automatically process ROBOT reports
Custom Fields: Support for TestRail custom fields.
Smart Failure Assignment: Auto assign test failures to TestRail user
TestRail Entities: Update / Create Test Plans, Runs, Milestone, Tests, Results
CI Application: Support Jenkins, TeamCity, Gitlab, Github, etc.

Sample Project

To demonstrate this integration, we have created a Sample ROBOT Framework Project (opens in a new tab).

ROBOT File

Lets take this simple Robot test file and run it

1-login.robot

*** Settings ***
Documentation  A test suite with a single test for valid login.
Library  ../resources/ChromeDriverProvider.py
Library  Selenium2Library
Test Teardown  Close All Browsers
 
*** Variables ***
${App URL}  https://rwa.railflow.io
${Login URL}  ${App URL}/signin
${Login button}  xpath=//*[@id="root"]/div/main/div[1]/form/button
${Login name}  xpath=//*[@id="root"]/div/div/div/div[1]/div[2]/h6[2]
${Error}  xpath=//*[@id="root"]/div/main/div[1]/div[1]/div[2]
 
*** Test Cases ***
Valid Login
  Open Browser To Login Page
  Input Text  username  Katharina_Bernier
  Input Text  password  s3cret
  Click Log in
  Welcome Page Should Be Open
  
Invalid Login
  Open Browser To Login Page  
  Input Text  username  Katharina_Bernier
  Input Text  password  whatever
  Click Log in
  Error Message Should Be Displayed
 
*** Keywords ***
Open Browser To Login Page
  ${DRIVER_PATH}  Get Chromedriver Path
  Open Browser  ${LOGIN URL}  Chrome  executable_path=${DRIVER_PATH}
  Maximize Browser Window
  Wait Until Page Contains Element  ${Login button}  5s
 
Click Log in
  Click Element  ${Login button}
 
Welcome Page Should Be Open    
  Wait Until Location Is  ${App URL}  5s  
  Wait Until Page Contains Element   ${Login name}  5s
  ${name}  Get Text  ${Login name}
  Should Be Equal As Strings  ${name}  @Katharina_Bernier  
  
Error Message Should Be Displayed
    Wait Until Location Is  ${Login URL}  5s
    Wait Until Page Contains Element  ${Error}  5s        
    ${error message}  Get Text  ${Error}
    Should Be Equal As Strings  ${error message}  Username or password is invalid

This will run the test above and produces the output.xml report file in the results folder. This report will be consumed by TestRail CLI or CI Plugins.

$ robot robot -d results tests/

Test Mapping

Sometimes test cases are already defined in TestRail and you may want to map them to ROBOT tests. Railflow allows users to map results of automated test execution back to the existing test cases in TestRail by providing test IDs. In order to do that a testrail.id=C123 tag should be added to a Robot test, where C123 is the ID of the existing test case.

Example: You can add one or more testrail.id tags in ROBOT test file:

1-login.robot

*** Settings ***
Documentation  A test suite with a single test for valid login.
Library  ../resources/ChromeDriverProvider.py
Library  Selenium2Library
Test Teardown  Close All Browsers
 
*** Variables ***
${App URL}  https://rwa.railflow.io
${Login URL}  ${App URL}/signin
${Login button}  xpath=//*[@id="root"]/div/main/div[1]/form/button
${Login name}  xpath=//*[@id="root"]/div/div/div/div[1]/div[2]/h6[2]
${Error}  xpath=//*[@id="root"]/div/main/div[1]/div[1]/div[2]
 
*** Test Cases ***
Valid Login
  [Tags]  testrail.id=C1825  testrail.id=C1826
  Open Browser To Login Page
  Input Text  username  Katharina_Bernier
  Input Text  password  s3cret
  Click Log in
  Welcome Page Should Be Open
  
Invalid Login
  Open Browser To Login Page  
  Input Text  username  Katharina_Bernier
  Input Text  password  whatever
  Click Log in
  Error Message Should Be Displayed
 
*** Keywords ***
Open Browser To Login Page
  ${DRIVER_PATH}  Get Chromedriver Path
  Open Browser  ${LOGIN URL}  Chrome  executable_path=${DRIVER_PATH}
  Maximize Browser Window
  Wait Until Page Contains Element  ${Login button}  5s
 
Click Log in
  Click Element  ${Login button}
 
Welcome Page Should Be Open    
  Wait Until Location Is  ${App URL}  5s  
  Wait Until Page Contains Element   ${Login name}  5s
  ${name}  Get Text  ${Login name}
  Should Be Equal As Strings  ${name}  @Katharina_Bernier  
  
Error Message Should Be Displayed
    Wait Until Location Is  ${Login URL}  5s
    Wait Until Page Contains Element  ${Error}  5s        
    ${error message}  Get Text  ${Error}
    Should Be Equal As Strings  ${error message}  Username or password is invalid

Valid Login test case is now mapped to Existing test case one and Existing test case two

Existing test cases in TestRail

Railflow CLI

Once the ROBOT output.xml report is generated, it can be uploaded to TestRail using the Railflow CLI (opens in a new tab) or CI plugins (opens in a new tab).

Railflow CLI command for ROBOT

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path "UI Tests/Railflow" -f robot -r results/output.xml -sm path

Test Run details Alt Test run in TestRail

Test Result details Alt Test results in TestRail

Test Result with screenshots (if present in report) Alt Test results in TestRail

CLI Reference

Railflow CLI will automatically create tests, runs, plans, milestones, etc. if they do not exist.

Environment Variables: To avoid exposing sensitive information like TestRail URL, License, username, and password in CLI arguments, users can set the following environment variables: RAILFLOW_LICENSE, RAILFLOW_TR_URL, RAILFLOW_TR_USER, RAILFLOW_TR_PASSWORD. Railflow CLI will always check these environment variables at run time.

⚠️

Use double quotes for argument values with spaces. Example: --project "demo project"

Key	Required	Description	Example
-v, --version	No	Outputs Railflow version number	-v
-k, --key	-k or -l	(online activation) License key. Can be set with RAILFLOW_LICENSE environment variable	-k XXXXX-XXXXX-XXXXX-XXXXX
-l, --license-file	-k or -l	(offline activation) File path or remote url license file	-l /files/ActivationFile.skm
-url, --url	Yes	TestRail instance URL. Can be set with RAILFLOW_TR_URL environment variable	-url https://example.testrail.io (opens in a new tab)
-u, --username	Yes	TestRail username. Can be set with RAILFLOW_TR_USER environment variable	-u test-username
-p, --password	Yes	TestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable	-p XtpHXiPLEODyhF
-pr, --project	Yes	TestRail project name	-pr "example project"
-path, --test-path	Yes	TestRail test cases path	-path "Section1/subsection2/ShoppingCart
-f, --format	Yes	Report format: JUnit, JUnit-Steps, TestNg, TestNg-Steps, Cucumber, NUnit, NUnit-SpecFlow, Allure, Robot, TRX, xUnit, PyTest-Railflow, Playwright (case insensitive)	-f junit
-r, --report-files	Yes	The file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as */surefire-reports/.xml can be used. E.g. use *target/surefire-reports/.xml to capture all XML files in target/surefire-reports** directory.	-r target/surefire-reports/.xml target/failsafe-reports/.xml
-sm, --search-mode	Yes	Specifies the test case lookup algorithm. `name:` search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified `-path` `path:` search for test case matching the name within the specified `-path`. If test case found, update the test case. If test case not found, create a new test case within specified `-path`	-sm path
-px, --proxy	No	HTTP or SOCKS proxy configuration E.g. socks://username:[email protected]:1080	-px socks://username:[email protected]:1080
-t, --timeout	No	Upload timeout (seconds)	-t 10
-tr, --test-run	No	TestRail test run name	-tr "Chrome Regression Run"
-tp, --test-plan	No	TestRail test plan Name	-tp "Shopping Cart Test Plan"
-mp, --milestone-path	No	TestRail milestone path	-mp Milestone1/Milestone2
-cf, --case-fields	No	TestRail test case custom fields. The format is `[Field label]=[value]` pairs, separated with space. E.g. "Case Field 1=value 1" "Case Field 2=value 2" ...	-cf "Case Field 1=value 1" "Case Field 2=value 2"
-rf, --result-fields	No	TestRail test result custom fields. The format is `[Field label]=[value]` pairs, separated with space. E.g. "Result Field 1=value 1" "Result Field 2=value 2" ...	-rf "Result Field 1=value 1" "Result Field 2=value 2"
-a, --assign	No	Smart Test Failure Assignment. Comma-separated list of TestRail users (email addresses). Railflow will assign failures based on a round robin algorithm.	-a [email protected],[email protected]
-af, --assign-file	No	Smart Test Failure Assignment. File path containing list of TestRail users (email addresses). `Note: One user per line`	-af /assignees.txt
-cn, --config-names	No	TestRail test plan configuration options. Configuration format is: [config_group_name]/[config_name]. E.g. "Operating Systems/Linux" "Browsers/Chrome" ...	-cn "Operating Systems/Linux" "Browsers/Chrome"
-cr, --close-run	No	If Railflow should close the corresponding run after uploading test results	-cr
-cp, --close-plan	No	If Railflow should close the corresponding plan after uploading test results	-cp
-dg, --disable-grouping	No	If Railflow should ignore report structure and just upload all tests into a folder which is set by test-path parameter	-dg
-tn, --template-name	No	The name of a template to use in TestRail. If it is not set, 'Test Case (Steps)' or the default one will be used	-tn "Test Case (Steps)"
-cst, --case-type	No	The name of a type for cases	-cst other
-csp, --case-priority	No	The name of a priority for cases	-csp medium
-th, --thread-number	No	The number of concurrent threads for exporting data. Default is 4	-th 8
-um, --upload-mode	No	Upload mode. Available values are: 0 (default) - create new test cases and do not overwrite existing ones; 1 - create new cases and overwrite existing ones; 2 - do not create new cases and overwrite existing ones; 3 - do not create new cases and do not overwrite existing ones	-um 1
-csf, --case-search-field	No	The name of the case field in TestRail which will be using for searching for existing test cases instead of test case title	-csf "Custom field"
-ds, --disable-stats	No	If Railflow should disable collecting usage and error logs	-ds
-fqtn, --fully-qualified-test-name	No	If checked, Railflow will use fully qualified test names from the report files for test names in TestRail	-fqtn
-us, --untested-status	No	The name of the status to use in TestRail for untested/skipped tests	-us Skipped
-ams, --attachment-max-size	No	Maximum size of an attachment. Can be set in Gb, Mb, Kb or b. If the unit is not provided considered as Mb. E.g. 1Gb, 2, 20Kb, 3Mb	-ams 1Gb
-atw, --attachment-type-whitelist	No	The comma-separated list of file extensions which are allowed to upload. E.g. `json, .html, .xml`	-atw "json, .html, .xml"
-atb, --attachment-type-blacklist	No	The comma-separated list of file extensions which are not allowed to upload. E.g. `json, .html, .xml`	-atb "json, .html, .xml"
-nr, --no-run	No	Do not create or update Test Run in TestRail	-nr
-tfn, --tags-field-name	No	The name of a test case field which will be holding tags from the report file if any. E.g. `Cucumber Tags`	-tfn "Cucumber Tags"
-h, --help	No	Show the help information	-h

Cypress Overview