Railflow CLI
Railflow Command Line Interface (CLI) is a cross plaform NPM based utility that allows teams to easily integrate their CICD systems or testing framework with TestRail. The CLI can be integrated into any CI application like Jenkins, Github Action, Gitlab, TeamCity, etc using the Railflow Command Line (opens in a new tab) or the Railflow CLI Docker Image (opens in a new tab).
Test Framework Support
Railflow supports a variety of testing framework reporting schemas out of the box and we're always adding support for new ones. Currently, Railflow supports the following
- JUnit
- TestNG
- NUnit
- Allure
- Cucumber BDD
- Robot Framework
- TRX
- xUnit
- PyTest
CLI Reference
Railflow's Command Line Interface (CLI) is wrapper around the TestRail REST API and allows users to process a variety of testing framework reports from any CICD system and automatically export them to TestRail. The CLI provides a host of options so that you can display test reports per your needs.
CLI Requirements
Node version: 20.x or newer
Enable TestRail API
Installation
Railflow CLI is a npm based utility. Installing Railflow is as simple as installing any other npm package.
Installing Railflow CLI
$ npm install railflowCLI Reference
Railflow CLI will automatically create tests, runs, plans, milestones, etc. if they do not exist.
Environment Variables:
To avoid exposing sensitive information like TestRail URL, License, username, and password in CLI arguments, users can set the following environment variables: RAILFLOW_LICENSE, RAILFLOW_TR_URL, RAILFLOW_TR_USER, RAILFLOW_TR_PASSWORD. Railflow CLI will always check these environment variables at run time.
⚠️
Use double quotes for argument values with spaces. Example: --project "demo project"
| Key | Required | Description | Example |
|---|---|---|---|
| -v, --version | No | Outputs Railflow version number | -v |
| -k, --key | -k or -l | (online activation) License key. Can be set with RAILFLOW_LICENSE environment variable | -k XXXXX-XXXXX-XXXXX-XXXXX |
| -l, --license-file | -k or -l | (offline activation) File path or remote url license file | -l /files/ActivationFile.skm |
| -url, --url | Yes | TestRail instance URL. Can be set with RAILFLOW_TR_URL environment variable | -url https://example.testrail.io (opens in a new tab) |
| -u, --username | Yes | TestRail username. Can be set with RAILFLOW_TR_USER environment variable | -u test-username |
| -p, --password | Yes | TestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable | -p XtpHXiPLEODyhF |
| -pr, --project | Yes | TestRail project name | -pr "example project" |
| -path, --test-path | Yes | TestRail test cases path | -path "Section1/subsection2/ShoppingCart |
| -f, --format | Yes | Report format: JUnit, JUnit-Steps, TestNg, TestNg-Steps, Cucumber, NUnit, NUnit-SpecFlow, Allure, Robot, TRX, xUnit, PyTest-Railflow, Playwright (case insensitive) | -f junit |
| -r, --report-files | Yes | The file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/surefire-reports/*.xml can be used. E.g. use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports directory. | -r target/surefire-reports/*.xml target/failsafe-reports/*.xml |
| -sm, --search-mode | Yes | Specifies the test case lookup algorithm. name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path path: search for test case matching the name within the specified -path. If test case found, update the test case. If test case not found, create a new test case within specified -path | -sm path |
| -px, --proxy | No | HTTP or SOCKS proxy configuration E.g. socks://username:[email protected]:1080 | -px socks://username:[email protected]:1080 |
| -t, --timeout | No | Upload timeout (seconds) | -t 10 |
| -tr, --test-run | No | TestRail test run name | -tr "Chrome Regression Run" |
| -tp, --test-plan | No | TestRail test plan Name | -tp "Shopping Cart Test Plan" |
| -mp, --milestone-path | No | TestRail milestone path | -mp Milestone1/Milestone2 |
| -cf, --case-fields | No | TestRail test case custom fields. The format is [Field label]=[value] pairs, separated with space. E.g. "Case Field 1=value 1" "Case Field 2=value 2" ... | -cf "Case Field 1=value 1" "Case Field 2=value 2" |
| -rf, --result-fields | No | TestRail test result custom fields. The format is [Field label]=[value] pairs, separated with space. E.g. "Result Field 1=value 1" "Result Field 2=value 2" ... | -rf "Result Field 1=value 1" "Result Field 2=value 2" |
| -a, --assign | No | Smart Test Failure Assignment. Comma-separated list of TestRail users (email addresses). Railflow will assign failures based on a round robin algorithm. | -a [email protected],[email protected] |
| -af, --assign-file | No | Smart Test Failure Assignment. File path containing list of TestRail users (email addresses). Note: One user per line | -af /assignees.txt |
| -cn, --config-names | No | TestRail test plan configuration options. Configuration format is: [config_group_name]/[config_name]. E.g. "Operating Systems/Linux" "Browsers/Chrome" ... | -cn "Operating Systems/Linux" "Browsers/Chrome" |
| -cr, --close-run | No | If Railflow should close the corresponding run after uploading test results | -cr |
| -cp, --close-plan | No | If Railflow should close the corresponding plan after uploading test results | -cp |
| -dg, --disable-grouping | No | If Railflow should ignore report structure and just upload all tests into a folder which is set by test-path parameter | -dg |
| -tn, --template-name | No | The name of a template to use in TestRail. If it is not set, 'Test Case (Steps)' or the default one will be used | -tn "Test Case (Steps)" |
| -cst, --case-type | No | The name of a type for cases | -cst other |
| -csp, --case-priority | No | The name of a priority for cases | -csp medium |
| -th, --thread-number | No | The number of concurrent threads for exporting data. Default is 4 | -th 8 |
| -um, --upload-mode | No | Upload mode. Available values are: 0 (default) - create new test cases and do not overwrite existing ones; 1 - create new cases and overwrite existing ones; 2 - do not create new cases and overwrite existing ones; 3 - do not create new cases and do not overwrite existing ones | -um 1 |
| -csf, --case-search-field | No | The name of the case field in TestRail which will be using for searching for existing test cases instead of test case title | -csf "Custom field" |
| -ds, --disable-stats | No | If Railflow should disable collecting usage and error logs | -ds |
| -fqtn, --fully-qualified-test-name | No | If checked, Railflow will use fully qualified test names from the report files for test names in TestRail | -fqtn |
| -us, --untested-status | No | The name of the status to use in TestRail for untested/skipped tests | -us Skipped |
| -ams, --attachment-max-size | No | Maximum size of an attachment. Can be set in Gb, Mb, Kb or b. If the unit is not provided considered as Mb. E.g. 1Gb, 2, 20Kb, 3Mb | -ams 1Gb |
| -atw, --attachment-type-whitelist | No | The comma-separated list of file extensions which are allowed to upload. E.g. json, .html, .xml | -atw "json, .html, .xml" |
| -atb, --attachment-type-blacklist | No | The comma-separated list of file extensions which are not allowed to upload. E.g. json, .html, .xml | -atb "json, .html, .xml" |
| -nr, --no-run | No | Do not create or update Test Run in TestRail | -nr |
| -tfn, --tags-field-name | No | The name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags | -tfn "Cucumber Tags" |
| -h, --help | No | Show the help information | -h |
Railflow CLI Example
npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]Export using License Key
npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txtExport using License File
npx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt