Integrating TeamCity with TestRail
TeamCity is one of the most popular commerical CI applications and is used by thousands of companies to define annd run development and QA pipelines across a variety of testing frameworks and tools like JUnit, Pytest, Selenium, Playwright, etc. While Jenkins is great at running generic tasks, it is not designed to track, triage, or report on test results across a large number of TeamCity jobs and servers.
Railflow TeamCity Plugin and Command Line (CLI) has been designed to seamlessly export a variety of test results reports from TeamCity Builds to TestRail. Once integrated, TestRail users can use TestRail's powerful dashboard and reporting capabilities, Test Results workflows, and Test Cases Inventory to triage test results that would otherwise be locked up in TeamCity.
Railflow for TeamCity provides the following capabilities for TestRail users
Railflow TeamCity Pluginfor plugin based rich UX controlsRailflow CLIfor a non-plugin based dockerized or NPM experienceSmart Failure Assignment: Ability to auto assign test failure to team membersBDD Reports: Support for Cucumber, Serenity, and other popular BDD frameworksFramework Reports: Support for TestNG, JUnit, Pytest, Allure, MSTest, Robot, and others.
Railflow can be setup with TeamCity in three ways.
1. TeamCity Plugin
TeamCity version: 2020.1 or newer
TeamCity plugins are the standard way of adding new capabilities to a TeamCity instance. Railflow TeamCity plugin exposes a new Post Builds step for easily integrating TeamCity with TestRail.
Install
Railflow can be downloaded from the TeamCity Plugin Marketplace (opens in a new tab) or directly from our website Railflow Teamcity Plugin (opens in a new tab)
Install the plugin via TeamCity's Plugin Administration menu. Install the Railflow plugin and then restart TeamCity.
Configure
To configure plugin, navigate to TeamCity Administration → Railflow section
License Key (online activation): Select License Key option and copy/paste the license key from your welcome email and click Activate License
License File (offline activation): If your Teamcity instance does not have outbound internet access, you may use this option to upload the license activation from your email. The license file has a .skm extension and part of your welcome email.
Railflow plugin configuration section allows you to define multiple TestRail servers. Once configured, they can be easily referenced in the TeamCity jobs.
Project Config
Railflow plugin's rich controls offer a variety of ways to to integrate TeamCity Projects with TestRail by using the Railflow Build Step: Railflow Plugin: TestRail Test Results Processor. This build step allows you to specify your test framework, test results location, and other TestRail configurations.
TeamCity Post Build Alert
To ensure that test results are posted to TestRail even if the TeamCity build fails (very common), set Execute step field to Even if some of the previous steps failed.
Build Step Reference
The Railflow build step provides users a host of options to integrate Teamcity with TestRail in a variety of ways and across any testing tool and framework. The reference below describes all the options and their usage.
| Field Name | Required | Description |
|---|---|---|
| TestRail Server | Yes | Select one of the server names configured in "Global settings configuration" |
| TestRail Project | Yes | The name of a project in TestRail to which results should be exported |
| Fail build if upload unsuccessful | N/A | If checked, the build will be marked as failed if for any reason the plugin could not upload the results. This could be due to Railflow issues, TestRail server issues, network issues, etc. |
| Override TestRail Credentials | No If specified, it overrides TestRail user credentials defined in Global Configuration | |
| Results File Pattern | Yes | The file path to the test report file(s) generated during the build. Ant-style patterns such as **/surefire-reports/*.xml can be used.E.g. use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports** directory. |
| Report Format | Yes | Results file format |
| Search Mode | Yes | Specifies the test case lookup algorithm. Name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified Test Case Path Path: search for test case matching the name within the specified Test Case Path. If test case found, update the test case. If test case not found, create a new test case within specified Test Case Path |
| Test Case Path | Yes | Path to where Railflow should upload test case definitions, must contain suite name in the beginning (for single-suite project, suite name is always 'Master'), e.g. Master/Section1/Section2 |
| Test Plan Name | No | Name of a test plan in TestRail to which test results will be added |
| Test Run Name | No | Name of a test run in TestRail to which test results will be added |
| Milestone Path | No | Path to a milestone in TestRail to which test run/plan will be added. E.g. Milestone1/Milestone2 |
| Smart Test Failure Assignment | No | A comma separated list of user emails for smart failure assignment. Each failed result will be assigned to a person from this list in a round robin fashion |
| Log Level | No | Logging level for the plugin log file (.railflow.log) |
| Test Case Type | No | Name of a case type in TestRail, e.g. Automated |
| Test Case Priority | No | Name of a case priority in TestRail, e.g. High |
| Test Case Template | No | Name of a TestRail template. If it is blank, Test Case (Steps) will be used. |
| Test Case Custom Fields | No | Values for case fields in TestRail can be specified in this field. The format is [TestRail field label]=[value] and each field name\value pair should start with the new line. E.g.: Custom Field One=foo Custom Field Two=bar |
| Test Result Custom Fields | No | Values for result fields in TestRail can be specified in this field. The format is [TestRail field label]=[value] and each field name\value pair should start with the new line. E.g.: Custom Result Field One=foo Custom Result Field Two=bar |
| Configuration Names | No | A list of configuration names in TestRail. The format is [Config Group Name]/[Config Name]. Each entry must start with the new line. E.g.: Operating Systems/Linux Browsers/Chrome |
| Case Search Field | No | The name of the case field in TestRail which will be using for searching for existing test cases instead of test case title |
| Tags Field Name | No | The name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags |
| Upload Mode | No | Test case upload mode: Create new test cases and do not overwrite existing ones: If test case not found, create a new test case within specified Test Case Path. If test case found, do not update the test case. Create new cases and overwrite existing ones: If test case not found, create a new test case within specified Test Case Path. If test case found, update the test case. Do not create new cases and overwrite existing ones: If test case not found, do not create a new test case and the corresponding test result will not be uploaded into TestRail. If test case found, update the test case. Do not create new cases and do not overwrite existing ones: If test case not found, do not create a new test case and the corresponding test result will not be uploaded into TestRail. If test case found, do not update the test case. |
| Disable Grouping | No | If checked, Railflow will ignore structure in report files and upload all test cases into one Section, defined by the Test Path parameter. |
| Close Run | No | If checked, Railflow will close the test run in TestRail and archive its tests and results |
| Close Plan | No | If checked, Railflow will close the test plan in TestRail and archive its tests and results |
| Fully Qualified Test Name | No | If checked, Railflow will use fully qualified test names from the report files for test names in TestRail |
2. Command Line Interface
TeamCity version: 2020.1 or newer
Node version: 20.x or newer
TeamCity plugins can be a pain to manage and can occassionally cause issues on the TeamCity server. In addition, any updates to plugins need to be carefully managed and planned with TeamCity admins. This is true of all TeamCity plugins.
Railflow Command Line (opens in a new tab) is a NPM based utility that can be run on any TeamCity setup (Linux, Windows, OSX, and Railflow CLI Docker Image (opens in a new tab)) without the need for any TeamCity plugins.
To integrate Railflow CLI TeamCity Project, add a new Build Step to your TeamCity Project Build Configuration as follows
Runner Type→ Command LineStep name→ Any name like 'Railflow TestRail Integration'Execute Step→ Even if some of the previous steps failedCustom Script→npx railflow -k %RAILFLOW_LICENSE% -URL %TESTRAIL_URL% [other cmdline parameters]
npx railflow -k %RAILFLOW_KEY% -url https://testrail.server.address/ -u %TESTRAIL_CREDS_USER% -p %TESTRAIL_CREDS_PASS% -pr \"Railflow Demo\" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2CLI reference
Railflow CLI will automatically create tests, runs, plans, milestones, etc. if they do not exist.
Environment Variables:
To avoid exposing sensitive information like TestRail URL, License, username, and password in CLI arguments, users can set the following environment variables: RAILFLOW_LICENSE, RAILFLOW_TR_URL, RAILFLOW_TR_USER, RAILFLOW_TR_PASSWORD. Railflow CLI will always check these environment variables at run time.
Use double quotes for argument values with spaces. Example: --project "demo project"
| Key | Required | Description | Example |
|---|---|---|---|
| -v, --version | No | Outputs Railflow version number | -v |
| -k, --key | -k or -l | (online activation) License key. Can be set with RAILFLOW_LICENSE environment variable | -k XXXXX-XXXXX-XXXXX-XXXXX |
| -l, --license-file | -k or -l | (offline activation) File path or remote url license file | -l /files/ActivationFile.skm |
| -url, --url | Yes | TestRail instance URL. Can be set with RAILFLOW_TR_URL environment variable | -url https://example.testrail.io (opens in a new tab) |
| -u, --username | Yes | TestRail username. Can be set with RAILFLOW_TR_USER environment variable | -u test-username |
| -p, --password | Yes | TestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable | -p XtpHXiPLEODyhF |
| -pr, --project | Yes | TestRail project name | -pr "example project" |
| -path, --test-path | Yes | TestRail test cases path | -path "Section1/subsection2/ShoppingCart |
| -f, --format | Yes | Report format: JUnit, JUnit-Steps, TestNg, TestNg-Steps, Cucumber, NUnit, NUnit-SpecFlow, Allure, Robot, TRX, xUnit, PyTest-Railflow, Playwright (case insensitive) | -f junit |
| -r, --report-files | Yes | The file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/surefire-reports/*.xml can be used. E.g. use target/surefire-reports/*.xml to capture all XML files in target/surefire-reports directory. | -r target/surefire-reports/*.xml target/failsafe-reports/*.xml |
| -sm, --search-mode | Yes | Specifies the test case lookup algorithm. name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path path: search for test case matching the name within the specified -path. If test case found, update the test case. If test case not found, create a new test case within specified -path | -sm path |
| -px, --proxy | No | HTTP or SOCKS proxy configuration E.g. socks://username:[email protected]:1080 | -px socks://username:[email protected]:1080 |
| -t, --timeout | No | Upload timeout (seconds) | -t 10 |
| -tr, --test-run | No | TestRail test run name | -tr "Chrome Regression Run" |
| -tp, --test-plan | No | TestRail test plan Name | -tp "Shopping Cart Test Plan" |
| -mp, --milestone-path | No | TestRail milestone path | -mp Milestone1/Milestone2 |
| -cf, --case-fields | No | TestRail test case custom fields. The format is [Field label]=[value] pairs, separated with space. E.g. "Case Field 1=value 1" "Case Field 2=value 2" ... | -cf "Case Field 1=value 1" "Case Field 2=value 2" |
| -rf, --result-fields | No | TestRail test result custom fields. The format is [Field label]=[value] pairs, separated with space. E.g. "Result Field 1=value 1" "Result Field 2=value 2" ... | -rf "Result Field 1=value 1" "Result Field 2=value 2" |
| -a, --assign | No | Smart Test Failure Assignment. Comma-separated list of TestRail users (email addresses). Railflow will assign failures based on a round robin algorithm. | -a [email protected],[email protected] |
| -af, --assign-file | No | Smart Test Failure Assignment. File path containing list of TestRail users (email addresses). Note: One user per line | -af /assignees.txt |
| -cn, --config-names | No | TestRail test plan configuration options. Configuration format is: [config_group_name]/[config_name]. E.g. "Operating Systems/Linux" "Browsers/Chrome" ... | -cn "Operating Systems/Linux" "Browsers/Chrome" |
| -cr, --close-run | No | If Railflow should close the corresponding run after uploading test results | -cr |
| -cp, --close-plan | No | If Railflow should close the corresponding plan after uploading test results | -cp |
| -dg, --disable-grouping | No | If Railflow should ignore report structure and just upload all tests into a folder which is set by test-path parameter | -dg |
| -tn, --template-name | No | The name of a template to use in TestRail. If it is not set, 'Test Case (Steps)' or the default one will be used | -tn "Test Case (Steps)" |
| -cst, --case-type | No | The name of a type for cases | -cst other |
| -csp, --case-priority | No | The name of a priority for cases | -csp medium |
| -th, --thread-number | No | The number of concurrent threads for exporting data. Default is 4 | -th 8 |
| -um, --upload-mode | No | Upload mode. Available values are: 0 (default) - create new test cases and do not overwrite existing ones; 1 - create new cases and overwrite existing ones; 2 - do not create new cases and overwrite existing ones; 3 - do not create new cases and do not overwrite existing ones | -um 1 |
| -csf, --case-search-field | No | The name of the case field in TestRail which will be using for searching for existing test cases instead of test case title | -csf "Custom field" |
| -ds, --disable-stats | No | If Railflow should disable collecting usage and error logs | -ds |
| -fqtn, --fully-qualified-test-name | No | If checked, Railflow will use fully qualified test names from the report files for test names in TestRail | -fqtn |
| -us, --untested-status | No | The name of the status to use in TestRail for untested/skipped tests | -us Skipped |
| -ams, --attachment-max-size | No | Maximum size of an attachment. Can be set in Gb, Mb, Kb or b. If the unit is not provided considered as Mb. E.g. 1Gb, 2, 20Kb, 3Mb | -ams 1Gb |
| -atw, --attachment-type-whitelist | No | The comma-separated list of file extensions which are allowed to upload. E.g. json, .html, .xml | -atw "json, .html, .xml" |
| -atb, --attachment-type-blacklist | No | The comma-separated list of file extensions which are not allowed to upload. E.g. json, .html, .xml | -atb "json, .html, .xml" |
| -nr, --no-run | No | Do not create or update Test Run in TestRail | -nr |
| -tfn, --tags-field-name | No | The name of a test case field which will be holding tags from the report file if any. E.g. Cucumber Tags | -tfn "Cucumber Tags" |
| -h, --help | No | Show the help information | -h |
npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txtnpx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txtRailflow Output
Railflow Jenkins Plugin and CLI both generate rich log output for viewing TestRail integration activity.
TeamCity Build Logs
Automatic Test Creation
Automatic Test Plan and Runs
Test Results Details
Automatic Milestones
Smart Failure Assignment
Smart Failure assignment is a very powerful feature of Railflow and allows teams to efficiently and strategically assign test failures to specified team members. Doing this automatically as part of the CI process means that teams don't waste valuable time during the test triage process.
To use Smart Failure Assignment feature, the users need to have Global Role under Project Access.
Example
Consider a TeamCity Selenium Webdriver build is failing with 5 test failures, and 2 users configured in the Smart Test Failure Assignment field.
TeamCity Build Logs
TestRail Results View In this example above, Railflow's Smart Assignment feature filtered all the test failures and distributed them equally across the specified users. This nifty feature greatly eliminates the manual test triage process and saves teams valuable time.
