Integrating Katalon with TestRail
Katalon Studio (opens in a new tab) is a low-code test automation platform for testing Web, API, Mobile, and Desktop applications.
Railfow Katalon plugin is a native plugin that seamlessly integrates into Katalon using Katalon extensions framework and exposes context menus and workflows for integrating Katalon with TestRail.
The Katalon Plugin provides the following capabilities
- Map Katalon project to TestRail project
- Dynamic querying of TestRail entity configurations
- Map Katalon execution to TestRail Runs, Plans, and Milestone
- Support for TestRail Test Configurations
- Automatically close Test Runs and Plans
- Dynamic creation of all TestRail entities
- Smart Test Failure Assignment
- Support for querying and mapping to TestRail custom properties
Private Plugin Installation
The Railflow Katalon plugin is a private plugin and not available on the Katalon marketplace. Katalon Plugin architecture allows users to install private plugins by copying the plugin jar file to the Plugins/platform folder in the Katalon project. This is by design and referenced in the Official Katalon plugin documentation (opens in a new tab).
To install the Railflow Katalon plugin, copy the railflow-katalon.jar file to the Plugins/platform folder in your Katalon project. Create the Platform folder if it does not exist.
Reload the plugins using the Profile → Reload Plugins menu (top right corner)
License Activation
Specify the Railflow license key in the Railflow: Plugin Configuration → Railflow Info tab. Once the license is successfully activated, the Railflow menu options (import, export, refresh) will be enabled.
Enable TestRail API
Enable TestRail API in Site Settings
- Administration → Site Settings → API
- Check
Enable APIand thenSave Settings
Plugin Configuration
All of the plugin configuration changes are saved to the Katalon project in a variety of files. It is recommended that you version control the entire project so that other team members can re-use your project's TestRail configurations.
Railflow Katalon plugin allows user to map their Katalon project to a TestRail project and then use the rich plugin capabilities to create various integration workflows.
TestRail Config
The TestRail Config menu allows users to specify their TestRail server connection details. This steps is required and fetches all the projects and project entities from TestRail.
If there are any changes to the TestRail project's setup like new custom fields or adjustments to existing fields, simply refresh the TestRail connection using TestRail Refresh button to fetch the latest project changes.
Project Config
Once the TestRail connection has been established, the next step is to map the Katalon project to a TestRail project. This step is required and allows the plugin to query all the TestRail project configurations and expose them in the plugin's configuration wizard.
Case Fields
In TestRail, test cases can have system (default) fields as well as user defined custom fields (opens in a new tab). These custom fields can have a variety of data types and can be required or optional based on the project setup. Railflow Katalon plugin queries all the projects fields automatically and exposes two ways to configure them.
- Project Level Configuration: All tests within the project will inherit the specified value
- Test Level Configuration: The tests can override the value from the project level. The value at the test level can by set by going to
Test Case > Integrationtab
Railflow plugin's intelligent project configuration wizard will add a trailing * to the case and results field name to indicate a required field.
Result Fields
In TestRail, result fields are also customizable (opens in a new tab) and are similar to case fields. Railflow Katalon plugin queries all the projects result fields automatically and allows you to set the desired values using the project configuration wizard.
Run Config
In TestRail, test results are structured into Test Plans, Runs, Milestones, and Configurations. Railflow Katalon plugin can create plans, runs, milestones, and configurations automatically at run time or you can select existing values from the dropdown controls.
Smart Assignment
Smart Assignment is one of the most popular and handy features in Railflow. If you are like most engineering teams, build failures are spread across a set of people for triage. This failure allocation is a fairly manual process. Smart Assignment feature can go through a list of TestRail users and assign failures in a round robin fashion.
Mapping Test Cases
Teams typically develop and plan tests in one of two main ways. To accomodate both approaches, Railflow Katalon plugin provides two ways to map tests to TestRail.
- Design and review tests in TestRail and then automate them in Katalon
- Design and review directly in Katalon and then reflect these tests in TestRail
Both Import and Export option are designed to Map Katalon tests with TestRail. All mapped tests will expose the TestRail Test ID link in the test's Integration tab. Once a test is Mapped, running the Katalon will automatically export the test results.
Import Test Cases
Import Test Cases will pull down all tests from the specified TestRail project and Test Path, and create the same folder structure in Katalon. This then allows the user to start automating the tests in Katalon. This approach is very common in teams where tests planning happens in TestRail and automation happens at a later time.
Export Test Cases
Some teams prefer to automate directly in Katalon and simply want an easy way to reflect the tests in TestRail's Test Cases section. This approach is very common in Agile teams. Export Test Cases will upload tests from the specified Katalon Test Path into TestRail's Test Cases section.
Katalon Test Results
Once the tests have been Mapped and all project and plugin settings have been configured, you can simply run the Katalon tests as you would normally run them. The Railflow Katalon plugin listens to all the test execution events and export the test results to TestRail in realtime.
