Integrating Playwright with TestRail

Playwright (opens in a new tab) is a one of the most popular browser automation tools similar to Selenium WebDriver (opens in a new tab). Railflow CLI consumes the Playwright JSON Reports (opens in a new tab) to provide a seamless integration with TestRail (opens in a new tab)

Railflow for Playwright provides the following capabilities

• Test Mapping: Map Playwright tests to existing tests in TestRail
• Auto Extraction: Automatically process Playwright JSON reports
• Custom Fields: Support for TestRail custom fields.
• Smart Failure Assignment: Auto assign test failures to TestRail user
• TestRail Entities: Update / Create Test Plans, Runs, Milestone, Tests, Results
• Test Steps: Ability to capture Playwright steps as TestRail Steps
• Screenshot Attachment: Automatically take and attach screenshots
• CI Application: Support for Jenkins, TeamCity, Gitlab, Github, etc.

Getting Started

To demonstrate this integration, we have created a sample project: Playwright Demo Project (opens in a new tab). This project contains some tests that pass and fail and produce screenshots.

Basic Flow

In this example, we will simply export all the playwright tests to TestRail. Railflow CLI will automatically create tests, milestones, runs, plans.

//playwright.config.ts
const config: PlaywrightTestConfig = {
   reporter: [['json', { outputFile: 'playwright-report/test-results.json' }]],
};

Run tests and produce json report

git clone https://github.com/railflow/playwright-railflow-demo/tree/main
npx playwright test

After test run ends, feed the playwright-report/test-results.json report to Railflow CLI

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path section1/section2 -f playwright -r ./playwright-report/test-results.json -sm path

CLI Reference

Test Mapping

In many organizations tests are first developed in TestRail and then eventually automated and mapped back to TestRail. Railflow uses a convienient annoation testrail.id to help user spcify the TestRail ID for any Playwright tests. This information is automatically added to JSON report and processed accordingly by the Railflow CLI.

import {expect, test} from '@playwright/test';
 
test.describe("Login tests", () => {
    test('some sample tests', async ({page}) => {
        
        //this means that this tests is mapped to two tests in TestRail
        //type contains the name of annotation - `testrail.id` and `description` is TestRail Test ID
        test.info().annotations.push(
            {type: 'testrail.id', description: 'C1825'},
            {type: 'testrail.id', description: 'C1826'}
        )
        //your test logic
    });
});

Test Steps

Playwright provides the ability to add test steps (opens in a new tab) to your tests. These are reported in the json report and can therefore be processed by Railflow CLI and mapped to Steps in TestRail. This is incredibly useful and powerful.

import { test, expect } from '@playwright/test';
 
test('test', async ({ page }) => {
  await test.step('Log in', async () => {
    // ...
  });
 
  await test.step('Outer step', async () => {
    // ...
    // You can nest steps inside each other.
    await test.step('Inner step', async () => {
      // ...
    });
  });
});

Metadata / Custom Fields

Railflow CLI looks out for a variety of annotations in the JSON report. The following can be specifed using Playwright annotations. There are a few reasons why you may find this helpful

• The Playwight title may not be super friend and you want to have a different test title in TestRail
• Sometimes using Railflow CLI for all tests may not be suitable and you want more granular control
• Specify TestRail Custom Fields

import {expect, test} from '@playwright/test';
 
test.describe("Login tests", () => {
 
    test('User logs into home cabinet using incorrect user name and password', async ({page}) => {
        test.info().annotations.push({type: 'testrail.title', description: 'This title was modified by Railflow!'},
            {type: 'testrail.case_type', description: 'Railflow'},
            {type: 'testrail.case_priority', description: 'Critical'},
            {type: 'testrail.case_field', description: "Required Text Field = Hello from Playwright annotations"},
            {type: 'testrail.case_field', description: "Estimate = 10s"},
            {type: 'testrail.result_field', description: "Custom field = Results from Playwright"},
            {type: 'testrail.result_field', description: "Version = 1.0"});
 
            // test logic 
    });
});

Test Results