Integrating Playwright with Cypress

Cypress (opens in a new tab) is a one of the most popular browser automation tools similar to Selenium WebDriver (opens in a new tab). Railflow Cypress Reporter (opens in a new tab) allows users to map Cypress tests with tests in TestRail and/or to automatically create tests in TestRail. The JSON report generated by the reporter is consumed by the Railflow CLI which is run as part of any CI system like Jenkins, TeamCity, Github, etc.

Railflow for Cypress provides the following capabilities

Test Mapping: Map Cypress tests to existing tests in TestRail
Auto Extraction: Automatically process Cypress JSON test results reports
Custom Fields: Support for TestRail custom fields.
Smart Failure Assignment: Auto assign test failures to TestRail user
TestRail Entities: Update / Create Test Plans, Runs, Milestone, Tests, Results
Screenshot Attachment: Automatically take and attach screenshots
CI Application: Support for Jenkins, TeamCity, Gitlab, Github, etc.

Getting Started

To demonstrate this integration, we have created a sample project: Cypress Demo Project (opens in a new tab). This project contains some tests that pass and fail and produce screenshots. The project uses the Railflow Cypress Reporter (opens in a new tab). This reporter is built on top of the Mocha JUnit Reporter (opens in a new tab) and has access to the same configuration options.

Our examples will store the railflow-reporter configurations in cypress.config.js. You can also pass them via cypress command line like cypress run --reporter railflow-cypress-junit-reporter --reporter-options "attachments=false"

Installation

Install Railflow Cypress Reporter

$ npm install railflow-cypress-junit-reporter --save-dev

Configure the Railflow Cypress Reporter by specifying the results directory using the [hash] keyword and enabling attachments for capturing screenshots in the cypress.config.js file.

cypress.config.js

const { defineConfig } = require('cypress')
 
module.exports = defineConfig({
  reporter: 'railflow-cypress-junit-reporter',
  reporterOptions: {
    mochaFile: 'results/test-results-[hash].xml',
    toConsole: true,
    "attachments": true
  },
})

Test Metadata

While this is not required, many users may need to map a Cypress tests to a specific test in TestRail or may want to speicfy other TestRail Test system attributes like Priority, Test Type, Name, Description or TestRail Custom Fields (opens in a new tab).

Test metabase can be specified at test or the suite level in Cypress as a env.railflow JSON object

Test Suite Level Example:

describe('Login Suite', {
    env: {
        railflow: {
            title: 'Sample Test Suite',
            case_type: 'Automated',
            case_priority: 'High',
            case_fields: ['case field1=value1', 'case field2=value2'],
            result_fields: ['result field1=value1', 'result field2=value2'],
            smart_failure_assignment: ['[email protected]', '[email protected]'],
            testrail_ids: [C1243, C445]
        }
    },
}, () => {
    it('Login Test', () => {
        expect(true).to.equal(true);
    })
})

Test Case Level Example:

describe('Login Suite', () => {
    it('Login Test', {
        env: {
            railflow: {
                title: 'Sample Test Case',
                case_type: 'Automated',
                case_priority: 'High',
                case_fields: ['case field1=value1', 'case field2=value2'],
                result_fields: ['result field1=value1', 'result field2=value2'],
                smart_failure_assignment: ['[email protected]', '[email protected]'],
                testrail_ids: [C1243, C445]
            }
        }
    }, () => {
        expect(true).to.equal(true)
    })
})

Config Options

Railflow provides the following configuration parameters

Parameter Name	Description
title	Name of the test suite or test case
case_type	One of the test case types defined in TestRail, e.g.: `Automated`, `Compatibility`
case_priority	One of the case priority defined in TestRail, e.g.: `Critical`, `High`
case_fields	An array of custom case field label/value pairs in `<field label>=<value>` form, e.g.: ['case field1=value1','case field2=value2']
result_fields	An array of custom result field label/value pairs in `<field label>=<value>` form, e.g.: ['result field1=value1','result field2=value2']
testrail_ids	An array of test case IDs in TestRail to add results to, e.g.: [1,2,3]
smart_failure_assignment	An array of TestRail user email IDS to automatically assign failed test cases to, e.g.: ['[email protected]','[email protected]']

Automatic Screenshots

To enable automatic screenshots for failed tests, add the hook below. Once configured, paths to the screenshot file is added in the junit-results.xml. This path will be extracted by the Railflow CLI later.

Add Cypress hook into <project_root>/cypress/support/index.js file
Enable attachments in report configuration (cypress.json file)

Adding Cypress hook

Cypress.on('test:after:run', function (test, runnable) {
    if (test.state === "failed") {
        const screenshot = `${Cypress.config('screenshotsFolder')}/${Cypress.spec.name}/${runnable.parent.title} -- ${test.title} (failed).png`;        
        if (!test.attachments) {
            test.attachments = [];
        }
        test.attachments.push(screenshot);
    }
});

Run and Enjoy

Now simply run your Cypress project. This will produce an enriched junit-results.xml file.

cypress run

Then call Railflow CLI specifying the junit-results.xml file. This will upload all the tests and test results to TestRail including screenshots.

npx railflow --key ABCDE-12345-FGHIJ-67890 --url https://testrail.your-server.com/ --username testrail-username --password testrail-password --project "Railflow Demo" --test-path section1/section2 --format junit --report-files cypress/results/*.xml --search-mode path

Viewing Results

Test run

Test run in TestRail

Test result details in TestRail

Playwright Robot Framework