GitHub Actions

First, follow the instructions on Getting Started to sign up and create a workspace for your test suite.

Then create an API key for your workspace in the Settings area (click the cog ⚙️ icon in the sidebar). This authentication token lets the CLI talk to your Launchable workspace.

Launchable record build and test results action

The Launchable record build and test results action enables you to integrate Launchable into your CI in simple way with less change. This action installs the CLI and runs launchable record build and launchable record test to send data to Launchable so that the test results will be analyzed in Launchable to improve your developer productivity.

Example usage

1name: Test
2
3on:
4  push:
5    branches: [main]
6  pull_request:
7    branches: [main]
8
9env:
10  LAUNCHABLE_TOKEN: ${{ secrets.LAUNCHABLE_TOKEN }}
11  LAUNCHABLE_DEBUG: 1
12  LAUNCHABLE_REPORT_ERROR: 1
13
14jobs:
15  tests:
16    runs-on: ubuntu-latest
17    steps:
18      - uses: actions/checkout@v2
19      - name: Test
20        run: <YOUR TEST COMMAND HERE>
21      - name: Record
22        uses: launchableinc/record-build-and-test-results-action@v1.0.0
23        with:
24          build_name: $GITHUB_RUN_ID
25          test_runner: <YOUR TEST RUNNER HERE>
26          report_path: <PATH TO TEST REPORT XML FILES>
27        if: always()
Test runner `test_runner` value Additional steps?
Ant ant
Bazel bazel
Behave behave Yes
CTest ctest Yes
cucumber cucumber Yes
Cypress cypress
GoogleTest googletest Yes
Go Test go-test Yes
Gradle gradle
Jest jest Yes
Maven maven
minitest minitest Yes
NUnit Console Runner nunit Yes
pytest pytest Yes
Robot robot
RSpec rspec Yes

Instructions for test runners/build tools

Android Debug Bridge (adb)

Currently, the CLI doesn't have a record tests command for ADB. Use the command for #gradle instead.

Ant

No special steps.

Bazel

Behave

First, in order to generate reports that Launchable can consume, add the --junit option to your existing behave command:

# run the tests however you normally do
behave --junit

CTest

First, run your tests with ctest -T test --no-compress-output. These options ensure test results are written to the Testing directory.

cucumber

First, run cucumber with the -f junit option, like this:

bundle exec cucumber -f junit -o reports

(If you use JSON, use the Launchable CLI instead.)

Cypress

GoogleTest

First, configure GoogleTest to produce JUnit compatible report files. See their documentation for how to do this. You'll end up with a command something like this:

# run the tests however you normally do
./my-test --gtest_output=xml:./report/my-test.xml

Go Test

First, in order to generate reports that Launchable can consume, use go-junit-report to generate a JUnit XML file after you run tests:

# install JUnit report formatter
go get -u github.com/jstemmer/go-junit-report

# run the tests however you normally do, then produce a JUnit XML file
go test -v ./... | go-junit-report -set-exit-code > report.xml

Gradle

**/build/**/TEST-*.xml.

Jest

First, in order to generate reports that Launchable can consume, use jest-junit to generate a JUnit XML file after you run tests.

# install jest-junit reporter
npm install jest-junit --save-dev
# or
yarn add --dev jest-junit

You'll need to configure jest-junit to include file paths in reports.

You can do this using environment variables:

Using environment variables
Using package.json

Recommended config:

export JEST_JUNIT_CLASSNAME="{classname}"
export JEST_JUNIT_TITLE="{title}"
export JEST_JUNIT_SUITE_NAME="{filepath}"

Minimum config:

export JEST_JUNIT_SUITE_NAME="{filepath}"

Then, run jest using jest-junit:

# run tests with jest-junit
jest --ci --reporters=default --reporters=jest-junit

Maven

Launchable supports test reports generated using Surefire, the default report plugin for Maven.

'./**/target/surefire-reports'

Note: The invocation above relies on the CLI to expand GLOBs like **.

minitest

First, use minitest-ci to output test results to a file. If you already store your test results on your CI server, it may already be installed.

NUnit Console Runner

Launchable CLI accepts NUnit3 style test report XML files produced by NUnit.

pytest

First, run tests with the --junit-xml option:

pytest --junit-xml=test-results/results.xml

pytest changed its default test report format from xunit1 to xunit2 in version 6. Unfortunately, the new xunit2 format does not include file paths, which Launchable needs.

Thefore, if you are using pytest 6 or newer, you must also specify junit_family=legacy as the report format. See Deprecations and Removals — pytest documentation for instructions.

Robot

output.xml

RSpec

First, use rspec_junit_formatter to output test results to a file in RSpec. If you already have a CI server storing your test results it may already be installed:

bundle exec rspec --format RspecJunitFormatter --out report/rspec.xml