Recording test results with the Launchable CLI

Launchable uses your test results from each Test Session to provide features.

After running tests, run the launchable record tests command to record test results in your workspace.

The general structure of this command is as follows:

launchable record tests <BUILD OPTIONS> <TOOL NAME> <PATHS TO REPORT FILES>

• <BUILD OPTIONS> can be one of the following:

  • --no-build, where no build information is captured with the test results. This is the best option for getting started quickly. The examples on this page all use this option.

  • --build <BUILD NAME>, which you can use if you recorded a build before running tests (for Predictive Test Selection only). See Recording builds with the Launchable CLI

    • Even if you plan to use Predictive Test Selection, you can always start recording tests without recording builds and then recording builds later.

  • --session <SESSION ID STRING>, which you can use if you have a complex test session layout. See Managing complex test session layouts

This command varies slightly based on your test runner/build tool. See below for specific instructions.

After recording tests, you can see your recorded test sessions on the Test Sessions page of the Launchable dashboard at app.launchableinc.com. You can click into each session to view details and failed tests.

✕

The CLI will also output a link to view each session's details in the web app.

Instructions for test runners/build tools

Android Compatibility Test Suite (CTS)

After running tests, point the CLI to your test report file(s):

launchable record tests --no build cts android-cts/results/**/*.xml

Android Debug Bridge (ADB)

Currently, the CLI doesn't have a record tests command for ADB. Use the command for #gradle instead.

Ant

After running tests, point the CLI to your test report file(s):

launchable record tests --no-build ant <PATH(S) TO JUNIT XML FILE(S)>

Then, follow the instructions for Ensuring record tests always runs.

Bazel

After running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build bazel

Then, follow the instructions for Ensuring record tests always runs.

Behave

First, to generate reports that Launchable can consume, add the --junit option to your existing behave command:

# run the tests however you usually do
behave --junit

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build behave ./reports/*.xml

Then, follow the instructions for Ensuring record tests always runs.

CTest

First, run your tests with ctest -T test --no-compress-output. These options ensure test results are written to the Testing directory.

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build ctest "Testing/**/Test.xml"

Then, follow the instructions for Ensuring record tests always runs.

cucumber

Launchable supports JSON and JUnit XML reports generated by cucumber (using -f junit or -f json. Follow the instructions for whichever one you use. JUnit XML is the default.

bundle exec cucumber -f junit -o reports

launchable record tests --no-build cucumber "./reports/**/*.xml"

Note: If you receive a warning message such as Cannot find test file Test-feature-example.xml, set the project's root directory path with the --base option:

launchable record tests --no-build --base /example/project cucumber "/example/project/reports/**/*.xml"

Cypress

After running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build cypress ./report/*.xml

Then, follow the instructions for Ensuring record tests always runs.

dotnet test

First, configure dotnet test to product NUnit report files. Install the NunitXml.TestLogger package from NuGet via your preferred method, then the -l option to your dotnet test command to enable the logger:

dotnet test ... -l nunit

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build dotnet "**/*.xml"

Then, follow the instructions for Ensuring record tests always runs.

GoogleTest

First, configure GoogleTest to produce JUnit-compatible report files. See their documentation for how to do this. You'll end up with a command something like this:

# run the tests however you normally do
./my-test --gtest_output=xml:./report/my-test.xml

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build googletest ./report

Then, follow the instructions for Ensuring record tests always runs.

Go Test

First, to generate reports that Launchable can consume, use go-junit-report to generate a JUnit XML file after you run tests:

# install JUnit report formatter
go get -u github.com/jstemmer/go-junit-report

# run the tests however you usually do, then produce a JUnit XML file
go test -v ./... | go-junit-report -set-exit-code > report.xml

Then point the CLI to your test report file to record test results:

launchable record tests --no-build go-test report.xml

Then, follow the instructions for Ensuring record tests always runs.

Gradle

After running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build gradle ./build/test-results/test/

Then, follow the instructions for Ensuring record tests always runs.

Notes:

• By default, Gradle's report files are saved to build/test-results/test/, but that might differ depending on how your Gradle project is configured.

• You can specify multiple directories if you do a multi-project build.

• A dedicated Gradle task to list all report directories might be convenient for a large project. See the upstream documentation for more details and insights.

• Alternatively, you can specify a glob pattern for directories or individual test report files (this pattern might already be set in your pipeline script for easy copy-pasting), e.g., gradle **/build/**/TEST-*.xml.

Jest

First, to generate reports that Launchable can consume, use jest-junit to generate a JUnit XML file after you run tests.

# install jest-junit reporter
npm install jest-junit --save-dev
# or
yarn add --dev jest-junit

You'll need to configure jest-junit to include file paths in reports.

You can do this using environment variables:

export JEST_JUNIT_CLASSNAME="{classname}"
export JEST_JUNIT_TITLE="{title}"
export JEST_JUNIT_SUITE_NAME="{filepath}"

export JEST_JUNIT_SUITE_NAME="{filepath}"

Then, run jest using jest-junit:

# run tests with jest-junit
jest --ci --reporters=default --reporters=jest-junit

Then, after running tests, point the CLI to your test report files to record test results:

# send test results to Launchable
launchable record tests --no-build jest your-junit.xml

Finally, follow the instructions for Ensuring record tests always runs.

Maven

After running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build maven "./**/target/surefire-reports"

You can pass multiple directories/patterns, e.g. launchable record tests --no-build maven '<pattern/directory 1>' '<pattern/directory 2>'.

Then, follow the instructions for Ensuring record tests always runs.

minitest

First, use minitest-ci to output test results to a file. It may already be installed if you store your test results on your CI server.

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build minitest "$CIRCLE_TEST_REPORTS/reports"

Then, follow the instructions for Ensuring record tests always runs.

NUnit Console Runner

After running tests, point the CLI to your test report file(s) to record test results:

launchable record tests --no-build nunit path/to/TestResult.xml

Then, follow the instructions for Ensuring record tests always runs.

prove for Perl

Launchable supports JUnit XML report generated by TAP::Harness::JUnit. Follow the instructions for generating a report.

# You must pass the environment variable JUNIT_NAME_MANGLE=none to generate the JUnit XML report in Launchable's supported format.
export JUNIT_NAME_MANGLE=none 
prove -Ilib --harness TAP::Harness::JUnit -r t

The command will generate a JUnit XML report with the name junit_output.xml.

After running tests, point the CLI to your test report file to record test results:

launchable record tests --no-build prove junit_output.xml

Then, follow the instructions for Ensuring record tests always runs.

pytest

Launchable supports JSON and JUnit XML reports generated by pytest (using --junit-xml or --report-log). Follow the instructions for whichever one you use. JUnit XML is the default.

pytest --junit-xml=test-results/results.xml

launchable record tests --no-build pytest ./test-results/

Robot

After running tests with robot, point the CLI to your test report files to record test results:

launchable record tests --no-build robot output.xml

Then, follow the instructions for Ensuring record tests always runs.

RSpec

First, use rspec_junit_formatter to output test results to a file in RSpec. If you already have a CI server storing your test results, it may already be installed:

bundle exec rspec --format RspecJunitFormatter --out report/rspec.xml

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build rspec ./report

Then, follow the instructions for Ensuring record tests always runs.

Other instructions

If you're not using any of these, see `raw` profile for custom test runners or `file` profile for unsupported test runners.