Recording test results with the Launchable CLI

Launchable uses your test results from each Test Session to provide features.

After running tests, run the launchable record tests command to record test results to your workspace.

The general structure of this command is:

launchable record tests <BUILD OPTIONS> <TOOL NAME> <PATHS TO REPORT FILES>

• <BUILD OPTIONS> can be one of:

  • --no-build where no build information is captured with the test results. This is the best option for getting started quickly. The examples on this page all use this option.

  • --build <BUILD NAME> which you can use if you recorded a build before running tests (for Predictive Test Selection only). See Recording builds with the Launchable CLI

    • Even if you do plan to use Predictive Test Selection, you can always start recording tests without recording builds and then start recording builds later.

  • --session <SESSION ID STRING> which you can use if you have a complex test session layout. See Managing complex test session layouts

This command varies slightly based on the test runner/build tool you use. See below for specific instructions.

After recording tests, you can see your recorded test sessions them on the Test Sessions page of the Launchable dashboard at app.launchableinc.com. You can click into each session to view details and failed tests.

✕

The CLI will also output a link to view each session's details in the webapp.

Instructions for test runners/build tools

Android Debug Bridge (adb)

Currently, the CLI doesn't have a record tests command for ADB. Use the command for #gradle instead.

Ant

After running tests, point the CLI to your test report file(s):

launchable record tests --no-build ant <PATH TO JUNIT XML>

Then, follow the instructions for Ensuring record tests always runs.

Bazel

After running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build bazel

Then, follow the instructions for Ensuring record tests always runs.

Behave

First, in order to generate reports that Launchable can consume, add the --junit option to your existing behave command:

# run the tests however you normally do
behave --junit

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build behave ./reports/*.xml

Then, follow the instructions for Ensuring record tests always runs.

CTest

First, run your tests with ctest -T test --no-compress-output. These options ensure test results are written to the Testing directory.

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build ctest Testing/**/Test.xml

Then, follow the instructions for Ensuring record tests always runs.

cucumber

Launchable supports JSON and JUnit XML reports generated by cucumber (using -f junit or -f json. Follow the instructions for whichever one you use. JUnit XML is the default.

bundle exec cucumber -f junit -o reports

launchable record tests --no-build cucumber ./reports/**/*.xml

Note: If you receive a warning message such as Cannot find test file Test-feature-example.xml, set the project's root directory path with the --base option:

launchable record tests --no-build --base /example/project cucumber /example/project/reports/**/*.xml

Cypress

After running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build cypress ./report/*.xml

Then, follow the instructions for Ensuring record tests always runs.

GoogleTest

First, configure GoogleTest to produce JUnit-compatible report files. See their documentation for how to do this. You'll end up with a command something like this:

# run the tests however you normally do
./my-test --gtest_output=xml:./report/my-test.xml

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build googletest ./report

Then, follow the instructions for Ensuring record tests always runs.

Go Test

First, in order to generate reports that Launchable can consume, use go-junit-report to generate a JUnit XML file after you run tests:

# install JUnit report formatter
go get -u github.com/jstemmer/go-junit-report

# run the tests however you normally do, then produce a JUnit XML file
go test -v ./... | go-junit-report -set-exit-code > report.xml

Then point the CLI to your test report file to record test results:

launchable record tests --no-build go-test report.xml

Then, follow the instructions for Ensuring record tests always runs.

Gradle

After running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build gradle ./build/test-results/test/

Then, follow the instructions for Ensuring record tests always runs.

Notes:

• By default, Gradle's report files are saved to build/test-results/test/, but that might be different depending on how your Gradle project is configured.

• You can specify multiple directories if you do multi-project build.

• For a large project, a dedicated Gradle task to list up all report directories might be convenient. See the upstream documentation for more details and insights.

• Alternatively, you can specify a glob pattern for directories or individual test report files (this pattern might already be specified in your pipeline script for easy copy-pasting), e.g. gradle **/build/**/TEST-*.xml.

Jest

First, in order to generate reports that Launchable can consume, use jest-junit to generate a JUnit XML file after you run tests.

# install jest-junit reporter
npm install jest-junit --save-dev
# or
yarn add --dev jest-junit

You'll need to configure jest-junit to include file paths in reports.

You can do this using environment variables:

export JEST_JUNIT_CLASSNAME="{classname}"
export JEST_JUNIT_TITLE="{title}"
export JEST_JUNIT_SUITE_NAME="{filepath}"

export JEST_JUNIT_SUITE_NAME="{filepath}"

Then, run jest using jest-junit:

# run tests with jest-junit
jest --ci --reporters=default --reporters=jest-junit

Then, after running tests, point the CLI to your test report files to record test results:

# send test results to Launchable
launchable record tests --no-build jest your-junit.xml

Finally, follow the instructions for Ensuring record tests always runs.

Maven

After running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build maven './**/target/surefire-reports'

Note: The invocation above relies on the CLI to expand GLOBs like **.

Then, follow the instructions for Ensuring record tests always runs.

minitest

First, use minitest-ci to output test results to a file. If you already store your test results on your CI server, it may already be installed.

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build minitest "$CIRCLE_TEST_REPORTS/reports"

Then, follow the instructions for Ensuring record tests always runs.

NUnit Console Runner

After running tests, point the CLI to your test report file(s) to record test results:

launchable record tests --no-build nunit path/to/TestResult.xml

Then, follow the instructions for Ensuring record tests always runs.

prove for Perl

Launchable supports JUnit XML report generated by TAP::Harness::JUnit. Follow the instructions for generating a report.

# You need to pass environment variable JUNIT_NAME_MANGLE=none to generate the JUnit XML report in Launchable's supported format.
export JUNIT_NAME_MANGLE=none 
prove -Ilib --harness TAP::Harness::JUnit -r t

The command will generate a JUnit XML report with name junit_output.xml.

After running tests, point the CLI to your test report file to record test results:

launchable record tests --no-build prove junit_output.xml

Then, follow the instructions for Ensuring record tests always runs.

pytest

Launchable supports JSON and JUnit XML reports generated by pytest (using --junit-xml or --report-log). Follow the instructions for whichever one you use. JUnit XML is the default.

pytest --junit-xml=test-results/results.xml

launchable record tests --no-build pytest ./test-results/

Robot

After running tests with robot, point the CLI to your test report files to record test results:

launchable record tests --no-build robot output.xml

Then, follow the instructions for Ensuring record tests always runs.

RSpec

First, use rspec_junit_formatter to output test results to a file in RSpec. If you already have a CI server storing your test results it may already be installed:

bundle exec rspec --format RspecJunitFormatter --out report/rspec.xml

Then, after running tests, point the CLI to your test report files to record test results:

launchable record tests --no-build rspec ./report

Then, follow the instructions for Ensuring record tests always runs.

Other instructions

If you're not using any of these, see `raw` profile for custom test runners or `file` profile for unsupported test runners.