Subsetting with the Launchable CLI

Overview

You use launchable subset to request a subset of tests from Launchable. You'll run this command before your standard test runner command. It generates a list of tests that you can pass into your test runner to run.

See Observing subset behavior if you want to test out subset behavior before running in production. Also, see Zero Input Subsetting for an alternative subsetting interface that is useful in some scenarios.

Options

Read this section first!

launchable subset takes various options:

  1. high-level options

  2. test runner

  3. test runner options

launchable subset <HIGH LEVEL OPTIONS> <TEST RUNNER> <TEST RUNNER OPTIONS>

<TEST RUNNER> is always a string representing the test runner in use, e.g., maven, ant, etc.

For brevity, the examples below do not include all high-level options, so read this section and the Subset section of the CLI reference before you continue. Test runner options are listed in each section.

Required options

Optimization target

At a minimum, you must specify an optimization target option, either

  • --confidence

  • --time

  • --target

See Choosing a subset optimization target for more information.

Build or session identifier

The examples below include the high-level --build <BUILD NAME> option, used for specifying the build for which to request test recommendations. This is the same build that you already created to record tests. The subset command goes in between these.

Before subsetting (simplified)

# build process
launchable record build --name $BUILD_NAME <OPTIONS>
<build process>

# test process
<test process>
launchable record tests --build $BUILD_NAME <OPTIONS>

After subsetting (simplified)

# build process
launchable record build --name $BUILD_NAME <OPTIONS>
<build process>

# test process
launchable subset --build $BUILD_NAME <OPTIONS> # and related commands
<test process>
launchable record tests --build $BUILD_NAME <OPTIONS>

Note that if you generate a test session manually (see Managing complex test session layouts) you'll want to use --session instead of --build.

Instructions for test runners/build tools

If you're not using any of these, use the `file` profile for unsupported test runners, the `raw` profile for custom test runners, or request a plugin.

Android Compatibility Test Suite (CTS)

This profile only supports Zero Input Subsetting. See that page for instructions.

Android Debug Bridge (ADB)

First, you'll request a subset of tests from your entire test suite. Then, you'll pass this list into adb to run.

Requesting a subset of tests

Find the adb command used to run tests in your CI script. These commands will go before that command.

First, duplicate the adb command you normally use to run tests and add the -e log true option. Then, output the result to a text file. For example:

adb shell am instrument <OPTIONS> -e log true com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner > test_list.txt

This command outputs the full list of tests that would normally run (without actually running them) to a file called test_list.txt

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> adb > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt. This file contains a list of test classes formatted for passing into your normal adb command, shown next.

Running a subset of tests

Now you can run only the subset of tests by adding the -e class $(cat launchable-subset.txt) option to your standard adb command, like this:

adb shell am instrument <OPTIONS> -e class $(cat launchable-subset.txt) com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
adb shell am instrument <OPTIONS> com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner

And the flow after:

# generate the complete list of tests in your suite
adb shell am instrument <OPTIONS> -e log true com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner > test_list.txt
# request a subset from the full list
cat test_list.txt | launchable subset  --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> adb > launchable-subset.txt
# run the results of the subset request
adb shell am instrument <OPTIONS> -e class $(cat launchable-subset.txt) com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner

Ant

First, you'll request a subset of tests from your complete suite. Then, you'll pass this list into your build.xml file to limit what Ant runs.

Requesting a subset of tests

First, find the ant command used to run tests in your CI script.

Before that command, add the launchable subset command to request a subset of tests from your full test suite:

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> ant <PATH TO SOURCE> > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • Set <PATH TO SOURCE> to the path(s) containing your test files. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this whole list into a subset and a remainder.

This creates a file called launchable-subset.txt. This file contains a list of test classes formatted for passing into your build.xml file, shown next.

Running a subset of tests

Separately, update your build.xml file to use launchable-subset.txt:

1<project>
2  …
3  <target name="check-launchable">
4    <available file="launchable-subset.txt" property="launchable"/>
5  </target>
6
7  <target name="junit" depends="jar,check-launchable">
8    <mkdir dir="${report.dir}"/>
9    <junit printsummary="yes">
10      <classpath>
11          <path refid="classpath"/>
12          <path refid="application"/>
13      </classpath>
14
15      <formatter type="xml"/>
16
17      <batchtest fork="yes" todir="${report.dir}">
18        <fileset dir="${src.dir}" >
19          <includesfile name="launchable-subset.txt" if="${launchable}" />
20          <include name="**/*Test.java" unless="${launchable}" />
21        </fileset>
22      </batchtest>
23    </junit>
24  </target>
25  …
26</project>

Finally, you run tests command as normal, such as:

ant junit <OPTIONS>

Bazel

First, you'll request a subset of tests from your entire test suite. Then, you'll pass this list to Bazel to run.

Requesting a subset of tests

Find the bazel command used to run tests in your CI script. These commands will go before that command.

First, run bazel query and output the result to a text file. For example:

bazel query 'tests(//...)' > test_list.txt

This command outputs the complete list of test targets that typically run (without running them) to a file called test_list.txt. The subset service will divide this list into a subset and a remainder.

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> bazel > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt that you can pass into Bazel.

Running a subset of tests

AppendYour the list of tests to run to your existing command, such as:

bazel test $(cat launchable-subset.txt)

Summary

In summary, here's the flow before:

# your standard command to run tests looks something like this
bazel test

And the flow after:

# generate the full list
bazel query 'tests(//...)' > test_list.txt
# request a subset
cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> bazel > launchable-subset.txt
# run the results of the subset request
bazel test $(cat launchable-subset.txt)

Behave

First, you'll request a subset of tests from your entire test suite. Then, you'll pass this list to Behave to run.

Requesting a subset of tests

Find the behave command used to run tests in your CI script. These commands will go before that command.

First, run find ./features/ (or a similar command for your environment) and output the result to a text file. For example:

find ./features/ > test_list.txt

This command writes the list of test files that typically run (without actually running them) to test_list.txt. The subset service will divide this list into a subset and a remainder.

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> behave > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt that you can pass into Behave.

Running a subset of tests

To run a subset, run behave with the -i option and pass in the subset list. For example:

behave <OPTIONS> -i "$(cat launchable-subset.txt)"

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
behave <OPTIONS>

And the flow after:

# generate the full list
find ./features/ > test_list.txt
# request a subset
cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> behave > launchable-subset.txt
# run the results of the subset request
behave <OPTIONS> -i "$(cat launchable-subset.txt)"

CTest

First, you'll request a subset of tests from your entire test suite. Then, you'll pass this list into CTest to run.

Requesting a subset of tests

Find the ctest command used to run tests in your CI script. These commands will go before that command.

First, run ctest with the --show-only option and output the result to a JSON file. For example:

ctest <OPTIONS> --show-only=json-v1 > test_list.json

This command creates the complete list of test files that typically run (without actually running them) to a file called test_list.json. The subset service will divide this list into a subset and a remainder list.

Next, pass the file you just created into launchable subset to request a subset from the full list.

launchable subset --build <BUILD NAME> --confidence <TARGET> ctest --output-regex-files --output-regex-files-dir=subsets test_list.json

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • The --output-regex-files instructs CLI to write the regular expression for

    the subset tests into the directory specified in --output-regex-files-dir.

This creates files under the subsets directory. subset_N are the files that contain regular expressions of the chosen subset of tests. If you use the --rest option, rest_N will contain the non-chosen tests.

Running a subset of tests

Then, run ctest for each subset output file:

for file in subset/subset_*; do
  ctest <OPTIONS> -T test --no-compress-output -R "$(cat "$file")"
done

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
ctest <OPTIONS> -T test --no-compress-output

And the flow after:

# generate the full list that would normally run
ctest --show-only=json-v1 > test_list.json
# request a subset
launchable subset --build <BUILD NAME> --confidence <TARGET> ctest --output-regex-files --output-regex-files-dir=subsets test_list.json
# run the results of the subset request
for file in subset/subset_*; do
  ctest <OPTIONS> -T test --no-compress-output -R "$(cat "$file")"
done

cucumber

First, you'll request a subset of tests from your entire test suite. Then, you'll pass this list to cucumber to run.

Requesting a subset of tests

First, find the bundle exec cucumber command used to run tests in your CI script.

Before that command, add the launchable subset command to request a subset of tests from your full test suite:

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --base $(pwd) cucumber <PATH TO .feature FILES> > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • Don't forget the --base $(pwd) option (or equivalent) before cucumber.

  • Set <PATH TO .feature FILES> to the glob expression representing your .feature files, e.g., features/**/*.feature. The CLI will look in those path(s) and generate the complete list of tests that would typically run. The subset service divides this list into a subset and a remainder list.

This creates a file called launchable-subset.txt. This file contains a list of test files formatted for passing into cucumber, shown next.

Running a subset of tests

Append the subset list to your bundle exec cucumber command to run a subset. For example:

bundle exec cucumber -f junit -o reports <OPTIONS> $(cat launchable-subset.txt)

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
bundle exec cucumber -f junit -o reports <OPTIONS>

And the flow after:

# request a subset from all features that would typically run
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION>     cucumber <PATH TO .feature FILES> > launchable-subset.txt
# run the results of the subset request
bundle exec cucumber -f junit -o reports <OPTIONS> $(cat launchable-subset.txt)

Cypress

Find the cypress run command used to run tests in your CI script. These commands will go before that command.

First, run find ./cypress/integration -type f (or a similar command for your platform) and output the result to a text file. For example:

find ./cypress/integration -type f > test_list.txt

This command writes the complete list of test files that typically run (without actually running them) to test_list.txt. The subset service will divide this list into a subset and a remainder list.

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> cypress > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt that you can pass into Cypress.

Running a subset of tests

To run a subset, use the --spec option with the subset list text file. For example:

cypress run --reporter junit <OPTIONS> --spec "$(cat launchable-subset.txt)"

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
cypress run --reporter junit <OPTIONS>

And the flow after:

# generate the complete list that would typically run
find ./cypress/integration -type f > test_list.txt
# request a subset from all features that would typically run
cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> cypress > launchable-subset.txt
# run the results of the subset request
cypress run --reporter junit <OPTIONS> --spec "$(cat launchable-subset.txt)"

dotnet test

This profile only supports Zero Input Subsetting. See that page for instructions.

GoogleTest

Find the GoogleTest command used to run tests in your CI script. These commands will go before that command.

First, invoke GoogleTest with the --gtest_list_tests option and output the result to a text file. For example:

./my-test <OPTIONS> --gtest_list_tests > test_list.txt

This command outputs the complete list of tests that normally run (without running them) to a file called test_list.txt. The subset service will divide this list into a subset and a remainder list.

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> googletest > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt that you can pass into GoogleTest.

Running a subset of tests

Add the --gtest_filter option to your existing command, such as:

./my-test <OPTIONS> --gtest_filter="$(cat launchable-subset.txt)"

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
./my-test <OPTIONS>

And the flow after:

# generate the full list
./my-test <OPTIONS> --gtest_list_tests > test_list.txt
# request a subset
cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> googletest > launchable-subset.txt
# run the results of the subset request
./my-test <OPTIONS> --gtest_filter="$(cat launchable-subset.txt)"

Go Test

Find the go test command used to run tests in your CI script. These commands will go before that command.

First, duplicate the go test command you normally use to run tests and add the -list option. Then, output the result to a text file. For example:

go test <OPTIONS> -list="Test|Example" ./... > test_list.txt

This command outputs the complete list of tests that normally run (without running them) to a file called test_list.txt. The subset service will divide this full list into a subset and a remainder.

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> go-test > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt that you can pass into go test.

Running a subset of tests

Add the -run option to your existing command, such as:

go test <OPTIONS> -run $(cat launchable-subset.txt) ./... | go-junit-report > report.xml

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
go test <OPTIONS> ./...

And the flow after:

# generate the full list
go test <OPTIONS> -list="Test|Example" . ./... > test_list.txt
# request a subset
cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> go-test > launchable-subset.txt
# run the results of the subset request
go test <OPTIONS> -run $(cat launchable-subset.txt) ./... | go-junit-report > report.xml

Gradle

First, you'll request a subset of tests from your full test suite. Then, you'll pass this list to Gradle.

Requesting a subset of tests

First, find the gradle command used to run tests in your CI script.

Before that command, add the launchable subset command to request a subset of tests from your full test suite:

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> gradle <PATH TO SOURCE> > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • Set <PATH TO SOURCE> to the path(s) containing your test files, e.g., project1/src/test/java project2/src/test/java. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this full list into a subset and a remainder.

This creates a file called launchable-subset.txt. This file contains a list of test classes formatted for passing into Gradle, like this:

--tests MyTestClass1 --tests MyTestClass2 ...

Running a subset of tests

Then simply pass this file into your existing command, like shown below.

Gradle
Gradle plugin for Android
gradle test <OPTIONS> $(cat launchable-subset.txt)
# equivalent to gradle test <OPTIONS> --tests MyTestClass1 --tests MyTestClass2 ...

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
gradle test <OPTIONS>

And the flow after:

# request a subset from all tests
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> gradle <PATH TO SOURCE> > launchable-subset.txt
# run the results of the subset request
gradle test <OPTIONS> $(cat launchable-subset.txt)

Gradle + TestNG

First, you'll request a subset of tests from your full test suite. Then, you'll pass this list to Gradle.

Requesting a subset of tests

First, find the gradle command used to run tests in your CI script.

Before that command, add the launchable subset command to request a subset of tests from your full test suite:

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> gradle --bare <PATH TO SOURCE> > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • Set <PATH TO SOURCE> to the path(s) containing your test files, e.g. project1/src/test/java project2/src/test/java. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this full list into a subset and a remainder.

  • Don't forget the --bare option after gradle!

This creates a file called launchable-subset.txt. This file contains a list of test classes formatted for passing into Gradle, like this:

com.example.FooTest
com.example.BarTest
...

Running a subset of tests

First, you need to add a dependency declaration to build.gradle so that the right subset of tests get executed when TestNG runs:

dependencies {
    ...
    testRuntime 'com.launchableinc:launchable-testng:1.3.0'
}

Then simply export the subset file path as an environment variable before you run gradle test, like shown below.

Gradle
Gradle plugin for Android
export LAUNCHABLE_SUBSET_FILE_PATH=$PWD/launchable-subset.txt
gradle test <OPTIONS>

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
gradle test <OPTIONS>

And the flow after:

# request a subset from all tests
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> gradle --bare <PATH TO SOURCE> > launchable-subset.txt
# run the results of the subset request using the `launchable-testng` plugin
export LAUNCHABLE_SUBSET_FILE_PATH=$PWD/launchable-subset.txt
gradle test <OPTIONS>

Jest

Find the jest command used to run tests in your CI script. These commands will go before that command.

First, duplicate the jest command you normally use to run tests and add the --listTests option. Then, output the result to a text file. For example:

jest <OPTIONS> --listTests > test_list.txt

This command creates the full list of test files that would normally run (without actually running them) to a file called test_list.txt. The subset service will divide this full list into a subset and a remainder.

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --base $(pwd) cypress > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • Don't forget the --base $(pwd) option before cypress.

This creates a file called launchable-subset.txt that you can pass into Jest.

Running a subset of tests

To run the subset, include the subset list after jest. For example:

jest <OPTIONS> $(cat launchable-subset.txt)

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
jest <OPTIONS>

And the flow after:

# generate the full list that would normally run
jest <OPTIONS> --listTests > test_list.txt
# request a subset from all features that would normally run
cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --base $(pwd) jest > launchable-subset.txt
# run the results of the subset request
jest <OPTIONS> $(cat launchable-subset.txt)

Maven

Find the mvn test command used to run tests in your CI script. These commands will go before that command.

First, duplicate the mvn test command you normally use to run tests, but change test to test-compile. For example:

mvn test-compile <OPTIONS>

This command creates .lst files that list the test classes that would normally run (without running them). The subset service will combine these and divide this full list into a subset and a remainder.

Next, run launchable subset to request a subset from the full list.

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> maven --test-compile-created-file <(find . -path '*/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst' -exec cat {} \;) > launchable-subset.txt

  • See #options for how to set <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • The <(find... section combines the .lst files across your projects into a single file for processing. You might need to change this for your platform.

This creates a file called launchable-subset.txt that you can pass into Maven.

Running a subset of tests

To run the subset, use the -Dsurefire.includesFile option. For example:

mvn test <OPTIONS> -Dsurefire.includesFile=$PWD/launchable-subset.txt

Summary

In summary, here's the flow before:

# your normal command to run tests looks something like this
mvn test <OPTIONS>

And the flow after:

# generate the full list(s) that would normally run
mvn test-compile <OPTIONS>
# request a subset from all features that would normally run
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> maven --test-compile-created-file <(find . -path '*/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst' -exec cat {} \;) > launchable-subset.txt
# run the results of the subset request
mvn test <OPTIONS> -Dsurefire.includesFile=$PWD/launchable-subset.txt

Maven + TestNG

Find the mvn test command used to run tests in your CI script. These commands will go before that command.

First, duplicate the mvn test command you normally use to run tests, but change test to test-compile. For example:

mvn test-compile <OPTIONS>

This command creates .lst files that list the test classes that would normally run (without running them). The subset service will combine these and divide this full list into a subset and a remainder.

Next, run launchable subset to request a subset from the full list.

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> maven --test-compile-created-file <(find . -path '*/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst' -exec cat {} \;) > launchable-subset.txt

  • See #options for how to set <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • The <(find... section combines the .lst files across your projects into a single file for processing.

This creates a file called launchable-subset.txt that you can pass into Maven.

Running a subset of tests

First, modify your pom.xml so that it includes Launchable TestNG integration as a test scope dependency:

<dependency>
  <groupId>com.launchableinc</groupId>
  <artifactId>launchable-testng</artifactId>
  <version>1.3.0</version>
  <scope>test</scope>
</dependency>

Then simply export the subset file path as an environment variable before you run mvn test, like shown below.

export LAUNCHABLE_SUBSET_FILE_PATH=$PWD/launchable-subset.txt
mvn test <OPTIONS>

Summary

In summary, here's the flow before:

# your normal command to run tests looks something like this
mvn test <OPTIONS>

And the flow after:

# generate the full list of tests
mvn test-compile <OPTIONS>
# request a subset from all tests
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> maven --test-compile-created-file <(find . -path '*/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst' -exec cat {} \;) > launchable-subset.txt
# run the results of the subset request using the `launchable-testng` plugin
export LAUNCHABLE_SUBSET_FILE_PATH=$PWD/launchable-subset.txt
mvn test <OPTIONS>

minitest

First, you'll request a subset of tests from your full test suite. Then, you'll pass this list into minitest to run.

Requesting a subset of tests

First, find the bundle exec rails test command used to run tests in your CI script.

Before that command, add the launchable subset command to request a subset of tests from your full test suite:

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> minitest <PATH TO .rb FILES> > launchable-subset.txt

  • See #options for how to set <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • Set <PATH TO .rb FILES> to the glob expression representing your .rb test files, e.g. test/**/*.rb. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this full list into a subset and a remainder.

This creates a file called launchable-subset.txt. This file contains a list of tests formatted for passing into minitest.

Running a subset of tests

To run a subset, pass the subset list into bundle exec rails test. For example:

bundle exec rails test <OPTIONS> $(cat launchable-subset.txt)

Summary

In summary, here's the flow before:

# your normal command to run tests looks something like this
bundle exec rails test <OPTIONS>

And the flow after:

# request a subset of your existing test suite
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> minitest <PATH TO .rb FILES> > launchable-subset.txt
# run the results of the subset request
bundle exec rails test <OPTIONS> $(cat launchable-subset.txt)

NUnit Console Runner

First, you'll request a subset of tests from your full test suite. Then, you'll pass this list into nunit3-console to run.

Requesting a subset of tests

Find the nunit3-console command used to run tests in your CI script. These commands will go before that command.

First, duplicate the nunit3-console command you normally use to run tests, and add the --explore option. For example:

nunit3-console <OPTIONS> --explore=test_list.xml path/to/myassembly.dll

This command writes the full list of tests that normally run (without running them) to  test_list.xml.

Next, pass the file you just created into launchable subset to request a subset from the full list.

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> nunit test_list.xml > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt. This file contains a list of test classes formatted for passing into your normal adb command, shown next.

Note: If you want to subset tests across multiple DLLs (for example, if multiple DLLs are combined into a logical 'suite'), you can run nunit3-console --explore... once for each DLL, then pass all the files into launchable subset, such as:

nunit3-console <OPTIONS> --explore=myassembly1.xml path/to/myassembly1.dll
nunit3-console <OPTIONS> --explore=myassembly2.xml path/to/myassembly2.dll
nunit3-console <OPTIONS> --explore=myassembly3.xml path/to/myassembly3.dll

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> myassembly1.xml myassembly2.xml myassembly3.xml > launchable-subset.txt

Running a subset of tests

Now you can run only the subset of tests by adding the --testlist option to your normal nunit3-console command, like this:

nunit3-console <OPTIONS> --testlist=launchable-subset.txt path/to/myassembly.dll [path/to/myassembly2.dll] [path/to/myassembly3.dll]

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
nunit3-console <OPTIONS> path/to/myassembly.dll

And the flow after:

# generate the full list of tests in your suite
nunit3-console <OPTIONS> --explore=test_list.xml path/to/myassembly.dll
# request a subset from the full list
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> nunit test_list.xml > launchable-subset.txt
# run the results of the subset request
nunit3-console <OPTIONS> --testlist=launchable-subset.txt path/to/myassembly.dll [path/to/myassembly2.dll] [path/to/myassembly3.dll]

prove for Perl

Find the prove command used to run tests in your CI script. These commands will go before that command.

First, pipe the test files you have into launchable subset to request a subset from the full list.

# Assuming your test directory is `./t`.
find ./t -name '*.t' | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> prove > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt. This file contains a list of tests formatted for passing into your normal prove command, shown next.

Running a subset of tests

Now you can run only the subset of tests by passing the launchable-subset.txt file into prove, like this:

# You must pass the environment variable JUNIT_NAME_MANGLE=none to generate the JUnit XML report in Launchable's supported format.
export JUNIT_NAME_MANGLE=none
prove <OPTIONS> -Ilib --harness TAP::Harness::JUnit -r $(cat launchable-subset.txt)

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
prove <OPTIONS> -Ilib --harness TAP::Harness::JUnit -r t

And the flow after:

# request a subset from the full list
find ./t -name '*.t' | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> prove > launchable-subset.txt
# run the results of the subset request
export JUNIT_NAME_MANGLE=none
prove <OPTIONS> -Ilib --harness TAP::Harness::JUnit -r $(cat launchable-subset.txt)

Playwright

First, you'll request a subset of tests from your entire test suite. Then, you'll pass this list to Playwright to run.

Requesting a subset of tests

Find the playwright test command used to run tests in your CI script.

First, list the result in a text file. For example:

find tests/*.spec.ts > test_list.txt

This command outputs the complete list of test targets that typically run (without running them) to a file called test_list.txt. The subset service will divide this list into a subset and a remainder.

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> playwright > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

Running a subset of tests

Append your the list of tests to run to your existing command, such as:

playwright test $(cat launchable-subset.txt)

Summary

In summary, here's the flow before:

playwright test ./tests

And the flow after:

# generate the test list
find ./tests/*.spec.ts > test_list.txt
# request a subset
cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> playwright > launchable-subset.txt
# run the results of the subset request
playwright test $(cat launchable-subset.txt)

pytest

Find the pytest command used to run tests in your CI script. These commands will go before that command.

First, duplicate the pytest command you normally use to run tests and add the --collect-only and -q options. Then output that to a file. For example:

pytest <OPTIONS> --collect-only -q > test_list.txt

This command writes the full list of tests that normally run (without running them) to test_list.txt.

Next, pipe the file you just created into launchable subset to request a subset from the full list.

cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> pytest > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt. This file contains a list of tests formatted for passing into your normal pytest command, shown next.

Running a subset of tests

Now you can run only the subset of tests by passing the launchable-subset.txt file into pytest, like this:

pytest <OPTIONS> --junit-xml=test-results/subset.xml $(cat launchable-subset.txt)

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
pytest <OPTIONS> --junit-xml=test-results/subset.xml

And the flow after:

# generate the full list of tests in your suite
pytest <OPTIONS> --collect-only -q > test_list.txt
# request a subset from the full list
cat test_list.txt | launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> pytest > launchable-subset.txt
# run the results of the subset request
pytest <OPTIONS> --junit-xml=test-results/subset.xml $(cat launchable-subset.txt)

Robot

Find the robot command used to run tests in your CI script. These commands will go before that command.

First, duplicate the robot command you normally use to run tests, and add the --dryrun and -o options. For example:

robot <OPTIONS> --dryrun -o test_list.xml

This command writes the full list of tests that normally run (without running them) to test_list.xml.

Next, pass the file you just created into launchable subset to request a subset from the full list.

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> robot test_list.xml > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-subset.txt. This file contains a list of tests formatted for passing into your normal pytest command, shown next.

Running a subset of tests

Now you can run only the subset of tests by passing the launchable-subset.txt file into robot, like this:

robot <OPTIONS> $(cat launchable-subset.txt) .

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
robot <OPTIONS>

And the flow after:

# generate the full list of tests in your suite
robot <OPTIONS> --dryrun -o test_list.xml
# request a subset from the full list
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> robot test_list.xml > launchable-subset.txt
# run the results of the subset request
robot <OPTIONS> $(cat launchable-subset.txt) .

RSpec

First, you'll request a subset of tests from your full test suite. Then, you'll pass this list into minitest to run.

Requesting a subset of tests

First, find the bundle exec rspec command used to run tests in your CI script.

Before that command, add the launchable subset command to request a subset of tests from your full test suite:

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> rspec <PATH(S) TO .rb FILES> > launchable-subset.txt

  • See #options for setting <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

  • Set <PATH TO .rb FILES> to the glob expression representing your .rb test files, e.g., spec/**/*_spec.rb. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this full list into a subset and a remainder.

This creates a file called launchable-subset.txt. This file contains a list of tests formatted for passing into RSpec.

Running a subset of tests

To run a subset, pass the subset list into bundle exec rails test. For example:

bundle exec rspec $(cat launchable-subset.txt) --format d --format RspecJunitFormatter --out rspec.xml <OPTIONS>

Summary

In summary, here's the flow before:

# Your normal command to run tests looks something like this
bundle exec rspec --format RspecJunitFormatter --out report/rspec.xml <OPTIONS>

And the flow after:

# request a subset of your existing test suite
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> rspec <PATH TO .rb FILES> > launchable-subset.txt
# run the results of the subset request
bundle exec rspec $(cat launchable-subset.txt) --format d --format RspecJunitFormatter --out rspec.xml <OPTIONS>

Other instructions

If you're not using any of these, see `raw` profile for custom test runners or `file` profile for unsupported test runners.

Checking for integration issues

Coming soon!

Last modified ago