Observing subset behavior

Sometimes teams want to observe the potential impact and behavior of running subsets in a real environment before they start using them. In other words, they want to measure the subsets' real-world efficacy against the simulation shown in Choosing a subset optimization target.

You can do this using observation mode, which is a special usage mode of launchable record session and, by extension, launchable subset.

To enable observation mode, add --observation to the launchable subset command you added to your pipeline after following Subsetting with the Launchable CLI:

launchable subset --observation <OTHER OPTIONS>

launchable record session \
  --build $BUILD_NAME \
  --observation
  ... [other options]

When observation mode is enabled for a test session, the output of each launchable subset command made against that test session will always include all tests, but the recorded results will be presented separately so you can compare running the subset against running the full suite.

For example, let's imagine you have a test suite with 100 tests that each takes 1 second to run (100 seconds in total):

• By default, if you requested a subset of this test suite with a 30% duration optimization target, the subset output would include 30 tests.

• However, with observation mode enabled, if you requested a subset of this test suite with a 30% duration optimization target, the subset output would include all 100 tests. Launchable will recognize this "full session" as a subset observation session when you record results.

Because you marked the session as an observation session, Launchable can analyze what would have happened if you had actually run a subset of tests, such as whether the subset would have caught a failing session and how much time you could have saved by running only the subset of tests:

✕

You can use this data to compare your real-world results with your Confidence curve (see Choosing a subset optimization target).