pytest (Integration)
Details about how Launchable integrates with pytest.
This is a reference page. See Getting Started, Sending data to Launchable, and Predictive Test Selection for more comprehensive usage guidelines.
Launchable interfaces with pytest via
the Launchable pytest plugin
the Launchable CLI
Launchable pytest plugin
We offer a new way to integrate Launchable, a native pytest plugin.
Installing the plugin
The Launchable pytest plugin is a Python3 package that you can install from PyPI.
{% hint style="warning" %} The plugin requires Python 3.7+, Pytest 4.2.0+, and Java 8+. {% endhint %}
If you use Pipenv, you can install the plugin into your repository:
pipenv install --dev pytest-launchable
Or, you can install the CLI in your CI pipeline by adding this to the part of your CI script where you install dependencies:
pip3 install pytest-launchable
You don't need to install Lanchable CLI separately because the plugin automatically installs the CLI and uses it internally.
Setting your API key
First, create an API key for your workspace at app.launchableinc.com. This authentication token allows the pytest plugin to talk to Launchable.
Then, make this API key available as the LAUNCHABLE_TOKEN
environment variable in your CI process. How you do this depends on your CI system:
CI system | Docs |
---|---|
Azure DevOps Pipelines | Set secret variables |
Bitbucket Pipelines | Variables and secrets |
CircleCI | Using Environment Variables |
GitHub Actions | How to configure a secret |
GitLab CI | GitLab CI/CD environment variables |
GoCD | Setting variables on an environment |
Jenkins |
(Create a global "secret text" to use in your job) |
Travis CI | Environment Variables |
Generate a config file
launchable-config
is a command-line tool to generate and validate configuration files. The Launchable pytest plugin uses this config.
First, generate a new config file:
# via pipenvpipenv run launchable-config --create# via piplaunchable-config --create
This generates a template .launchable.d/config.yml
file in the current directory that looks like this:
# Launchable test session configuration file# See https://docs.launchableinc.com/resources/cli-reference for detailed usage of these options#schema-version: 1.0build-name: commit_hashrecord-build:# Put your git repository location heresource: .max_days: 30record-session:subset:# mode is subset, subset-and-rest, or record-onlymode: subset# you must specify one of target/confidence/time# examples:# target: 30% # Create a variable time-based subset of the given percentage. (0%-100%)# confidence: 30% # Create a confidence-based subset of the given percentage. (0%-100%)# time: 30m # Create a fixed time-based subset. Select the best set of tests that run within the given time bound. (e.g. 10m for 10 minutes, 2h30m for 2.5 hours, 1w3d for 7+3=10 days. )target: 30%record-tests:# The test results are placed here in JUnit XML formatresult_dir: launchable-test-result
You can then edit the config file per the directions below.
Recording test results (pytest plugin)
Update your config file
In .launchable.d/config.yml
:
Check that the
source
option in therecord-build
section points to your Git repository (the default is.
, the current directory).Check that the
mode
option in thesubset
section is set torecord-only
Verify your config file
Verify the contents of the .launchable.d/config.yml
file:
# via pipenvpipenv run launchable-config --verify# via piplaunchable-config --verify
If any problems are reported, edit the file accordingly.
Use the plugin with pytest
Then, just add an --launchable
option to the pytest command. It is very easy:
pytest --launchable <your-pytest-project>
If the configuration file is not in the current directory, use the --launchable-conf-path
option:
pytest --launchable --launchable-conf-path <path-to-launchable-configuration-file> <your-pytest-project>
This will:
Create a build in your Launchable workspace
Run your tests
Submit your test reports to Launchable
Leave XML reports in the
launchable-test-result
by default
Requesting and running a subset of tests
In .launchable.d/config.yml
:
Check that the
source
option in therecord-build
section points to your Git repository (the default is.
, the current directory).Check that the
mode
option in thesubset
section is set tosubset
orsubset_and_rest
based on your needsCheck that one of the three optimization target options are set (
target
,confidence
, ortime
)
Verify your config file
Verify the contents of the .launchable.d/config.yml
file:
# via pipenvpipenv run launchable-config --verify# via piplaunchable-config --verify
If any problems are reported, edit the file accordingly.
Use the plugin with pytest
Then, just add an --launchable
option to the pytest command. It is very easy:
pytest --launchable <your-pytest-project>
If the configuration file is not in the current directory, use the --launchable-conf-path
option:
pytest --launchable --launchable-conf-path <path-to-launchable-configuration-file> <your-pytest-project>
This will:
Create a build in your Launchable workspace
Request a subset of tests based on your optimization target
Run those tests (or run all the tests if
subset_and_rest
mode is chosen)Submit your test reports to Launchable
Leave XML reports in the
launchable-test-result
by default
Launchable CLI
See Recording test results with the Launchable CLI and Subsetting with the Launchable CLI for more information.