Testing

ChimeraX currently has both a hand-rolled test suite and a Pytest suite; it is preferred that additional tests use Pytest.

pytest

The pytest test suite can be invoked from the command line, but it is easier to invoke it using ChimeraX’s top level Makefile. To run the tests, simply run:

make pytest

in the root directory. This runs three rounds of tests:

  1. tests/test_env.py with both Python and the ChimeraX binary, to ensure there’s no difference between running ChimeraX and calling python -m chimerax.core.

  2. Wheel tests; i.e. all tests that are marked with @pytest.mark.wheel

  3. All other tests, using a production session produced in conftest.py

In the latter two cases, the appropriate tests/test_imports_{app,wheel}.py is run before the rest of the suite, since if a module is unimportable then that is a critical error that must be addressed before other failures.

Test Order

In general it is good practice if tests do not depend on each other, but in our case different runs of the test suite do have an order. Wheel tests must be run first, for several reasons:

  1. Creating the session that is used for testing the ChimeraX app changes values found in the top-level chimerax namespace

  2. During session creation, a global chimerax.core.toolshed.Toolshed object is produced, which has its own problems detailed below

It would be possible to run these tests at the same time if these issues were addressed.

Configuration

The configuration for pytest is found in the top level pyproject.toml.

Pytest is configured to look for tests in both the top-level tests folder and any bundle’s tests folder.

Custom markers include:

  • wheel: used to mark tests that should be run in the same environment that would be present if a user had downloaded the ChimeraX PyPi package.

Please send email to chimerax-programmers@cgl.ucsf.edu before changing the configuration.

Uploading Test Data

ChimeraX deals with a huge variety of data, much of which is held in large files. To avoid making the ChimeraX repo too large, test data is not stored in the repo. It is instead stored on Plato, the RBVI’s server cluster. To add test data to Plato, place it in the test-data folder, which is next to the prereqs folder, in a subdirectory that has the same name as the bundle directory in the ChimeraX repository.

Downloading Test Data

There is a script at utils/sync-test-data.sh that will download test data from Plato given the name of a bundle’s folder, e.g.

./utils/sync-test-data.sh -b dicom

will download the test data for the dicom bundle to src/bundles/dicom/tests/data.

Adding Tests

First, add a folder called tests in your bundle’s directory. Any Python file in this directory named test_*.py or *_test.py will be picked up and run by Pytest. Then, in those files, any function that starts with test_ will be run.

def test_something():
    # test code here

If your test requires a session, the recommended way to do this is to use the test_production_session fixture. There is no need to import this fixture, as it is automatically available in all test files.

def test_something(test_production_session):
    # test code here

If you need to parametrize the test to run over several different cases, or need to mark the test as for the wheel, you’ll need to import pytest

import pytest

@pytest.mark.parametrize('input,expected', [('a', 'b'), ('c', 'd')])
def test_something(test_production_session, input, expected):
    # test code here

If your test uses the data directory, you can enumerate test cases using this pattern so that when the test case is printed in the Pytest log it shows just the name of the file or folder and no the whole absolute path to the data:

import os
import pytest

test_data_dir = os.path.join(os.path.dirname(__file__), 'data')

test_cases = os.listdir(test_data_dir)

@pytest.mark.parametrize('data', test_cases)
def test_something(test_production_session, data):
    test_file = os.path.join(test_data_dir, data)
    # test code here

Finally, if you have a set of tests that do require data, it’s best to mark those tests to be skipped if the data is not present.

import os
import pytest

test_data_dir = os.path.join(os.path.dirname(__file__), 'data')

if not os.path.exists(test_data_dir):
    pytest.skip("Skipping <BUNDLE> tests because test data is unavailable.",
    allow_module_level=True,
)

test_cases = os.listdir(test_data_dir)

@pytest.mark.parametrize('data', test_cases)
def test_something(test_production_session, data):
    test_file = os.path.join(test_data_dir, data)
    # test code here

Wheel Import Tests

The wheel import tests attempt to import every module in ChimeraX that is destined for inclusion in the PyPi package. The list of modules that are not included can be found in utils/wheel/filter_modules.py.

The wheel is created by copying the chimerax directory out of a built ChimeraX and then running that script to delete excluded modules. To keep data centralized, add exclusions to that file instead of tests/test_imports_wheel.py.