Parametrized fixture based on configuration file

[jmlemetayer]

Hi folks,

I would like to be able to test multiple embedded boards at the same time using a configuration file.

My idea was to have at the end something like that:

# config.yaml

devices:
  - name: board_one
    tty_port: /dev/ttyUSB0

  - name: board_two
    tty_port: /dev/ttyUSB1

# test_a.py
import pytest

def test_one():
    pass

@pytest.mark.not_board_one
def test_two():
    pass

To do this I have started a local plugin like this:

# conftest.py
import pytest
import yaml

config = dict()
devices = list()

def pytest_configure(config):
    # Get 'config_file' from ini or options
    read_config(config_file)

def read_config(config_files):
    global config
    global devices

    for config_file in config_files:
        with open(config_file, "r") as stream:
            config = yaml.safe_load(stream)

    for device in config.get("devices", []):
        devices.append(Device(**device))

class Device:
    def __init__(self, name, tty_port):
        self.name = name
        self.tty_port = tty_port

def get_devices():
    global devices
    return devices

@pytest.fixture(scope="session", params=get_devices(), autouse=True)
def device(request):
    return request.param

I did it like this because I would like to have my test to run on all boards. But I do not want to have the device fixture mandatory, so the tests stay quite simple.

My idea was to later add some tty_port and even serial fixtures on top of that. So that a test can request a working serial link with the board just by doing:

# conftest.py
@pytest.fixture(scope="session")
def tty_port(device):
    return device.tty_port

@pytest.fixture(scope="session")
def serial(tty_port):
    # Open and configure a serial link using the tty_port
    return serial

# test_b.py

def test_three(serial):
    # Do some serial communications

But this is not working... I have an error like this:

test_a.py::test_one[device0] SKIPPED (got empty parameter set ['device'], ...)

Clearly the devices list is still empty when the fixture is constructed.

Is it possible to do something like that ? What will be the correct way to do it ? Parsing the yaml in the get_devices function is working, but I lose all the getini and getoption mechanism (not shown in the example).

Thanks in advance.

[portyu9]

Hi @jmlemetayer

The reason params=get_devices() is empty is that the fixture decorator is evaluated while the module/plugin is being imported. Your devices are only populated later in pytest_configure.

For parameters derived from pytest configuration, use pytest_generate_tests, which runs during collection and has access to metafunc.config.

For example:

@pytest.fixture(scope="session")
def device(request):
    # Expensive setup can happen here.
    return connect_to_device(request.param)


def pytest_generate_tests(metafunc):
    if "device" in metafunc.fixturenames:
        config_file = metafunc.config.getoption("--config-file")
        devices = load_device_descriptions(config_file)

        metafunc.parametrize(
            "device",
            devices,
            indirect=True,
            scope="session",
            ids=lambda d: d.name,
        )

Then:

def test_something(device):
    ...

The important separation is:

• collection time: determine which test cases/device descriptions exist;
• fixture setup time: open connections and create expensive resources.

That also avoids opening hardware/resources just to collect the test suite.

Please mark this answer as accepted if it helped, thank you.

[jmlemetayer]

Your solution should work. Thanks ! Using pytest_generate_tests is something I didn't know at the time.

My solution was different:

# conftest.py
import pathlib

import pytest
import yaml


class Device:
    def __init__(self, name: str, tty_port: str) -> None:
        self.name = name
        self.tty_port = tty_port


device_key = pytest.StashKey[Device]()


def pytest_addoption(parser: pytest.Parser) -> None:
    parser.addoption(
        "--config-file",
        type=pathlib.Path,
        default=pathlib.Path("devices.yaml"),
        help="YAML file listing available devices.",
    )
    parser.addoption(
        "--device-name",
        help="Name of the device to test (defaults to the first one in the file).",
    )


def pytest_configure(config: pytest.Config) -> None:
    config_file = config.getoption("config_file")
    with open(config_file) as stream:
        raw_devices = yaml.safe_load(stream)["devices"]

    devices = [Device(**d) for d in raw_devices]

    wanted = config.getoption("device_name")
    if wanted is None:
        device = devices[0]
    else:
        device = next((d for d in devices if d.name == wanted), None)
        if device is None:
            raise pytest.UsageError(f"No device named {wanted!r} in {config_file}")

    config.stash[device_key] = device


@pytest.fixture(scope="session")
def device(request: pytest.FixtureRequest) -> Device:
    return request.config.stash[device_key]

# test_example.py
def test_device_name(device):
    print(device.name, device.tty_port)

• One single device tested at a time (but you can run multiple pytest on each device in parallel)
• No params= so no problem
• Reading config file and selecting the device in pytest_configure and store the Device into a config.stash key
• The device session fixture get the Device from the config.stash