Recommended Practice for testing multiple images

[JoshuaWatt]

We have a test setup where different tests need to be run against different images programmed to the DUT; Ideally I would like something that looks like this:

@pytest.fixture(scope="function")
def image_A():
    # Some magic to program image A to DUT if it's not already

@pytest.fixture(scope="function")
def image_B():
   # Some more magic to program image B to DUT if it's not already

def test_boot_A(image_A, command):
    """
    Test that image A boots successfully
    """
    command.run_check("true")

def test_boot_B(image_B, command):
    """
    Test that image A boots successfully
    """
    command.run_check("true")

So a given test can ensure that a specific image is programmed to the device before it's run by simply pulling in the correct fixture. Importantly, the fixtures should not do anything if they know the image is already programmed to the DUT because it can take a long time, so we don't want to program it every test function (counterintuitively, this is why the image fixtures are scoped to "function"; we want them to check the image every function, but only (re-)program the device if the image is different than the current one.

The methods for "programming" a device can very widely and some devices may have multiple choices for how to be programmed; to deal with this I've made a ImageProgrammerProtocol Abstract Base class that the specific methods can derive from to implement the specific method they what to use. That implementation and the implementation for a "manual" programmer that just requires to user to manually program the device via whatever means they want is shown below:

import attr

from abc import ABC, abstractmethod
from labgrid import target_factory, step
from labgrid.driver.common import Driver


class ImageProgrammerProtocol(ABC):
    def __attrs_post_init__(self):
        super().__attrs_post_init__()
        self.__current_image = None

    @abstractmethod
    def write_image(self, image, capsys):
        pytest.fail("Class must be subclassed with actual programming implementation")

    @Driver.check_active
    def ensure_image(self, image, capsys):
        if self.__current_image is not None and self.__current_image == image:
            return self.__current_image

        self.write_image(image, capsys)
        self.__current_image = image
        return self.__current_image


@target_factory.reg_driver
@attr.s(eq=False)
class ManualImageProgrammer(ImageProgrammerProtocol, Driver):
    @Driver.check_active
    @step(args=["image"])
    def write_image(self, image, capsys):
        strategy = self.target.get_driver("Strategy")

        with capsys.disabled():
            strategy.transition("off")
            self.target.interact(
                "Use {image} to flash target '{name}', then press ENTER".format(
                    image=image, name=self.target.name
                )
            )

The corresponding environment might look something like:

targets:
  main:
    ...
    ManualImageProgrammer: {}

Now, our test file looks like:

@pytest.fixture(scope="session")
def programmer(target):
    return target.get_driver("ImageProgrammerProtocol")

@pytest.fixture(scope="function")
def image_A(programmer):
    programmer.ensure_image("/path/to/A.img")

@pytest.fixture(scope="function")
def image_B():
    programmer.ensure_image("/path/to/B.img")

This works really well for our fixed CI tests, since we know that we will always be using a specific image programming driver and flashing specific images to the device for testing, however it is inflexible because the images to be tested are hard coded in the code which means the environment must use the specific programmer the images required (for complex reasons, an image isn't always a binary that can be dd'd to some disk), even if the user would rather use a different method of programming that is more convenient for them. Ideally, there would be some way of specifying the images in environment file and then some way of "selecting" one of them to be programmed to the device.... my first though would be make some ImageProtocol class that contains the actual image programming parameters and then extend the strategy to allow switching between images, which might look something like this:

targets:
  main:
    ...
    ManualImageProgrammer: {}
    Image:
       name: ImageA
       path: image-a-path
    Image:
       name: ImageB
       path: image-b-path
images:
    image-a-path: /path/to/A.img
    image-b-path: /path/to/B.img

@pytest.fixture(scope="function")
def image_A(strategy):
    strategy.ensure_image("ImageA")

@pytest.fixture(scope="function")
def image_B(strategy):
    strategy.ensure_image("ImageB")

This would allow the user to specify exactly which images and programming strategies should be used for the tests since it's all in the env file.

I think if I removed the "current image" state from our ImageProgrammerProtocol ABC, this would even allow mixing and matching programmers, although I'm not sure how the activation/deactivation would work:

targets:
  main:
    ...
    ManualImageProgrammer:
      name: manual
    CustomImageProgrammer:
      name: custom
    Image:
      name: ImageA
      path: image-a-path
      bindings:
        programmer: manual
    Image:
      name: ImageB
      path: image-b-path
      bindings:
        programmer: custom
images:
    image-a-path: /path/to/A.img
    image-b-path: /path/to/B.img

Does this makes sense as a path forward? Is there a better way to do what we are trying to do?