Show list of images that would be processed by the first task of a job

[tcompa]

Starting from this situation

we want the client to expose the information of "which images will be processed by the first task in the workflow"

This requires merging together a few different filters (TBD):

1. workflowtask.input_filters (where workflowtask is the first one that would be executed, that is the one in "First task (Optional)")
2. dataset.filters

and then calling the POST /api/v2/project/1/dataset/1/images/query endpoint with the appropriate filters.

UI for showing the image list:

1. We start by showing it within the modal (which should be larger)
2. It's hidden by default, and expansible
3. The backend responds with paginated output
4. In case this turns out to be too complex a visualization, we can always open the image list in a different page/tab

[jluethi]

• Let's use the current image list interface for this
• Don't allow users to change type filters
• Prepopulate attribute filters, allow users to change them (or even remove them!)

[jluethi]

Image list & selection is shown in the first modal page of running a workflow:

It is shown on Continue workflow
It's shown on Run Workflow if the image list is not empty

It's not shown on restart workflow (as the dataset will be empty then)

Scope limits:

1. This does not help with running converters on subset
2. A user cannot set a workflow that will convert everything, but only run Cellpose on a subset. The way to do that: Convert everything first in a first job. Then run a second job with another filter set

[tcompa]

For the first implementation, the pre-populated filters should be a combination of the following filters (if set), starting from the lowest-priority one:

1. dataset.filters
2. worfklowtask.input_filters (where worfklowtask is the first task that would be run, i.e. the one currently called "First task")

(note: this is only for the first implementation, namely the one which does not yet require the new filters data structure defined e.g. in fractal-analytics-platform/fractal-server#1401 and fractal-analytics-platform/fractal-server#2153).

[tcompa]

Note: the current table will not have the view/edit/delete buttons, for the moment.

Type filters (if set) are immutable.
In the current preliminary implementation, also attributes filters are immutable. As soon as we have e.g. fractal-analytics-platform/fractal-server#2153 ready, we will let the user change attributes filters and then set them as part of the submission context.

[zonia3000]

In the API call should I also set use_dataset_filters=true?

This is a preliminary screenshot:

Should we keep the "All images" / "Dataset filters" switch?

[tcompa]

Should we keep the "All images" / "Dataset filters" switch?

No, there should be a single way of viewing this table.

In the API call should I also set use_dataset_filters=true?

• We should start from the "all images" situation (that is, no filters), and include a single set of filters in the POST query.
• The filters to be included in the request are the ones built by combining dataset.filters (lower priority) and worfklowtask.input_filters - as in Show list of images that would be processed by the first task of a job #682 (comment).
• For the "attributes" filters, the combination should be equivalent to what in Python would be

request_filters["attributes"] = dataset.filters["attributes"]
request_filters["attributes"].update(worfklowtask.input_filters["attributes")

• The same applies to types

[tcompa]

• New configuration variable: ENABLE_INTERACTIVE_ATTRIBUTE_FILTERS.
• Default behavior: user cannot modify attribute filters interactively from the job-submission modal.
• Custom behavior (obtained by setting the config variable to some value): attribute filters can be modified from the job-submission modal.
• This variable does not affect the fact that one can always see the image list from the job-submission modal.

[jluethi]

For attribute filters, the worfklowtask.input_filters will go away and we will have job attribute filters, right? As in, what is hidden behind the ENABLE_INTERACTIVE_ATTRIBUTE_FILTERS flag

[tcompa]

For attribute filters, the worfklowtask.input_filters will go away and we will have job attribute filters, right?

That's correct (but not yet available in the backend).

As in, what is hidden behind the ENABLE_INTERACTIVE_ATTRIBUTE_FILTERS flag

The configuration variable will enable the possibility of setting job.attribute_filters. By default, it will not be possible. By enabling the variable (e.g. on our staging instance), we will be able to set attribute filters interactively in the context of job submission.

[tcompa]

As a clarification: the future list of attribute filters would be:

1. dataset.attribute_filters, which cannot be modified upon job submission
2. job.attribute_filters, to be set upon job submission (if the the config variable is set). These will be pre-populated with dataset.attribute_filters (if any), and then modified interactively. If they are set, they will replace dataset.attribute_filters, and be the only filters to be used in the backend runner. Ref Replace attribute filters rather than merging them fractal-server#2155