Should all datasets and models in the scivision catalog be loadable?

[edwardchalstrey1]

This is a question that came up in today's group session - currently as I see it, there are two main aspects of scivision as a package/platform which will/do have the following functions as part of the API:

1. The ability to search and add to the data/model catalogs
   • query() - finds models and datasets that match based on "task" & "format" (in models.json and datasources.json)
   • add_dataset() - WIP: adds a new dataset to datasources.json catalog
   • add_model() - WIP: adds a new model to models.json catalog
2. The ability to load a dataset and model that can run on it from the catalog
   • load_pretrained_model() - installs the model and allows it to be used
   • load_dataset() - downloads the dataset and gets it into a format that can be used

So the question is: Does everything in the catalogs and can be found with query need to be usable via the scivision functions load_pretrained_model and load_dataset or is it ok, or preferable even, to also allow datasets and models to be added to the catalog that cannot be loaded with these functions directly?

Allowing this would mean that models and datasets that can't easily be set up to be used by those functions would still be find-able (via query), even if getting them to work would require extra manual steps by the user. Would this mean that the catalog could be a lot larger than it would otherwise be?

One idea @scotthosking had - perhaps there could be a flag/label called "instalable" within each model/dataset catalog entry that indicates whether it can be installed and loaded via scivision. Similarly, there could be a flag like "configurable" for models that are either not pre-trained or could have parameter adjustments that affect their output.

[mooniean]

In that case, what would the query of a non-installable dataset return? A link to where the dataset is available?

[edwardchalstrey1]

Basically yes.

So currently what the query returns is a join on "task" and "data format" e.g. "segmentation" and "tif" - meaning it gives you a dictionary that has each model-dataset match which the catalog suggests will work, but it's just based on those two fields matching, you still have to run load_pretrained_model or load_dataset on them both afterwards to use them - so my idea is to note those catalog entries where that will not be possible and you need to do manual steps to install the model or download the data

[edwardchalstrey1]

@quantumjot comment in meeting: We could use an EXPERIMENTAL tag for features, models etc