Should there adapters for image visualisation (e.g. segmentation)

[edwardchalstrey1]

I’ve added a second Hugging Face plugin for scivision, this time for image segmentation (rather than image classification). At the moment, as you’ll see in this example notebook, the outputs of these models are what you would see if you used the huggingface transformers package directly - there’s no attempt to display the results in an interesting way with a nice example dataset. I’ve had a chat with @acocac about that earlier this week, but it does raise an interesting question - what kind of adapters should be included for visualisation in scivision? Should there be an adapter for image segmentation visualisation?

In the case of classification, the output of models is typically a prediction result - this doesn't require a visualisation to understand, but I think segmentation does, especially for a non-programmer scientist.

[quantumjot]

I like the idea of a few core jupyter widgets (for example) that could help with data visualisation. I guess that, if we did it the right way, these could be reused in jupyter notebooks and dynamic web apps too?

[edwardchalstrey1]

That's a nice idea, I've worked with Jupyter widgets before and they are quite fun/handy. In principle, a scivis-widgets package with useful decorators for CV functions that have a certain output, could almost be separate from scivision itself, since it would be usable in other contexts, even if it works more simply with scivision itself - interesting thought!

[acocac]

Agree with @quantumjot, we should explore generalisable widgets for visualising model outputs. Not sure if we target them with interactable features (e.g. sliders, zoom-in/out, timelapse).

[edwardchalstrey1]

On HuggingFace itself, you get a window on the right that allows you to try out segmentation on an image e.g. this one - so the adapter code I'm imagining would do something similar. Unfortunately the code they use for this does not appear to be open, so may have to reinvent the wheel a little.