TEM formats

[jan-janssen]

TEM file formats implemented in hyperspy:
https://hyperspy.readthedocs.io/en/stable/user_guide/io.html#supported-formats

hyperspy supports most formats and is also used as part of TEMMETA to access formats which are not natively supported.

[din14970]

Yep exactly, one of the big selling points of Hyperspy. One of the reasons temmeta came into existance was because the hyperspy reader for Velox EMD was broken, so I wrote my own from scratch. Now that the Hyperspy one seems to be fixed (though still includes a small bug I intend to make a PR for), I'm wondering whether it is best to change it up and make temmeta use hyperspy for importing datasets (with the risk that if hyperspy breaks TEMMETA breaks) or just copy and adapt the code from the different readers directly into temmeta.

[jan-janssen]

I guess it depends on the ratio of how much of Hyperspy you use. If you only use 10% of their code, maintaining compatibility is maybe not reasonable, but if it is a larger share or you see potential use for more of it I would always use the code as a dependency. There are already people who feel responsible for maintaining the hyperspy module, people who use it and give feedback, that is a great thing, I see no scientific reason to duplicate this work and copying the code would be duplicating in my perspective.
Sure there is the risk that hyperspy might break in a future version and that would also break temmeta but you can fix this by restricting the versions of hyperspy you support. So if it is possible to derive your classes directly from hyperspy I believe that would be the right direction to go. Changing it later is only more difficult.

Two examples from the pyiron project:

• We still work on deriving our atoms class from the ASE atoms: A new Atoms class derived from ase pyiron#680
• I am still fixing bugs in ASE when I see them: https://gitlab.com/ase/ase/-/merge_requests?state=all&author_username=jan-janssen

Obviously one could ask why we need pyiron when we use the same structure class as ASE, still what we add is: data management, job management and the primary focus on jupyter notebooks. We therefore position ourselves as an extension to ASE which goes beyond parsers for the simulation codes and creates an environment for developing simulation protocols. With this clear split I hope that we can have even more collaboration between both projects in the future. And I could foresee a similar development for TEMMETA and hyperspy. With your focus on the actual needs of the experimentalists you already add a new perspective which extends the data science driven approach of hyperspy - as far as I understand.