ODD PI 25.2 Objective 7: Increase data format support in VirtualiZarr

[maxrjones]

Motivation

Expand virtualization capabilities of VirtualiZarr to support other formats.

Description

The VirtualiZarr library allows users to benefit from Zarr’s async capabilities even for datasets that are not stored as Zarr. However, there are some limitations to the types of data that can be virtualized by VirtualiZarr and stored as virtual Icechunk stores. This objective will increase the types of datasets that are virtualizable.

Acceptance Criteria

• Add support for virtual DataTrees. This will enable Zarr-based entry points to multiple Cloud-Optimized GeoTIFFs (COGs) and virtualizing hierarchical HDF5 and GRIB files (add’l work to support those formats will be required).
• Support storing virtual GeoTIFFs and COGs in Icechunk by stabilizing the aiocogeo-rs library and implementing a VirtualiZarr reader using aiocogeo-rs

Sub-tasks

Support for virtual DataTrees

Tasks

Preview Give feedback

1. Add a group option for the icechunk writer (to_icechunk doesn't have a group option zarr-developers/VirtualiZarr#341)
2. Manually test that Xarray DataTrees can be stored and loaded using Icechunk #42
   upstream +1
3. Add a rountrip test for writing/reading a manually constructed virtual DataTree using Icechunk (allow writing DataTree objects containing references to disk zarr-developers/VirtualiZarr#244)
4. Implement datatree_to_icechunk such that the previously added test passes
5. Add a VirtualiZarrDataTree accessor for convenient access to datatree_to_icechunk
6. Generate a simple test HDF5 file that contains nested groups
7. Add a test for opening the HDF5 file using open_virtual_datatree (Support for groups zarr-developers/VirtualiZarr#84)
8. Implement open_virtual_datatree with HDF5VirtualBackend as a supported reader such that the previously added test passes
9. Add documentation for virtualizing DataTrees

Support storing virtual GeoTIFFs and COGs

Tasks

Preview Give feedback

1. Stabilize metadata reading in aiocogeo-rs
2. Ensure metadata reading in aiocogeo-rs is working with remote stores like S3
3. Implement decoding and reading specific COG tiles in aicogeo-rs
4. Implement reads across multiple COG tiles in aiocogeo-rs
5. Add Python bindings for aiocogeo-rs
6. Add Xarray backend for aiocogeo-rs' Python bindings
7. Add VirtualiZarr reader for aiocogeo-rs, using xarray backend for inlining coordinates
8. Add documentation for new VirtualiZarr GeoTIFF/COG reader