Introduce Multi-Storage Client (MSC) as an optional dependency #754
Conversation
dreamtalen
force-pushed
the
msc-image
branch
2 times, most recently
from
08a450a to
11bfea4
Compare
dreamtalen
changed the title
dreamtalen
force-pushed
the
msc-image
branch
3 times, most recently
from
941e074 to
40e72aa
Compare
|
/blossom-ci |
pyproject.toml
Outdated
| @@ -59,6 +59,7 @@ dev = [ | |||
| "interrogate==1.5.0", | |||
| "coverage==6.5.0", | |||
| "ruff==0.0.290", | |||
| "multi-storage-client>=0.12.2", | |||
There was a problem hiding this comment.
Why is this only in the dev optional dependency list? Wouldn't it be better to put it in some storage specific optional dependency list?
There was a problem hiding this comment.
Make sense, moved it to a storage dependency list
pyproject.toml
Outdated
| @@ -94,6 +95,7 @@ all = [ | |||
| "nvidia-modulus[dev]", | |||
| "nvidia-modulus[makani]", | |||
| "nvidia-modulus[fignet]", | |||
| "multi-storage-client[boto3]", | |||
There was a problem hiding this comment.
Why only install boto3 and not other backends in the all dep list?
There was a problem hiding this comment.
Thanks for pointing out! I removed the boto3 here as it should be installed in the example folder instead.
| type: s3 | ||
| options: | ||
| region_name: us-east-1 | ||
| endpoint_url: https://pbss.s8k.io |
There was a problem hiding this comment.
Remove this reference from here. A modulus example should be runnable by anyone without any specific network domain requirements. Please refactor to point this example to a publicly available zarr dataset. Here are a couple of dataset suggestions:
- CMIP6 archive on AWS: https://registry.opendata.aws/cmip6/
- ARCO ERA5 dataset on google cloud: https://github.com/google-research/arco-era5?tab=readme-ov-file#025-pressure-and-surface-level-data
There was a problem hiding this comment.
Thanks, I added examples of CMIP6.
dreamtalen
force-pushed
the
msc-image
branch
2 times, most recently
from
e479897 to
e20346a
Compare
Modulus Pull Request
Description
This PR introduces Multi-Storage Client (MSC) as an optional dependency for Modulus, with examples.
The Multi-Storage Client (MSC) is a unified, high-performance Python client designed to seamlessly interface with various object and file storage systems. It supports:
MSC provides a generic interface to interact with objects and files across various storage services. This lets you spend less time learning each storage service's unique interface and lets you change where data is stored without having to change how your code accesses it.
We have successfully completed several PoCs (ICON, CorrDiff) using MSC training from S3-compatiable object stores on Modulus workloads.
Checklist
Dependencies