Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

weather-tools as a library #237

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

weather-tools as a library #237

wants to merge 2 commits into from

Conversation

deepgabani8
Copy link
Collaborator

@deepgabani8 deepgabani8 commented Sep 21, 2022

Fixes #206 .

@alxmrs
Copy link
Collaborator

alxmrs commented Sep 21, 2022

This is a good start. However, the abstractions we provide in the Pipeline object should make adding variations to the pipeline easy (where we expect there to be changes). Our internal pipelines should be a guide for what kinds of changes we can expect from users. For example, users should be able to:

  • choose custom file openers (for their own file types) / ways users can create EE assets
  • create filters or combine URIs into groups to suite specific pipeline needs
  • specify if they want a batch or streaming pipeline

These only reflect our initial variants -- and only for the EE destination. I think that we could benefit from implementing this feature later, as we've still only seen a handful of variations of our pipeline.

@deepgabani8
Copy link
Collaborator Author

Yes, It is an abstraction of what weather-mv pipeline is doing which the user can inherit and override pipeline-level functions.

Like this going one step deeper, the user should be able to override bq, ee, rg level functions. For that we can import them in weather_mv's init and make extendable classes.
Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Factor weather-tools as a library that users can extend.
2 participants