Proposal for the new schedule and topics of part 3

[pl-marasco]

DISCLAIMER: This is just a proposal and it's my understanding of the outcome of the last meeting. Everybody feels free to open a polite and open-minded discussion about all the aspects of the scheduling.

Part-3: Unlocking the Power of Space Data with Pangeo & OpenEO

• 14:00 Introduction to the afternoon session (... minutes)

*Data discoverability and searchability

• An overview of STAC (... minutes)
• Overview of different sources and platforms

Different data exploitation approaches

• Hands-on "The Easy-peasy life with OpenEO" (... minutes)
  • Snow coverage example
• How to exploit data on Pangeo
  • Snow coverage pure xarray version

Data and pre-processing general knowledge

• Cloud-friendly access to archival data with Kerchunk (... minutes)
• Introduction to chunking (... minutes)

Parallelization of the process

• Scaling with Dask (... minutes)
• Scaling with OpenEO (how it works underneath) (... minutes)

How to go beyond

• Understanding how to create a custom algorithm: UDF (OpenEO) and ufunc (Xarray) (... minutes)
• Common workflow that combines the best of the two “worlds” (... minutes)
• 17:45 Wrap-up and feedback survey (15 minutes)

@clausmichele Unfortunately I don't have all the GitHub names of the OpenEO group, could you please point this discussion as well to your community?

@tinaok @acocac @annefou

[tinaok]

Thank you @pl-marasco, this looks good for me,
@clausmichele can you please confirm this is ok with you so that we can start to create the table of contents, place holders

[bschumac]

Part-3: Unlocking the Power of Space Data with Pangeo & OpenEO

• 14:00 Introduction to the afternoon session (5 minutes) -> suggestion @aljacob / @annefou

*Data discoverability and searchability

• An overview of STAC and different sources and platforms (openeo.cloud, STAC browser, STAC Index ...) (15 minutes) -> @m-mohr

14:30 Data and pre-processing general knowledge

• Introduction to chunking examples with netcdf, zarr and Kerchunk (30 minutes) -> @tinaok
• Parallelization with Dask (30 minutes) -> @tinaok

16:00 Different data exploitation approaches

• How to exploit data on OpenEO (30 minutes) -> @clausmichele
  Snow coverage example
• How to exploit data on Pangeo (30 minutes) -> @pl-marasco
  Snow coverage pure xarray version

17:00 How to go beyond

• Understanding how to create a custom algorithm: UDF (OpenEO) and ufunc (Xarray) (20 minutes) -> @jdries
• Scaling with OpenEO (how it works underneath) (30 minutes) -> @jdries

17:45 Wrap-up and feedback survey (15 minutes) @bschumac / @annefou

[tinaok]

We have prepared https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/blob/main/tutorial/part3/chunking_introduction.ipynb
Which will talk about chunking_zarr_kerchunk.
And I propose that as the material for
Data and pre-processing general knowledge

We have also prepared https://github.com/pangeo-data/pangeo-openeo-BiDS-2023/blob/main/tutorial/part3/scaling_dask.ipynb (see updated version in merge request)

That is for
Parallelization of the process
Scaling with Dask

[bschumac]

Global feedback question? -> discuss with beginning questionnaire

[acocac]

@pl-marasco the proposed outline for the part 3 looks good to me too.

@tinaok and I have opened PR #14 that restructures the Jupyter book according to Part 1 and 2 from the README, and part 3 proposed in this discussion. We've opened an accompanying issue #15 to validate which sections require to be completed.

[clausmichele]

@tinaok I can confirm for my snow workflow part, but I need a feedback from VITO for the parts where thay are also involved.
@soxofaan and @Pratichhya could you ask Jeroen (invited him to this repo) about these parts: Scaling with OpenEO (how it works underneath) (... minutes) and Understanding how to create a custom algorithm: UDF (OpenEO) and ufunc (Xarray) (... minutes) ?

[tinaok]

@clausmichele thank you. Who will be contributing to 'Data discoverability and searchability' part from openeo side? would you mind tag the person?

[clausmichele]

Tomorrow morning at 10 we have an internal meeting and define all the responsible persons for each part.

[Pratichhya]

@tinaok, could you please add https://github.com/jdries too, to this repository?

[tinaok]

Invitations to @jdries @bschumac were sent before, it is 'awaiting response'

[soxofaan]

@soxofaan .. could you ask Jeroen ...

Sorry, the notifications for this repo apparently end up in another mailbox I'm used to, so I missed this. I assume Jeroen is already in the loop by now?

[jdries]

@clausmichele
Scaling with openEO is actually something I'll have to work out still. I'm guessing it's simply going to be slide based, maybe with some concrete cases like continental mapping.
For UDF's: we now also put them in part 2 I believe, but we could show another case, or dive into it a bit deeper.

[clausmichele]

@tinaok @acocac @pl-marasco the EODC Team will take care about this part An overview of STAC (... minutes), the length will be around 15/20 minutes. The content (to be finalized, it will be posted here) will be more or less the following:

• What kind of problem is STAC trying to solve, main concepts
• What are the main components (Catalog, Collection, Item, Asset)
• With a public Catalog, show a sample of Collection, Item, Asset as JSON (Firefox renders it nicely)
• Show the STAC Browser referring to the same Collection/Items/Assets we showed before
• Mention that we support it in openEO
• Mention the extensions (raster, etc), useful for interoperability with third party libraries.
• Open Source, open to new extensions.

[jdries]

Key takeaways for this IMO:

• not any 'valid' STAC is also a 'usable' STAC, the openEO perspective is the topic of this PR: https://github.com/openEOPlatform/documentation/pull/55/files
• link to FAIR principles, to understand the motivation for creating quality metadata

And of course it's nice to show how there's this nice synergy between tools in the STAC ecosystem, like openEO batch job output being visualized in the STAC browser:

[m-mohr]

Link to the STAC slides: https://docs.google.com/presentation/d/1S5QeRNGwfRSLGrw0w0wIR90nIYxhvimoIO97vqMZs4M/edit?usp=sharing

Open for feedback.

[jdries]

I'm still shuffling around my content a bit, trying to find a good flow:

In part 2, I have 30minutes for 'advanced workflows', that I can probably fill with this:

• complex workflows with predefined functions (worldwater, rank composites)
• FAIR metadata generation
• Short note on costs: https://docs.openeo.cloud/federation/accounting.html#platform-credit-rates

In part 3,
Advanced workflows: show a UDF example

In the 'under the hood/scaling' part, I can then discuss:

• continental mapping
• an actual 'under the hood' look of a backend

I also have some 'orphan' topics that could be used to fill up the agenda if nobody else is mentioning them:

• dataset sampling (like for ML training data extraction) -> there's actually a nice link to make with local processing/Pangeo here
• zonal statistics (at large scale)

[Pratichhya]

Hi @acocac,

Could you please check if https://github.com/Open-EO/FuseTS/blob/main/notebooks/Local/FuseTS%20-%20Peak%20Valley%20Detection_holoviews.ipynb plots works for you?
There seems to be a bug in holoview. Though I can create it in my local environment, it requires some installation when tested in a new environment.

I would also recommend not to separately install vam.whittaker, there could be wheel version conflict. Therefore use https://github.com/Open-EO/FuseTS/blob/peak_valley_notebook/environment_whittaker.yml

[acocac]

@Pratichhya I'm on holidays today. I'd suggest that @tinaok @pl-marasco @annefou help to test it today if urgent. I'll try to have a look at the issue this weekend.

[Pratichhya]

Sure sure it's ok @acocac. Have a nice holiday 😄

[pl-marasco]

@Pratichhya first I will try on my local machine and then test on PangeoEOSC deployment. I'll get back to you soon

[pl-marasco]

@Pratichhya despite all my tests I'm facing the same issue as @acocac

In all the tests I've conducted the issue is always linked to the vam-whittaker. I've been testing your yml file for the conda but it doesn't work.
The only way to get a usable env is through docker
Forcing installation without dependencies didn't work as well

[tinaok]

Does anyone knows where can I find the link to snow coverage example? @pl-marasco @acocac ?
I updated chunking and dask notebook, but the datas I can update that to example which we will use before the chunking-dask (i.e. snow) or after (related to UDF / apply u_func example)

[clausmichele]

@tinaok the example using openEO + STAC + xarray + dask is this one: https://github.com/EO-College/cubes-and-clouds/blob/main/lectures/3.1_data_processing/exercises/_alternatives/31_data_processing_stac.ipynb
There's not yet a version without openEO. I know that @pl-marasco said he was trying to convert it but didn't hear back from him recently.

[pl-marasco]

@clausmichele sorry to disappear from the radar for a little while but, afert quite some tests, I had I had to abbandon the idea of using the CDSE as data source.
The coonverted notebook relay on the Element84/AWS source and is almost ready (hope to release it on Monday).
The only adaptation that I had too do is the time window that now is focus one year later then the original.

[jdries]

Note that I already made a commit to actually only import the whittaker dependency when it's actually used. Problem is, I'm not sure how to do a bugfix release right now. So checking out from source would be one way to at least fix it, the other is to simply take the small piece of code out of FuseTS, and use it without all the other dependencies.

…

________________________________ From: pl.marasco ***@***.***> Sent: Friday, November 3, 2023 6:54 PM To: pangeo-data/pangeo-openeo-BiDS-2023 ***@***.***> Cc: Jeroen Dries ***@***.***>; Mention ***@***.***> Subject: Re: [pangeo-data/pangeo-openeo-BiDS-2023] Proposal for the new schedule and topics of part 3 (Discussion #11) @Pratichhya<https://github.com/Pratichhya> despite all my tests I'm facing the same issue as @acocac<https://github.com/acocac> In all the tests I've conducted the issue is always linked to the vam-whittaker. I've been testing your yml file for the conda but it doesn't work. The only way to get a usable env is through docker Forcing installation without dependencies didn't work as well — Reply to this email directly, view it on GitHub<#11 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABNJPSHOBJM3K2RKYUZOV63YCUVWPAVCNFSM6AAAAAA6E7WCXCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TINRZG42DQ>. You are receiving this because you were mentioned.Message ID: ***@***.***> VITO Disclaimer: http://www.vito.be/e-maildisclaimer

[jdries]

FYI, I went for option 2 and copied the relevant code which does not have dependency issues. This also will make it a lot easier for users to play with themselves during the workshop.