- RAF dfp zorder joins
- RAF dfp zorder merge
- WAF
- Feature to be able to launch script both locally and in cluster (even if cluster settings are to be set manually and script is only copied)
- Push filtered airports dataset somewhere public and use it in the snippets
- Rename snippet file names to comply with convention problem-solution
- Address different Spark versions
- Specify spark version in snippet itself (do not hardcode it)
- Handle the fact that the Spark version used by the user locally may not be the same as the one used to test the script (raise warning or failure in case of wrong Spark versions)
- All needed snippets are present for SFO
-
TODO for each snippet today (especially zorder-select one)
- For each snippet we have a correct name, documentation, and it works
- Add WHAT TO LOOK FOR in comments
- Be more explicit about where the snippet has been tested
-
Create Databricks cluster from settings in the snippet
-
Extract common building-blocks in a dedicated folder (e.g. merge, streaming foreachBatch, showMaxMinStats, ...)