Lots and lots of workflows for CoDSummit

[philwinder]

Brain dump

• IPFS data statistics workflow
• Format conversion, like csv to parquet, or csv to json
• Validation
• Descriptions

---

Use Cases

IPFS Data Statistics

Important to do this first, because there's no point working on ${DataType} use cases if there are no ${DataTypes} in IPFS -- {insert data type here}.

Metadata -> Pushing metadata info to external database (security concern, whitelist?)

Web-Focussed Image Compression

Metadata | predicate image/*-> Image Processing -> Merge

Web-Focussed Video Compression

Metadata | predicate video/*-> Video Processing -> Merge

Transcription of Video and Audio

Metadata | predicate [video|audio]/* -> Transcription model(s) -> Merge

CSV/Parquet Summary Statistics

Interesting one here, parquet doesn't have a registered mime type yet. Wonder if tika can parse?

Metadata | predicate [text/csv| application/parquet] -> Load and produce summaries of data -> Merge

${DataType} Enrichment

For example, given CSV file, columns have a lat and long, run job that converts lat/long to country/city and creates output csv with the same row format, then merge back together with original data.

Metadata | predicate ${DataType} -> Parse columnar data type | predicate ${ColumnDataType} -> Data Enrichment -> Merge

Image Dataset Analysis

https://cleanvision.readthedocs.io/

Metadata | predicate image/*-> Image analysis -> Merge

[davidgasquez]

Adding a couple more:

• Packaging the output as a datapackage.json. This will make it easier to integrate all the existing tooling (and checks) around them. Should be a simple workflow!
• Something like this might be cool to: https://github.com/deepchecks/deepchecks.

[philwinder]

Finished all workflows for Boston. Can obviously add more workflows in the future.