Initial Release of weather-tools

The inaugural release of weather-tools.

Current State

Currently, there are three tools in development: weather-dl, weather-mv, and weather-sp. The first tool is in its beta stage, and the latter two are in alpha. Since this is the start of the project's changelog, I will now quickly summarize the features of each tool:

weather-dl: the Weather Downloader

Weather Downloader ingests weather data to cloud buckets.

• Downloads weather data from ECMWF through their MARS and CDS APIs.
• Supports pipeline Dry-runs.
• Downloads are filesystem agnostic. Data can be ingested to GCS, S3, Azure Blobstore, or a local filesystem.
• Manifests of downloads are recoded in Firebase.
• A ConfigParser-based DSL lets users select data to download and control how data is sharded in a general manner.

weather-mv: the Weather Mover

Weather Mover loads weather data from cloud storage into Google BigQuery.

• Weather data from any filesystem can be uploaded in batch to Google BigQuery.
• Both NetCDF and Grib data are explicitly supported. Later, any XArray-readable dataset will be supported.
• All rows include an "import time" to keep track of when the data was ingested.
• Weather data can be filtered by geographic area or by variable type.
• Supports inference if BigQuery Schema from parts of the dataset.
• Streaming pipelines for ingesting real-time data into BigQuery is supported.

weather-sp: the Weather Spitter

Splits NetCDF and Grib files into several files by variable.

• NetCDF and Grib data splitting is supported.
• Grib data is split by variable and leveltype.
• Buckets with mixtures of data types (Grib and NetCDF) can be processed at once.
• The root of the output path is computed for you; users have control over the parent directory.
• Dry-runs of splits are supported.

Recent Changes

• Adding back an example config. by @alxmrs in #30
• Handle NaNs in data by @pramodg in #33
• Add utf-8 encoding to file read in setup by @CillianFn in #36
• Bump urllib3 from 1.25.11 to 1.26.5 in /weather_dl by @dependabot in #37
• Support un-indexed / single valued coordinates. by @pramodg in #39
• Set up empty dataset, not table by @lakshmanok in #41
• Docs fix - typo & stale links by @CillianFn in #44
• Basic support for grib files. by @pramodg in #40
• Test example configs by @CillianFn in #56
• Read the docs config by @CillianFn in #61
• weather-mv: Now using Streaming Inserts into BQ by @alxmrs in #62
• weather-mv: Implemented streaming import of data into BigQuery. by @alxmrs in #58
• Added script to help contributors test each other's updates to weather-tools. by @alxmrs in #63
• Github action to publish package by @saveriogzz in #31
• Updated python package name to google-weather-tools. by @alxmrs in #67
• Updated the standard example configs to use Reanalyses instead of Ensemble Means. by @alxmrs in #66
• Setting initial versions of each weather-tool. by @alxmrs in #68

New Contributors

• @pramodg made their first contribution in #33
• @CillianFn made their first contribution in #36
• @dependabot made their first contribution in #37
• @lakshmanok made their first contribution in #41
• @saveriogzz made their first contribution in #31

Full Changelog: https://github.com/google/weather-tools/commits/v0.1.0