era5 CodeBuild pipeline failed due to dockerhub pull rate limit on miniconda

[rsignell-usgs]

@zflamig , the CodeBuild pipeline failed due to dockerhub pull rate limit on miniconda

Step 1/9 : FROM continuumio/miniconda3:4.7.12
347 | toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
348

@ocefpaf, do you know if there is a different place to pull the Miniconda Container from now?

[rsignell-usgs]

Tipped off by @zflamig , I see in this AWS Announcement it says:

"Within weeks, AWS will deliver a new public container registry that will allow developers to share and deploy container images publicly. This new registry will allow developers to store, manage, share, and deploy container images for anyone to discover and download. Developers will be able to use AWS to host both their private and public container images, eliminating the need to use different public websites and registries. Public images will be geo-replicated for reliable availability around the world and offer fast downloads to quickly serve up images on-demand. Anyone (with or without an AWS account) will be able to browse and pull containerized software for use in their own applications. "

[ocefpaf]

@ocefpaf, do you know if there is a different place to pull the Miniconda Container from now?

I don't know about miniconda but miniforge is on quay.io to avoid that.

See https://quay.io/repository/condaforge/miniforge3

[rsignell-usgs]

@zflamig what say ye? I think it would be cool to try the miniforge3 container. @ocefpaf says it's a lightweight drop-in replacement for miniconda.

[rsignell-usgs]

@zflamig, you switched to miniforge3 somewhere, right? Where is that code?

[zflamig]

https://github.com/zflamig/dask-worker/blob/master/Dockerfile#L1 here we are using the miniforge3 now

[RichardScottOZ]

Thanks for all this hard work gentlemen, you have saved me a ton of time getting some basic testing done.

[RichardScottOZ]

@rsignell-usgs - this is where I got to, for if you have any ideas https://github.com/RichardScottOZ/dask-era5/blob/main/notebook/pangeoFargateTrialSA.ipynb