conda-forge CDN cloning last updated ~869mins ago

[jakirkham]

Looks like we are running into an issue with CDN cloning for the conda-forge channel

According to the conda-forge.org/status page, it has been ~869mins since that last update (screenshot below):

xref: conda-forge/status#176

[jakirkham]

A recent example of this is libtorch. The oldest package here was uploaded ~4hrs ago and still has a download count of 0

Edit: This can also be confirmed with conda search 'conda-forge:libtorch[subdir=osx-arm64]=2.1.2=*_4'. Please details below

% conda search 'conda-forge:libtorch[subdir=osx-arm64]=2.1.2=*_4'
Loading channels: done
No match found for: libtorch==2.1.2[subdir=osx-arm64,build=*_4]. Search: *libtorch*==2.1.2[subdir=osx-arm64,build=*_4]

PackagesNotFoundError: The following packages are not available from current channels:

  - libtorch==2.1.2[subdir=osx-arm64,build=*_4]

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Note: conda search 'conda-forge:libtorch[subdir=osx-arm64]=2.1.2=*_3' (the previous build) works

[h-vetinari]

Another example is pyarrow-core / pyarrow-all which saw their first builds a few hours ago.

[jakirkham]

The conda-forge status page is now showing the CDN as having synced recently

---

Do see some improvements with libtorch, but still a lot of packages are being missed

---

IOW this is still a live issue, but it does seem like the CDN is trying to do something

[h-vetinari]

IOW this is still a live issue, but it does seem like the CDN is trying to do something

I'm cautiously optimistic. Basically my guess is that the cloning is operational again, and now it takes "just" the usual 1-2 hours (or sometimes 5, if a clone fails), for the packages to show up in the CDN.

[h-vetinari]

All the pyarrow-core packages have now made it through the CDN.

[jezdez]

Yes, this was a temporary problem that @dholth has fixed within the cloning system.

[jakirkham]

Thanks Jannis! 🙏

Do we know what the cause was?

[dholth]

We are working on a permanent fix that does not fill the disk.
When the clone process is working, it takes about 8-12 minutes per iteration. There may be delays in other parts of the process like anaconda.org or the CDN. Do we have a way to measure the time that individual packages take to clone?

[jakirkham]

Thanks Daniel! 🙏

Ok gotcha. So this is the disk overfilling issue we have seen before. Good to know it's a familiar issue

I don't think we have measurements collected external to Anaconda. Though I would be happy to be wrong. @wolfv or @beckermr is this something either of you have looked at before? No worries if not

Is this measured on the Anaconda side somehow?