reachable_registries doesn't find the General registry

[danielmatz]

On my Mac, everything works fine. But on a Linux cluster at work, I'm having an issue with reachable_registries.

Julia knows about both the General registry and our private registry.

(jl_SqSi8q) pkg> registry status
Registry Status
 [09fcaf40] EG
 [23338594] General

But reachable_registries only returns our private registry.

julia> length(reachable_registries())
1
julia> first(reachable_registries()).name
"EG"

I've tried removing and re-adding both registries, without success.

I'm on Julia 1.7.2.

julia> versioninfo()
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, broadwell)

[mattBrzezinski]

On your Mac are you using the same version of Julia? I suspect something is different between 1.6 and 1.7 here causing the issue.

[danielmatz]

Yes, I'm also using 1.7.2 on my Mac.

[mattBrzezinski]

Yes, I'm also using 1.7.2 on my Mac.

Hmm interesting. Okay, the way the code works is by default looking at Base.DEPOT_PATH. From there it looks at the sub-directory of all these dirs for Registries to try and get the names of all those available.

If you start up a REPL you can mimic this by doing...

julia> depots = Base.DEPOT_PATH
3-element Vector{String}:
 "/Users/mattbr/.julia"
 "/Applications/Julia-1.7.app/Contents/Resources/julia/local/share/julia"
 "/Applications/Julia-1.7.app/Contents/Resources/julia/share/julia"

julia> for d in depots
         try
           println(readdir(joinpath(d, "Registries")))
         catch
           println("DNE $d")
         end
       end
[".DS_Store", "General", "Invenia"]
DNE /Applications/Julia-1.7.app/Contents/Resources/julia/local/share/julia
DNE /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia

Might give some insight into what's going on

[danielmatz]

julia> depots = Base.DEPOT_PATH
3-element Vector{String}:
 "/home/dmatz/.julia"
 "/software/x86_64/julia/julia-1.7.2/local/share/julia"
 "/software/x86_64/julia/julia-1.7.2/share/julia"

julia> for d in depots
                try
                  println(readdir(joinpath(d, "Registries")))
                catch
                  println("DNE $d")
                end
              end
DNE /home/dmatz/.julia
DNE /software/x86_64/julia/julia-1.7.2/local/share/julia
DNE /software/x86_64/julia/julia-1.7.2/share/julia

I think you meant lowercase "registries", but macOS didn't mind the capital R. If I use lowercase on Linux, I get:

julia> for d in depots
                try
                  println(readdir(joinpath(d, "registries")))
                catch
                  println("DNE $d")
                end
              end
["EG", "General.tar.gz", "General.toml"]
DNE /software/x86_64/julia/julia-1.7.2/local/share/julia
DNE /software/x86_64/julia/julia-1.7.2/share/julia

Which is interesting... my Mac result including a "General" directory, but Linux just has a tarball. Is that OK?

[mattBrzezinski]

Hmmm I'm not sure why there's a tarball on Linux, I just asked a co-worker who uses Linux to run this and they have directories instead.

[danielmatz]

When I do registry rm General, that tarball goes away. And when I do registry add General, it comes back... Did the way registries get downloaded change in a recent Julia release, and you and your coworkers have a directory around from an earlier version?

And I'm guessing that reachable_registries looks for directories only, not tarballs?

[mattBrzezinski]

When I do registry rm General, that tarball goes away. And when I do registry add General, it comes back... Did the way registries get downloaded change in a recent Julia release, and you and your coworkers have a directory around from an earlier version?

And I'm guessing that reachable_registries looks for directories only, not tarballs?

I'm not sure what's changed w/ Julia specifically to this, I don't pay too much attention to releases. My coworker has 1.6 installed, and also installed 1.7 for this test, both times they were directories.

Yes, reachable_registeries() only looks for directories here.

[danielmatz]

Ah hah, check this out in the 1.7 release notes:

Registries downloaded from the Pkg Server (not git) are no longer uncompressed into files but instead read directly from the compressed tarball into memory. This improves performance on filesystems which do not handle a large number of files well. To turn this feature off, set the environment variable JULIA_PKG_UNPACK_REGISTRY=true.

[mattBrzezinski]

Ah there we go! We'll need to update the README to include that, we need the information from the Registry.toml file to get all the information so it'll need to be unpacked.

[danielmatz]

You can use Tar.jl and CodecZlib.jl to extract Registry.toml:

julia> open(".julia/registries/General.tar.gz") do io
          Tar.extract(GzipDecompressorStream(io)) do header
               header.path == "Registry.toml"
           end
       end
"/var/folders/9w/2h0b6sxs1f52vsyjfvczy58462lf_p/T/jl_LPiI4R"

There may also be a way to extract it into an IOBuffer instead of a temp file...

[mattBrzezinski]

You can use Tar.jl and CodecZlib.jl to extract Registry.toml:

julia> open(".julia/registries/General.tar.gz") do io
          Tar.extract(GzipDecompressorStream(io)) do header
               header.path == "Registry.toml"
           end
       end
"/var/folders/9w/2h0b6sxs1f52vsyjfvczy58462lf_p/T/jl_LPiI4R"

There may also be a way to extract it into an IOBuffer instead of a temp file...

Ah yeah, we can definitely make these changes! I'm quite busy with other work, if you'd like to make the PR I can do code review on it and get it through! If not, I can make these changes at some point.

[danielmatz]

Sure, I'll take a shot at it.

[danielmatz]

See #38