Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avoid loading pkg until needed #1

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
41 changes: 17 additions & 24 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,32 +14,25 @@ jobs:
strategy:
fail-fast: false
matrix:
julia-version:
- "nightly"
os:
- ubuntu-latest
julia-arch:
- x64
include:
- os: ubuntu-latest
julia-arch: x64
julia-version: 'nightly'
- os: windows-latest
julia-arch: x64
julia-version: 'nightly'
- os: macos-latest
julia-arch: aarch64
julia-version: 'nightly'
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@v1
- uses: actions/checkout@v4
- uses: julia-actions/setup-julia@v2
with:
version: ${{ matrix.julia-version }}
arch: ${{ matrix.julia-arch }}
- name: Cache artifacts
uses: actions/cache@v2
env:
cache-name: cache-artifacts
- uses: julia-actions/julia-buildpkg@v1
- uses: julia-actions/julia-runtest@v1
- uses: julia-actions/julia-processcoverage@v1
- uses: codecov/codecov-action@v2
with:
path: ~/.julia/artifacts
key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
restore-keys: |
${{ runner.os }}-test-${{ env.cache-name }}-
${{ runner.os }}-test-
${{ runner.os }}-
- uses: julia-actions/julia-buildpkg@latest
- uses: julia-actions/julia-runtest@latest
- uses: julia-actions/[email protected]
continue-on-error: true
- uses: julia-actions/[email protected]
continue-on-error: true
files: lcov.info
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@

Manifest.toml
69 changes: 67 additions & 2 deletions src/LazyArtifacts.jl
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,72 @@ using Artifacts: Artifacts,
export artifact_exists, artifact_path, artifact_meta, artifact_hash,
select_downloadable_artifacts, find_artifacts_toml, @artifact_str

# define a function for satisfying lazy Artifact downloads
using Pkg.Artifacts: ensure_artifact_installed
using Base.BinaryPlatforms: AbstractPlatform, HostPlatform
using Base: SHA1

# We are mimicking Pkg.Artifacts.ensure_artifact_installed here so that we can
# check if the artifact is already installed before loading Pkg.
# Then if we need to load Pkg we do it in a subprocess to avoid precompilation
# complexity.

"""
ensure_artifact_installed(name::String, artifacts_toml::String;
platform::AbstractPlatform = HostPlatform(),
pkg_uuid::Union{Base.UUID,Nothing}=nothing,
verbose::Bool = false,
quiet_download::Bool = false,
io::IO=stderr)

Ensures an artifact is installed, downloading it via the download information stored in
`artifacts_toml` if necessary. Throws an error if unable to install.
"""
function ensure_artifact_installed(name::String, artifacts_toml::String;
platform::AbstractPlatform=HostPlatform(),
pkg_uuid::Union{Base.UUID,Nothing}=nothing,
verbose::Bool=false,
quiet_download::Bool=false,
io::IO=stderr)
meta = artifact_meta(name, artifacts_toml; pkg_uuid=pkg_uuid, platform=platform)
if meta === nothing
error("Cannot locate artifact '$(name)' in '$(artifacts_toml)'")
end

return ensure_artifact_installed(name, meta, artifacts_toml;
platform, verbose, quiet_download, io)
end

function ensure_artifact_installed(name::String, meta::Dict, artifacts_toml::String;
platform::AbstractPlatform=HostPlatform(),
verbose::Bool=false,
quiet_download::Bool=false,
io::IO=stderr)

hash = SHA1(meta["git-tree-sha1"])
if !artifact_exists(hash)
# loading Pkg is a bit slow, so we only do it if we need to
# and do it in a subprocess to avoid precompilation complexity
code = """
Pkg = Base.require_stdlib(Base.PkgId(Base.UUID("44cfe95a-1eb2-52ea-b672-e2afdf69b78f"), "Pkg"));
Pkg.Artifacts.try_artifact_download_sources(
$(repr(name)),
Base.$(repr(hash)),
$(repr(meta)),
$(repr(artifacts_toml));
platform = Base.BinaryPlatforms.$(repr(platform)),
verbose = $(repr(verbose)),
quiet_download = $(repr(quiet_download)),
io = stderr
)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, unfortunately running a sub-process with dynamic code like this is very incompatible with --trim

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #1 (comment) for some discussion

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to do this only during pre-compilation? That behavior on its own is fine, but doing it at runtime is problematic for --trim

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to do this only during pre-compilation?

If we revert JuliaPackaging/JLLWrappers.jl#66 yes 😛

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really agree that lazy artifacts are incompatible with AOT compilation - but they are definitely incompatible with the idea of providing a self-contained executable/package

My request was for this to be branched on Base.generating_output() because the true branch is not inferable/compilable for juliac, but we have ways to let juliac know that only the false branch is ever taken at run-time.

In that case, you're still running an AOT-compiled executable and the download works - it just happens lazily.

Copy link
Member Author

@IanButterworth IanButterworth Sep 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Didn't see Cody's reply before sending this)
I think the question here is more about legality, rather than preference that something orchestrating --trim would have for artifact downloads a la PackageCompiler.

As it is now seems more generally legal.

Copy link

@topolarity topolarity Sep 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, I fear that Base.require_stdlib is still going to be effectively impossible for us to infer

We're not even allowed to load the stdlib for inference on its own functions, because that would break the same pre-compilation assumptions we're skirting in the other branch.

I'm a bit worried that this change just makes LazyArtifacts incompatible with --trim unfortunately

We might have to find a way to load Pkg eagerly for --trim alone (which unfortunately means that AOT-compiled applications don't get this speed up)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if you can infer it directly, but you should be able to pre-emulate it, since it is just a shim for finding the right .ji file to load, and that can be pre-computed and run directly (esp. for --trim)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right but then I need to infer Pkg.Artifacts.try_artifact_download_sources and also ensure that its code is available in the loaded pkgimage (or our own sysimage)

"""
out = readchomp(pipeline(`$(Base.julia_cmd()) -E $code`, stderr=io))
return Meta.parse(out)
else
return artifact_path(hash)
end
end

if Base.generating_output()
precompile(Tuple{typeof(Artifacts._artifact_str), Module, String, Base.SubString{String}, String, Base.Dict{String, Any}, Base.SHA1, Base.BinaryPlatforms.Platform, Base.Val{LazyArtifacts}})
end

end
Loading