FR: An alternative API design for easier apt.install #123

thesayyn · 2024-12-04T18:10:57Z

apt = use_extension("@rules_distroless//apt:extensions.bzl", "apt")
apt.configure(
    architectures = ["amd64", "arm64"]
)
apt.sources_list(
    "deb [arch=amd64,arm64] https://snapshot.ubuntu.com/ubuntu/20240301T030400Z noble main",
    "deb [arch=amd64,arm64] https://snapshot.ubuntu.com/ubuntu/20240301T030400Z noble-security main",
    "deb [arch=amd64,arm64] https://snapshot.ubuntu.com/ubuntu/20240301T030400Z noble-updates main",
)
apt.install(
    "ncurses-base",
    "libncurses6",
    "tzdata",
    "coreutils:arm64",
    "libstdc++6:i386"
)
apt.lock(into = ":lock.json")

The text was updated successfully, but these errors were encountered:

jjmaestro · 2024-12-11T13:04:16Z

This is serendipitous 😄

I didn't see this issue until I got notified when you mentioned it in #56 but last week I was also drafting an almost exact API!

I've been thinking about it since my reviews to those old PRs and the issues I found with the architectures. I 200% love this, the current API conflates the architecture of sources with the architecture of install packages. This new API mirrors much closer to how Debian works, I think it will be awesome to have.

TL;DR

Here's a quick overview of how I was drafting a new API, incorporating some of your

MODULES.bazel

apt = use_extension("@rules_distroless//apt:extensions.bzl", "apt")

apt.sources(
    name = "bookworm",
    file = "//apt/sources/bookworm.sources",
)

apt.sources(
    name = "docker",
    file = "//apt/sources/docker.list",
)

apt.install(
    name = "base"
    # architectures = ["arm64"],
    sources = ["@apt//sources/bookworm"],
    packages = "//apt/packages/bookworm.yaml",
    lock = "//apt/packages/bookworm.lock.json",
)

# defaults to the "nolock = True" behavior
apt.install(
    name = "docker"
    # architectures = ["arm64"],
    sources = ["@apt//sources/docker"],
    packages = "//apt/packages/docker.yaml",
)

use_repo(apt, "apt")

Commands:
These commands can be optional / removed if FR: Detect manifest changes and automatically run lock #130 is implemented.

# update sources repo
$ bazel run @apt//sources/bookworm:update

# make / update install lock
$ bazel run @apt//install/base:lock

//apt/sources/bookworm.sources (DEB822 .sources format):

Types: deb
Architectures: amd64 arm64
URIs: http://deb.debian.org/debian
Suites: bookworm bookworm-updates
Components: main contrib non-free non-free-firmware

Types: deb
Architectures: amd64 arm64
URIs: http://deb.debian.org/debian-security
Suites: bookworm-security
Components: main contrib non-free non-free-firmware

//apt/sources/docker.list (one-line .list format):

deb [arch=amd64,arm64] https://download.docker.com/linux/debian bookworm stable

//apt/packages/bookworm.yaml:

packages:
  - coreutils  # for commands like `ls`
  # for apt list --installed
  - dpkg
  - apt
  - perl
  # docker packages
  - docker

//apt/packages/docker.yaml:

packages:
  - docker

BUILD

load("@rules_oci//oci:defs.bzl", "oci_image")

oci_image(
    name = "base",
    architecture = select({
        "@platforms//cpu:arm64": "arm64",
        "@platforms//cpu:x86_64": "amd64",
    }),
    os = "linux",
    tars = [
        "@apt//install/bookworm",
    ],
)

oci_image(
    name = "docker",
    base = ":base",
    tars = [
        "@apt//install/docker",
    ],
)

Comments

Here's a long version of the TL;DR above with more explanations. Also, I've used folded sections to avoid a "wall of text".

apt.sources: support DEB822 .sources format

`apt.sources`: support `DEB822` `.sources` format

I think the new API should support the "new" ([since 2015! apt v1.1]) DEB822 format.

This format is the same as other Debian files (e.g. the metadata files that apt will download from the configured sources) so it should be easy to support. Also, the old one-line .list format has a few disadvantages, ends up being more verbose for complex setups and/or complicates the parsing so it would be nice to have an alternative.

Following Debian's convention re. sources file extension, I think apt.sources should receive a file and, depending on the extension use the old or new format:

//apt/sources/bookworm.sources (DEB822)::

Types: deb
Architectures: amd64 arm64
URIs: http://deb.debian.org/debian
Suites: bookworm bookworm-updates
Components: main contrib non-free non-free-firmware

Types: deb
Architectures: amd64 arm64
URIs: http://deb.debian.org/debian-security
Suites: bookworm-security
Components: main contrib non-free non-free-firmware

//apt/sources/docker.list (one-line):

deb [arch=amd64,arm64] https://download.docker.com/linux/debian bookworm stable

apt.sources(
    name = "bookworm",
    file = "//apt/sources/bookworm.sources",
)

apt.sources(
    name = "docker",
    file = "//apt/sources/docker.list",
)

apt "monorepo"

`apt` "monorepo"

Could it be possible (and would it make sense) to have everything under one @apt "monorepo" with different packages for sources, install, ...?

Maybe something like:

@apt//
    sources/
        <SOURCE_NAME>/
            <ARCH>/
                Packages.gz
            pool/
                <PACKAGE_NAME>/
                    <ARCH>/
                        <VERSION>/
                            :data
                            :control
    install/
        <INSTALL_NAME>/
            <PACKAGE_NAME>/
                <ARCH>/
                    :data --> //sources/<SOURCE_NAME>/pool/<PACKAGE_NAME>/<ARCH>/<VERSION>:data
                    :control

It would centralize access, removes potential name collisions, simplifies not having to do use_repo for every repo (or having one very long use_repo(...)). Plus I think it would also simplify the access to e.g. the resolve lock and remove the "reported incorrect imports of repositories via use_repo" warning for the _resolve repos in the tests.

And it would centralize and "mirror apt commands" like:

bazel run @apt//source/<SOURCE_NAME>:update to update the sources, downloading the latest Packages index (and, if we had some internal parsing and generation of an index lock, re-generate it).
bazel run @apt//install/<INSTALL_NAME>:lock to lock the installation (equivalent to the current bazel run @INSTALL_NAME//:lock).

`sources`

Similar to how Debian supports multiple .list and .sources files in /etc/apt/sources.list.d, we could have them all under @apt//sources:

use_repo(apt, "apt")

apt.install(
    name = "noble_core",
    sources = [
        "@apt//sources/noble",
        "@apt//sources/foo",
    ],
    ...
)

`pool`

Similar to how the Debian mirrors organize their structure, they separate the metadata from the pool of files to download. Thus, when resolving a file from a source, it could be mirrored to @apt//sources/<SOURCE_NAME>/pool/<PACKAGE_NAME>.

This would be the equivalent to the current package repo. That is, instead of having @<PACKAGE_NAME>~<VERSION>... repos, they could be namespaced under pool following the same structure as the package repos and the v2 lock that I proposed in #95: <PACKAGE_NAME>/<ARCH>/<VERSION>

`install` package

With this structure, packages are only downloaded once into the monorepo and all the installs are references to them (like currently, but instead of pointing at per-package repos it would be one @apt repo and Bazel package structure).

apt.install package list

`apt.install` package list

I think the packages section of the manifest file can still be valuable, it will definitely help un-clutter MODULES.bazel and organize stuff in medium-to-big projects.

I would still use YAML because it supports comments which can be useful when documenting why certain packages or certain constraints are used, e.g.

//apt/install/packages.yaml:

packages:
# main packages
"coreutils"
  - "dpkg"
  - "apt"
- foo # needed because XYZ
- bar
# extra packages
...

apt.install(
    ...
    packages = "//apt/install/packages.yaml",
    ...
)

apt.install lock file

`apt.install` `lock` file

Similarly, I think lock should still be part of install "command". I would probably remove nolock and default to running lockless silently, expecting users to know how to generate a lock with the bazel run command.

I'd have a separate command to update the sources though: bazel run @<INSTALL_NAME>//:lock for installs and bazel run @<SOURCE_NAME>//:update for sources, mirroring apt update.

Also, these commands can be optional / removed if #130 is implemented.

multiple install

multiple `install`

Support multiple installs, each its own repo:

apt.install(
    name = "base",
    packages = "//apt/install/base.yaml",
)

apt.install(
    name = "docker",
    packages = "//apt/install/docker.yaml",
)

use_repo(apt, "noble_core", "noble_extra")

multiple sources

multiple `sources`

Mirror /etc/apt/sources.list.d/ having many source files:

apt.sources(
    name = "bookworm",
    file = "//apt/sources/bookworm.sources",
)

apt.sources(
    name = "docker",
    file = "//apt/sources/docker.list",
)

sources repos

`sources` repos

If "no monorepo", I still think each sources should be its own repository so that we can mix-and-match sources in many installs:

apt.sources(
    name = "noble",
    ...
)
use_repo(apt, "noble")

apt.sources(
    name = "bookworm",
    ...
)
use_repo(apt, "bookworm")

apt.sources(
    "foo",
    ...
)
use_repo(apt, "foo")

apt.install(
    sources = ["@noble", "@foo"],
    ...
)

apt.install(
    sources = ["@bookworm", "@foo"],
    ...
)

architectures per-install

`architectures` per-`install`

Instead of (or in addition to) apt.configure, there could be architectures per-install:

apt.sources(
    name = "foo",
    "deb [arch=amd64,arm64] ...",
)

apt.sources(
    name = "bar",
    "deb [arch=arm64,s390x] ...",
)

apt.install(
    name = "install_amd64",
    sources = [
        "@foo",
    ],
    architectures = ["amd64"],
    packages = [
        "pkg1",
        "pkg2",
    ],
)

apt.install(
    name = "install_arm64",
    sources = [
        "@foo",
        "@bar",
    ],
    architectures = ["arm64"],
    packages = [
        "pkg2",
    ],
)

I think this could be useful, regardless of and on top of the apt install :<ARCH> syntax, which would still allow something like:

apt.install(
    name = "install_mixed",
    sources = [
        "@foo",
        "@bar",
    ],
    architectures = ["amd64", "arm64"],
    packages = [
        "pkg1:amd64",
        "pkg2",
    ],
)

jimmyt857 · 2025-01-14T17:17:47Z

All the ideas look nice! @jjmaestro, one minor suggestion for you to consider: for a really excellent developer experience, make liberal use of comments in the lockfile to include extra information.

We use uv for managing python packages, and the requirements.txt files it generates (which serve as lock files) are a good inspiration here. Example:

# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.in -o requirements.txt --override overrides.txt
...
...
colorama==0.4.6
    # via
    #   anyscale
    #   halo
    #   log-symbols
coloredlogs==15.0.1
    # via onnxruntime
colorful==0.5.6
    # via ray
comm==0.2.2
    # via
    #   ipykernel
    #   ipywidgets
contextlib2==21.6.0
    # via ml-collections
contourpy==1.3.1
    # via matplotlib
coverage==7.6.10
    # via pytest-cov
...
...

In python requirements.txt files, dependency edges are not necessary to include as real data. Having them included as comments though is incredibly helpful, making it much easier to understand and validate our dependencies. And of course having the generation command at the top is great for copy+paste. I'm sure there are other things that can be added too (source url? tar size?) to make it easy to diagnose various issues.

jjmaestro · 2025-01-15T10:43:27Z

I think this would be clear from the format of the lock itself. If you check it out, for every package there's a dependencies key with the list of dependencies for the package, so there's no need to add them as comments I think :)

thesayyn mentioned this issue Dec 10, 2024

Supporting non standard package sources #56

Open

jjmaestro mentioned this issue Dec 10, 2024

Drop the WORKSPACE API for apt. #125

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FR: An alternative API design for easier apt.install #123

FR: An alternative API design for easier apt.install #123

thesayyn commented Dec 4, 2024 •

edited

Loading

jjmaestro commented Dec 11, 2024

`apt.sources`: support `DEB822` `.sources` format

`apt` "monorepo"

`sources`

`pool`

`install` package

`apt.install` package list

`apt.install` `lock` file

multiple `install`

multiple `sources`

`sources` repos

`architectures` per-`install`

jimmyt857 commented Jan 14, 2025

jjmaestro commented Jan 15, 2025

FR: An alternative API design for easier apt.install #123

FR: An alternative API design for easier apt.install #123

Comments

thesayyn commented Dec 4, 2024 • edited Loading

jjmaestro commented Dec 11, 2024

TL;DR

Comments

apt.sources: support DEB822 .sources format

apt "monorepo"

sources

pool

install package

apt.install package list

apt.install lock file

multiple install

multiple sources

sources repos

architectures per-install

jimmyt857 commented Jan 14, 2025

jjmaestro commented Jan 15, 2025

thesayyn commented Dec 4, 2024 •

edited

Loading

`apt.sources`: support `DEB822` `.sources` format

`apt` "monorepo"

`sources`

`pool`

`install` package

`apt.install` package list

`apt.install` `lock` file

multiple `install`

multiple `sources`

`sources` repos

`architectures` per-`install`