Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

oci_image isn't setting architecture in the image config section #117

Closed
kriswuollett opened this issue Mar 23, 2023 · 9 comments · Fixed by #172
Closed

oci_image isn't setting architecture in the image config section #117

kriswuollett opened this issue Mar 23, 2023 · 9 comments · Fixed by #172
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Milestone

Comments

@kriswuollett
Copy link

kriswuollett commented Mar 23, 2023

Haven't had a chance to debug image.sh.tpl yet, but it doesn't seem that architecture is being set in the images while an env with the values to debug this issue does show up:

WORKSPACE
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

http_archive(
    name = "rules_cc",
    sha256 = "3d9e271e2876ba42e114c9b9bc51454e379cbf0ec9ef9d40e2ae4cec61a31b40",
    strip_prefix = "rules_cc-0.0.6",
    urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.6/rules_cc-0.0.6.tar.gz"],
)

http_archive(
    name = "rules_pkg",
    sha256 = "8c20f74bca25d2d442b327ae26768c02cf3c99e93fad0381f32be9aab1967675",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_pkg/releases/download/0.8.1/rules_pkg-0.8.1.tar.gz",
        "https://github.com/bazelbuild/rules_pkg/releases/download/0.8.1/rules_pkg-0.8.1.tar.gz",
    ],
)

load("@rules_pkg//:deps.bzl", "rules_pkg_dependencies")

rules_pkg_dependencies()

git_repository(
    name = "aspect_gcc_toolchain",
    commit = "4bd1f94536ee92b7c49673931773038d923ee86e",
    remote = "https://github.com/aspect-build/gcc-toolchain",
)

load("@aspect_gcc_toolchain//toolchain:repositories.bzl", "gcc_toolchain_dependencies")

gcc_toolchain_dependencies()

load("@aspect_gcc_toolchain//toolchain:defs.bzl", "ARCHS", "gcc_register_toolchain")

gcc_register_toolchain(
    name = "gcc_toolchain_aarch64",
    target_arch = ARCHS.aarch64,
)

gcc_register_toolchain(
    name = "gcc_toolchain_armv7",
    target_arch = ARCHS.armv7,
)

gcc_register_toolchain(
    name = "gcc_toolchain_x86_64",
    sysroot_variant = "x86_64-X11",
    target_arch = ARCHS.x86_64,
)

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "dd926a88a564a9246713a9c00b35315f54cbd46b31a26d5d8fb264c07045f05d",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.38.1/rules_go-v0.38.1.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.38.1/rules_go-v0.38.1.zip",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_download_sdk", "go_register_toolchains", "go_rules_dependencies")

go_rules_dependencies()

go_download_sdk(name = "go_1_20_2_host", version = "1.20.2")
go_download_sdk(name = "go_1_20_2_linux_amd64", version = "1.20.2", goos = "linux", goarch = "amd64")
go_download_sdk(name = "go_1_20_2_linux_arm64", version = "1.20.2", goos = "linux", goarch = "arm64")

go_register_toolchains()

http_archive(
    name = "bazel_gazelle",
    sha256 = "ecba0f04f96b4960a5b250c8e8eeec42281035970aa8852dda73098274d14a1d",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.29.0/bazel-gazelle-v0.29.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.29.0/bazel-gazelle-v0.29.0.tar.gz",
    ],
)

load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")

gazelle_dependencies()

http_archive(
    name = "rules_oci",
    sha256 = "48642588e91e992772b94de06234da6601854fda0ee32a91ce8ef303cf5e5837",
    strip_prefix = "rules_oci-0.3.7",
    url = "https://github.com/bazel-contrib/rules_oci/releases/download/v0.3.7/rules_oci-v0.3.7.tar.gz",
    # TODO(https://github.com/bazel-contrib/rules_oci/issues/116) Remove once non-standard ports supported for registries
    patch_args = ["-p1"],
    patches = ["//bazel:rules_oci_registry.patch"],
)

load("@rules_oci//oci:dependencies.bzl", "rules_oci_dependencies")

rules_oci_dependencies()

load("@rules_oci//oci:repositories.bzl", "LATEST_CRANE_VERSION", "oci_register_toolchains")

oci_register_toolchains(
    name = "oci",
    crane_version = LATEST_CRANE_VERSION,
)

load("@rules_oci//oci:pull.bzl", "oci_pull")

oci_pull(
    name = "distroless_cc",
    digest = "sha256:f252d3ca44b7f2c718c67c2020feee7b7fd4ee6bff8a192dfea6e599b7d820ad",
    # TODO(https://github.com/bazel-contrib/rules_oci/issues/74) Use latest images
    # digest = "sha256:fb402c45f3ef485ccd56ca2af2a58615fc47c4978bb3004e8663a83456791f48",
    image = "gcr.io/distroless/cc",
    platforms = [
        "linux/amd64",
        "linux/arm64",
    ],
)
BUILD.bazel
load("@aspect_bazel_lib//lib:transitions.bzl", "platform_transition_filegroup")
load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
load("@rules_pkg//pkg:tar.bzl", "pkg_tar")
load("@rules_oci//oci:defs.bzl", "oci_image", "oci_push", "oci_image_index")

go_library(
    name = "hello_lib",
    srcs = ["hello.go"],
    importpath = "example.com/go/hello",
    visibility = ["//visibility:private"],
)

go_binary(
    name = "hello",
    embed = [":hello_lib"],
    visibility = ["//visibility:public"],
)

PLATFORMS = {
    "linux_x86_64": {"arch": "amd64"},
    "linux_aarch64": {"arch": "arm64"},
}

[platform_transition_filegroup(
    name = "hello_{}".format(platform),
    srcs = [":hello"],
    target_platform = "//platforms:{}".format(platform),
) for platform in PLATFORMS]

[pkg_tar(
    name = "hello_tar_{}".format(platform),
    srcs = [":hello_{}".format(platform)],
    include_runfiles = True,
    strip_prefix = "/",
) for platform in PLATFORMS]

[oci_image(
    name = "hello_image_{}".format(platform),
    architecture = "{}".format(attrs["arch"]),
    os = "linux",
    base = "@distroless_cc",
    entrypoint = ["/examples/go/hello/hello_/hello"],
    tars = [
        ":hello_tar_{}".format(platform),
    ],
    env = {
        "image_arch": "{}".format(attrs["arch"]),
        "image_os": "linux",
    },
) for platform, attrs in PLATFORMS.items()]

[oci_push(
    name = "hello_image_push_{}".format(platform),
    image = ":hello_image_{}".format(platform),
    repository = "localhost:5000/examples/go/hello",
    repotags = ["latest_{}".format(platform)],
) for platform, attrs in PLATFORMS.items()]

oci_image_index(
    name = "hello_image_linux",
    images = [
        ":hello_image_{}".format(platform)
        for platform in PLATFORMS
    ],
)

oci_push(
    name = "hello_image_linux_push",
    image = ":hello_image_linux",
    repository = "localhost:5000/examples/go/hello",
    repotags = ["latest"],
)

Note how architecture is amd64, but Env.image_arch env was set to arm64 using regctl image inspect localhost:5000/examples/go/hello:latest_linux_aarch64:

{
  "created": "1970-01-01T00:00:00Z",
  "author": "Bazel",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "User": "0",
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/busybox",
      "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt",
      "image_os=linux",
      "image_arch=arm64"
    ],
    "Entrypoint": [
      "/examples/go/hello/hello_/hello"
    ],
    "WorkingDir": "/"
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d",
      "sha256:f38a81323cd8a2d7636775c206e494eb1cdad64d2b008967be69a134eb4a736b",
      "sha256:6998679213707b9968867d5627bfa0f77da0baad13d2120ce41d0606c8f9d591",
      "sha256:e83d4114481dc897d49d5a9a8b68bf71c4f08f1a5c1adff44603c56b958fed53",
      "sha256:be43041c1d44a317622939fc256141b51ece61fa5264aca8cf561ee331f9bba0"
    ]
  },
  "history": [
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "bazel build ...",
      "author": "Bazel"
    },
    {
      "created": "0001-01-01T00:00:00Z"
    }
  ]
}

Using bazel aquery it can be seen that the platform is being properly passed for use by crane:

  Command Line: (exec bazel-out/k8-fastbuild/bin/examples/go/hello/image_hello_image_linux_aarch64.sh \
    mutate \
    oci:layout/bazel-out/k8-fastbuild/bin/external/distroless_cc_linux_amd64/layout \
    --tag \
    oci:registry/hello_image_linux_aarch64 \
    '--platform=linux/arm64' \
    '--append=bazel-out/k8-fastbuild/bin/examples/go/hello/hello_tar_linux_aarch64.tar' \
    --entrypoint \
    /examples/go/hello/hello_/hello \
    '--env=image_arch=arm64' \
    '--env=image_os=linux' \
    '--output=bazel-out/k8-fastbuild/bin/examples/go/hello/hello_image_linux_aarch64')

Despite its actual contents, Docker will refuse to run it for linux/arm64:

$ docker run --platform linux/arm64 --rm localhost:5000/examples/go/hello:latest_linux_aarch64
Unable to find image 'localhost:5000/examples/go/hello:latest_linux_aarch64' locally
latest_linux_aarch64: Pulling from examples/go/hello
fc251a6e7981: Already exists 
7be4d3667295: Already exists 
3508a3839ffc: Already exists 
a1f1879bb7de: Already exists 
253e4cd162a9: Already exists 
Digest: sha256:19a4c2699e4313ca91e47081f44cf913beb2b6f09ad8eb673cd7aa15f0d6292e
Status: Downloaded newer image for localhost:5000/examples/go/hello:latest_linux_aarch64
WARNING: image with reference localhost:5000/examples/go/hello was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64
docker: Error response from daemon: image with reference localhost:5000/examples/go/hello:latest_linux_aarch64 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.
See 'docker run --help'.
@kriswuollett
Copy link
Author

Confirmed in my example that the image content was arm64 by using crane to fix the config generated from oci_image.

hello.go:

package main

import (
	"fmt"
	"runtime"
)

func main() {
	println(fmt.Sprintf("Hello, from %s/%s!", runtime.GOOS, runtime.GOARCH))
}

Setup:

sudo apt-get install qemu binfmt-support qemu-user-static
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
REF=localhost:5000/examples/go/hello:latest_linux_aarch64
crane config "$REF" | jq '.architecture = "arm64"' | crane edit config "$REF"

Tests:

$ bazel run //examples/go/hello
INFO: Invocation ID: 5bdc5b4a-787a-4550-9e66-50e9d3d71a82
INFO: Analyzed target //examples/go/hello:hello (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //examples/go/hello:hello up-to-date:
  bazel-bin/examples/go/hello/hello_/hello
INFO: Elapsed time: 0.145s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/examples/go/hello/hello_/hello
Hello, from linux/amd64!
$ docker run --rm --platform linux/arm64  localhost:5000/examples/go/hello:latest_linux_aarch64
Hello, from linux/arm64!
$ docker run --rm localhost:5000/examples/go/hello:latest_linux_x86_64
Hello, from linux/amd64!

@thesayyn
Copy link
Collaborator

When you got a base image in your oci_image, oci_image automatically uses base image's arch and os. basically, os, arch, variant is no-op when you have got a base image.
The reason is that if you choose to override base image's os and arch then base image might contain binaries that are only compatible with overridden os, arch. Does that make sense?

This is a silent error we should handle better. either we say that the base image's os and arch doesn't match the specified os, arch attribute or error out when both os, arch and base attribute are specified.

@kriswuollett
Copy link
Author

kriswuollett commented Mar 24, 2023

The explanation sort of makes sense, but the devex doesn't. So I'm assuming the image example only works since it is statically linked and doesn't depend on anything else?

AFAICT, I pointed to a multi-arch (Docker list style due to other issue since distroless went OCI in March). My expectation is that oci_image would use os and arch to transition the base dependency to that os/arch -- the tar content already was transitioned. Maybe oci_image should have platform parameter instead of os/architecture to do all transitions for me? I don't think I should have to multiple oci_pull single os/arch images?

@thesayyn
Copy link
Collaborator

i agree about the DX point. since there is no transition on oci_image (we did not implement it because other various issues around optional toolchains and cc toolchains. DX is horrible as of writing this), the select statement in oci_pull basically no-op unless --platforms flag is passed

@thesayyn thesayyn added documentation Improvements or additions to documentation enhancement New feature or request labels Mar 24, 2023
@thesayyn
Copy link
Collaborator

thesayyn commented Mar 24, 2023

I don't think I should have to multiple oci_pull single os/arch images?

oci_pull#platforms already expands to multiple oci_pull rules. basically an external repository per platform. naming pattern is <name>_<os>_<arch>. you can run bazel query //external:* and look for repos that start with name of oci_pull call.

@kriswuollett
Copy link
Author

Ah, thanks! Was just about to ask the question for that -- I can see //external:distroless_base_linux_arm64 for example. I think I recalled seeing those labels in error messages when I had something else set up incorrectly.

@kriswuollett
Copy link
Author

My updated target's pushed images work without modification now, although I haven't tried any binaries that are dynamically linked to distro libraries yet:

[oci_image(
    name = "hello_image_{}".format(platform),
    architecture = "{}".format(attrs["arch"]),
    base = "@distroless_base_linux_{}".format(attrs["arch"]),
    entrypoint = ["/examples/go/hello/hello_/hello"],
    env = {
        "image_arch": "{}".format(attrs["arch"]),
        "image_os": "linux",
    },
    os = "linux",
    tars = [
        ":hello_tar_{}".format(platform),
    ],
) for platform, attrs in PLATFORMS.items()]

Yes, some documentation about the multi-image labels that could be used from oci_pull would be great. And if possible one day, oci_image handling transitions would be cool. :-)

@michaelschuett-tomtom
Copy link

michaelschuett-tomtom commented May 10, 2024

Sorry to resurrect this one. It doesn't seem that the information here is correct still or maybe i am just doing it wrong but I don't see those repos being created with the _linux_{platform} prefix anymore when using oci_pull with platforms. Do any docs exist on how to achieve this today?

Follow up:
#228
https://github.com/bazel-contrib/rules_oci/tree/main/examples/multi_arch

these look to be the way you have to do this today.

@thesayyn
Copy link
Collaborator

thesayyn commented Jun 8, 2024

@michaelschuett-tomtom we still generate platform specific repos for multi platform images.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants