Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker: moves everything to consistent docker base layer #3613

Merged
merged 3 commits into from
Dec 5, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ GenericContainer<?> createZipkinContainer(@Nullable GenericContainer<?> storage)

final GenericContainer<?> zipkin;
if (RELEASE_VERSION == null) {
zipkin = new GenericContainer<>(parse("ghcr.io/openzipkin/java:17.0.8_p7"));
zipkin = new GenericContainer<>(parse("ghcr.io/openzipkin/java:17.0.9_p8"));
List<String> classpath = new ArrayList<>();
for (String item : System.getProperty("java.class.path").split(File.pathSeparator)) {
Path path = Paths.get(item);
Expand Down
121 changes: 14 additions & 107 deletions build-bin/README.md
100755 → 100644
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ On deploy:
`build-bin` holds portable scripts used in CI to test and deploy the project.

The scripts here are portable. They do not include any CI provider-specific logic or ENV variables.
This helps `.travis.yml` and `test.yml` (GitHub Actions) contain nearly the same contents, even if
This helps `test.yml` (GitHub Actions) and alternatives contain nearly the same contents, even if
certain OpenZipkin projects need slight adjustments here. Portability has proven necessary, as
OpenZipkin has had to transition CI providers many times due to feature and quota constraints.

Expand Down Expand Up @@ -66,9 +66,6 @@ the scripts it uses.
The `on:` section obviates job creation and resource usage for irrelevant events. Notably, GitHub
Actions includes the ability to skip documentation-only jobs.

Combine [configure_test] and [test] into the same `run:` when `configure_test` primes file system
cache.

Here's a partial `test.yml` including only the aspects mentioned above.
```yaml
on:
Expand All @@ -84,54 +81,13 @@ jobs:
test:
steps:
- name: Checkout Repository
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
fetch-depth: 0 # full git history
fetch-depth: 0 # full git history for license check
- name: Test
run: build-bin/configure_test && build-bin/test
```

### Example Travis setup
`.travis.yml` is a monolithic configuration file broken into stages, of which the default is "test".
A simplest Travis `test` job configures tests in `install` and runs them as `script`, but only on
relevant event conditions.

The `if:` section obviates job creation and resource usage for irrelevant events. Travis does not
support file conditions. A `before_install` step to skip documentation-only commits will likely
complete in less than a minute (10 credit cost).

Here's a partial `.travis.yml` including only the aspects mentioned above.
```yaml
git:
depth: false # TRAVIS_COMMIT_RANGE requires full commit history.

jobs:
include:
- stage: test
if: branch = master AND tag IS blank AND type IN (push, pull_request)
name: Run unit and integration tests
before_install: | # Prevent test build of a documentation-only change.
if [ -n "${TRAVIS_COMMIT_RANGE}" ] && ! git diff --name-only "${TRAVIS_COMMIT_RANGE}" -- | grep -qv '\.md$'; then
echo "Stopping job as changes only affect documentation (ex. README.md)"
travis_terminate 0
fi
install: ./build-bin/configure_test
script: ./build-bin/test
```

When Travis only runs tests (something else does deploy), there's no need to use stages:
```yaml
git:
depth: false # TRAVIS_COMMIT_RANGE requires full commit history.

if: branch = master AND tag IS blank AND type IN (push, pull_request)
before_install: | # Prevent test build of a documentation-only change.
if [ -n "${TRAVIS_COMMIT_RANGE}" ] && ! git diff --name-only "${TRAVIS_COMMIT_RANGE}" -- | grep -qv '\.md$'; then
echo "Stopping job as changes only affect documentation (ex. README.md)"
travis_terminate 0
fi
install: ./build-bin/configure_test
script: ./build-bin/test
run: |
build-bin/configure_test
build-bin/test
```

## Deploy
Expand All @@ -154,77 +110,28 @@ The `on:` section obviates job creation and resource usage for irrelevant events
cannot implement "master, except documentation only-commits" in the same file. Hence, deployments of
master will happen even on README change.

Combine [configure_deploy] and [deploy] into the same `run:` when `configure_deploy` primes file
system cache.

Here's a partial `deploy.yml` including only the aspects mentioned above. Notice env variables are
explicitly defined and `on.tags` is a [glob pattern](https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet).

```yaml
on:
push:
tags: '[0-9]+.[0-9]+.[0-9]+**' # Ex. 8.272.10 or 17.0.8_p7
tags: '[0-9]+.[0-9]+.[0-9]+**' # e.g. 8.272.10 or 15.0.1_p9
branches: master

jobs:
deploy:
steps:
- name: Checkout Repository
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
fetch-depth: 1 # only needed to get the sha label
- name: Deploy
- name: Configure Deploy
run: build-bin/configure_deploy
env:
GH_USER: ${{ secrets.GH_USER }}
GH_TOKEN: ${{ secrets.GH_TOKEN }}
run: | # GITHUB_REF will be refs/heads/master or refs/tags/MAJOR.MINOR.PATCH
build-bin/configure_deploy &&
build-bin/deploy $(echo ${GITHUB_REF} | cut -d/ -f 3)
```

### Example Travis setup
`.travis.yml` is a monolithic configuration file broken into stages. This means `test` and `deploy`
are in the same file. A simplest Travis `deploy` stage has two jobs: one for master pushes and
another for version tags. These jobs are controlled by event conditions.

The `if:` section obviates job creation and resource usage for irrelevant events. Travis does not
support file conditions. A `before_install` step to skip documentation-only commits will likely
complete in less than a minute (10 credit cost).

As billing is by the minute, it is most cost effective to combine test and deploy on master push.

Here's a partial `.travis.yml` including only the aspects mentioned above. Notice YAML anchors work
in Travis and `tag =~` [condition](https://github.com/travis-ci/travis-conditions) is a regular
expression.
```yaml
git:
depth: false # full git history for license check, and doc-only skipping

_terminate_if_only_docs: &terminate_if_only_docs |
if [ -n "${TRAVIS_COMMIT_RANGE}" ] && ! git diff --name-only "${TRAVIS_COMMIT_RANGE}" -- | grep -qv '\.md$'; then
echo "Stopping job as changes only affect documentation (ex. README.md)"
travis_terminate 0
fi

jobs:
include:
- stage: test
if: branch = master AND tag IS blank AND type IN (push, pull_request)
before_install: *terminate_if_only_docs
install: |
if [ "${TRAVIS_SECURE_ENV_VARS}" = "true" ] && [ "${TRAVIS_PULL_REQUEST}" = "false" ]; then
export SHOULD_DEPLOY=true
./build-bin/configure_deploy
else
export SHOULD_DEPLOY=false
./build-bin/configure_test
fi
script:
- ./build-bin/test || travis_terminate 1
- if [ "${SHOULD_DEPLOY}" != "true" ]; then travis_terminate 0; fi
- travis_wait ./build-bin/deploy master
- stage: deploy
# Ex. 8.272.10 or 17.0.8_p7
if: tag =~ /^[0-9]+\.[0-9]+\.[0-9]+/ AND type = push AND env(GH_TOKEN) IS present
install: ./build-bin/configure_deploy
script: ./build-bin/deploy ${TRAVIS_TAG}
- name: Deploy
# GITHUB_REF will be refs/heads/master or refs/tags/1.2.3
run: build-bin/deploy $(echo ${GITHUB_REF} | cut -d/ -f 3)
```
7 changes: 5 additions & 2 deletions build-bin/docker/configure_docker_push
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/sh
#
# Copyright 2015-2020 The OpenZipkin Authors
# Copyright 2015-2021 The OpenZipkin Authors
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
Expand Down Expand Up @@ -50,5 +50,8 @@ fi
# See https://github.com/multiarch/qemu-user-static
#
# Mirrored image use to avoid docker.io pulls:
# docker tag multiarch/qemu-user-static:5.1.0-7 ghcr.io/openzipkin/multiarch-qemu-user-static:latest
# docker tag multiarch/qemu-user-static:7.2.0-1 ghcr.io/openzipkin/multiarch-qemu-user-static:latest
#
# Note: This image only works on x86_64/amd64 architecture.
# See: https://github.com/multiarch/qemu-user-static#supported-host-architectures
docker run --rm --privileged ghcr.io/openzipkin/multiarch-qemu-user-static --reset -p yes
5 changes: 5 additions & 0 deletions build-bin/docker/docker_arch
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
set -ue

# Normalize docker_arch to what's available
#
# Note: s390x and ppc64le were added for Knative
docker_arch=${DOCKER_ARCH:-$(uname -m)}
case ${docker_arch} in
amd64* )
Expand All @@ -38,6 +40,9 @@ case ${docker_arch} in
s390x* )
docker_arch=s390x
;;
ppc64le* )
docker_arch=ppc64le
;;
* )
>&2 echo "Unsupported DOCKER_ARCH: ${docker_arch}"
exit 1;
Expand Down
12 changes: 6 additions & 6 deletions build-bin/docker/docker_args
Original file line number Diff line number Diff line change
Expand Up @@ -46,21 +46,21 @@ if [ -n "${DOCKER_TARGET}" ]; then
fi

# When non-empty, becomes the base layer including tag appropriate for the image being built.
# Ex. ghcr.io/openzipkin/java:17.0.8_p7-jre
# e.g. ghcr.io/openzipkin/java:21.0.1_p12-jre
#
# This is not required to be a base (FROM scratch) image like ghcr.io/openzipkin/alpine:3.14.2
# This is not required to be a base (FROM scratch) image like ghcr.io/openzipkin/alpine:3.12.3
# See https://docs.docker.com/glossary/#parent-image
if [ -n "${DOCKER_PARENT_IMAGE}" ]; then
docker_args="${docker_args} --build-arg docker_parent_image=${DOCKER_PARENT_IMAGE}"
fi

# When non-empty, becomes the build-arg alpine_version. Ex. "3.18.2"
# When non-empty, becomes the build-arg alpine_version. e.g. "3.12.3"
# Used to align base layers from https://github.com/orgs/openzipkin/packages/container/package/alpine
if [ -n "${ALPINE_VERSION}" ]; then
docker_args="${docker_args} --build-arg alpine_version=${ALPINE_VERSION}"
fi

# When non-empty, becomes the build-arg java_version. Ex. "17.0.8_p7"
# When non-empty, becomes the build-arg java_version. e.g. "21.0.1_p12"
# Used to align base layers from https://github.com/orgs/openzipkin/packages/container/package/java
if [ -n "${JAVA_VERSION}" ]; then
docker_args="${docker_args} --build-arg java_version=${JAVA_VERSION}"
Expand All @@ -69,13 +69,13 @@ if [ -n "${JAVA_VERSION}" ]; then
java_major_version=$(echo ${JAVA_VERSION}| cut -f1 -d .)
case ${java_major_version} in
8) java_home=/usr/lib/jvm/java-1.8-openjdk;;
1?) java_home=/usr/lib/jvm/java-${java_major_version}-openjdk;;
1?|2?|3?) java_home=/usr/lib/jvm/java-${java_major_version}-openjdk;;
esac

if [ -n "${java_home}" ]; then docker_args="${docker_args} --build-arg java_home=${java_home}"; fi
fi

# When non-empty, becomes the build-arg maven_classifier. Ex. "module" or "exec"
# When non-empty, becomes the build-arg maven_classifier. e.g. "module" or "exec"
# Used as the classifier arg to ./build-bin/maven/maven_unjar. Allows building two images with the
# same Dockerfile, varying on classifier, like openzipkin/zipkin vs openzipkin/zipkin-slim
if [ -n "${MAVEN_CLASSIFIER}" ]; then
Expand Down
2 changes: 1 addition & 1 deletion build-bin/docker/docker_build
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ version=${2:-}
docker_args=$($(dirname "$0")/docker_args ${version})

echo "Building image ${docker_tag}"
DOCKER_BUILDKIT=1 docker build --pull ${docker_args} --tag ${docker_tag} .
DOCKER_BUILDKIT=1 docker build --network=host --pull ${docker_args} --tag ${docker_tag} .
4 changes: 2 additions & 2 deletions build-bin/docker/docker_push
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,8 @@ for repo in ${docker_repos}; do
done

docker_args=$($(dirname "$0")/docker_args ${version})
docker_archs=${DOCKER_ARCHS:-amd64 arm64}
# Note: s390x and ppc64le were added for Knative
docker_archs=${DOCKER_ARCHS:-amd64 arm64 s390x ppc64le}

echo "Will build the following architectures: ${docker_archs}"

Expand Down Expand Up @@ -114,4 +115,3 @@ else
docker manifest push -p ${tag}
done
fi

7 changes: 7 additions & 0 deletions build-bin/docker_push
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,18 @@ case ${version} in
;;
esac

# Don't attempt ppc64le until there's a package for recent LTS versions.
# See https://github.com/openzipkin/zipkin/issues/3443
export DOCKER_ARCHS="amd64 arm64 s390x"

build-bin/docker/docker_push openzipkin/zipkin ${version}
DOCKER_TARGET=zipkin-slim build-bin/docker/docker_push openzipkin/zipkin-slim ${version}

# testing images only push to ghcro.io
export DOCKER_RELEASE_REPOS=ghcr.io
# Don't attempt unfamiliar archs on test images
export DOCKER_ARCHS="amd64 arm64"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@llinder fyi, this is the maybe non-obvious trick to reduce arch for some images


for name in $(ls docker/test-images/*/Dockerfile|cut -f3 -d/); do
DOCKER_FILE=docker/test-images/${name}/Dockerfile build-bin/docker/docker_push openzipkin/${name} ${version}
done
2 changes: 1 addition & 1 deletion build-bin/maven/maven_unjar
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ fi

if ! test -f ${artifact_id}.jar && [ ${is_release} = "true" ]; then
mvn_get="mvn -q --batch-mode -Denforcer.fail=false \
org.apache.maven.plugins:maven-dependency-plugin:3.1.2:get \
org.apache.maven.plugins:maven-dependency-plugin:3.6.1:get \
-Dtransitive=false -DgroupId=${group_id} -DartifactId=${artifact_id} -Dversion=${version}"

if [ -n "${classifier}" ]; then
Expand Down
4 changes: 2 additions & 2 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright 2015-2021 The OpenZipkin Authors
# Copyright 2015-2023 The OpenZipkin Authors
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
Expand All @@ -17,7 +17,7 @@
# Use latest version here: https://github.com/orgs/openzipkin/packages/container/package/java
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG java_version=17.0.8_p7
ARG java_version=17.0.9_p8

# We copy files from the context into a scratch container first to avoid a problem where docker and
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally.
Expand Down
2 changes: 1 addition & 1 deletion docker/test-images/zipkin-cassandra/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
# Use latest version here: https://github.com/orgs/openzipkin/packages/container/package/java
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG java_version=17.0.8_p7
ARG java_version=17.0.9_p8

# We copy files from the context into a scratch container first to avoid a problem where docker and
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally.
Expand Down
2 changes: 1 addition & 1 deletion docker/test-images/zipkin-cassandra/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ cat > pom.xml <<-'EOF'
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.30</version>
<version>1.7.36</version>
</dependency>
</dependencies>
</project>
Expand Down
8 changes: 4 additions & 4 deletions docker/test-images/zipkin-elasticsearch6/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright 2015-2021 The OpenZipkin Authors
# Copyright 2015-2023 The OpenZipkin Authors
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
Expand All @@ -17,7 +17,7 @@
# Use latest version here: https://github.com/orgs/openzipkin/packages/container/package/java
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG java_version=17.0.8_p7
ARG java_version=17.0.9_p8

# We copy files from the context into a scratch container first to avoid a problem where docker and
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally.
Expand All @@ -35,7 +35,7 @@ WORKDIR /install
# Use latest 6.x version from https://www.elastic.co/downloads/past-releases#elasticsearch
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG elasticsearch6_version=6.8.13
ARG elasticsearch6_version=6.8.23

# Download only the OSS distribution (lacks X-Pack)
RUN \
Expand All @@ -49,7 +49,7 @@ COPY --from=scratch /config/ ./config/

FROM ghcr.io/openzipkin/java:${java_version}-jre as zipkin-elasticsearch6
LABEL org.opencontainers.image.description="Elasticsearch OSS distribution on OpenJDK and Alpine Linux"
ARG elasticsearch6_version=6.8.13
ARG elasticsearch6_version=6.8.23
LABEL elasticsearch-version=$elasticsearch6_version

# Add HEALTHCHECK and ENTRYPOINT scripts into the default search path
Expand Down
6 changes: 3 additions & 3 deletions docker/test-images/zipkin-elasticsearch7/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
# Use latest version here: https://github.com/orgs/openzipkin/packages/container/package/java
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG java_version=17.0.8_p7
ARG java_version=17.0.9_p8

# We copy files from the context into a scratch container first to avoid a problem where docker and
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally.
Expand All @@ -35,7 +35,7 @@ WORKDIR /install
# Use latest 7.x version from https://www.elastic.co/downloads/past-releases#elasticsearch
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG elasticsearch7_version=7.10.1
ARG elasticsearch7_version=7.17.15

# Download only the OSS distribution (lacks X-Pack)
RUN \
Expand All @@ -49,7 +49,7 @@ COPY --from=scratch /config/ ./config/

FROM ghcr.io/openzipkin/java:${java_version}-jre as zipkin-elasticsearch7
LABEL org.opencontainers.image.description="Elasticsearch OSS distribution on OpenJDK and Alpine Linux"
ARG elasticsearch7_version=7.10.0
ARG elasticsearch7_version=7.17.15
LABEL elasticsearch-version=$elasticsearch7_version

# Add HEALTHCHECK and ENTRYPOINT scripts into the default search path
Expand Down
Loading
Loading