Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shift a couple of stages from Jenkinsfile into Dockerfile #364

Merged
merged 8 commits into from
Dec 4, 2024
Merged

Shift a couple of stages from Jenkinsfile into Dockerfile #364

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
890d049
Shift a couple of stages from Jenkinsfile into Dockerfile
james-smith-za Dec 2, 2024
64816a1
Simplify dockerfile and move a few things back to Jenkinsfile
james-smith-za Dec 3, 2024
54c3944
Correct path in .ci/Dockerfile
james-smith-za Dec 3, 2024
f97db81
Correct path again
james-smith-za Dec 3, 2024
2bf3f2e
Shift creation and activation of venv back to Dockerfile
james-smith-za Dec 3, 2024
a1fe8cf
Rework for Jenkins testing
bmerry Dec 4, 2024
a8c99b4
Merge pull request #365 from ska-sa/jenkins-ci
james-smith-za Dec 4, 2024
41c50b3
Update copyright block for consistency with other files
james-smith-za Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions Dockerfile.CI
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# syntax = docker/dockerfile:1

################################################################################
# Copyright (c) 2024, National Research Foundation (SARAO)
#
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Copyright (c) 2024, National Research Foundation (SARAO)
#
# Copyright 2024 National Research Foundation (SARAO)

For consistency with other files.

#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU Lesser General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
# details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
################################################################################

FROM ubuntu:24.04

WORKDIR /tmp/spead2
COPY . .

# Install dependencies
# jq and iproute2 are used to find the interface IPv4 address
RUN apt-get update && apt-get -y --no-install-recommends install libpython3-dev python3-venv jq iproute2
RUN python3 -m venv /venv
RUN .ci/install-sys-pkgs.sh
RUN PATH="/venv/bin:$PATH" .ci/py-requirements.sh

# Install Python package
RUN PATH="/venv/bin:$PATH" pip install -v \
--config-settings=setup-args=--native-file=ci.ini \
--config-settings=setup-args=-Dibv=enabled \
--config-settings=setup-args=-Dibv_hw_rate_limit=enabled \
--config-settings=setup-args=-Dmlx5dv=enabled \
.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd prefer to just use the Dockerfile for the stuff that needs to be done as root. I feel like one gets better debuggability if the steps that work on the code under test are done in Jenkinsfile stages so that you can get a clear stage view of what happened, how long each stage took etc, rather than just "the Dockerfile failed to build".

Suggested change
RUN apt-get update && apt-get -y --no-install-recommends install libpython3-dev python3-venv jq iproute2
RUN python3 -m venv /venv
RUN .ci/install-sys-pkgs.sh
RUN PATH="/venv/bin:$PATH" .ci/py-requirements.sh
# Install Python package
RUN PATH="/venv/bin:$PATH" pip install -v \
--config-settings=setup-args=--native-file=ci.ini \
--config-settings=setup-args=-Dibv=enabled \
--config-settings=setup-args=-Dibv_hw_rate_limit=enabled \
--config-settings=setup-args=-Dmlx5dv=enabled \
.
RUN apt-get update && apt-get -y --no-install-recommends install libpython3-dev python3-venv jq iproute2
RUN .ci/install-sys-pkgs.sh

Also change COPY . . to just copy install-sys-pkgs.sh. At present any code change will invalidate the entire Dockerfile, so you're not going to get any caching.

24 changes: 2 additions & 22 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@

pipeline {
agent {
docker {
dockerfile {
label 'spead2'
filename 'Dockerfile.CI'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you take my suggestions above, the Dockerfile only needs the .ci directory, so it can move to .ci/Dockerfile and change this to

Suggested change
filename 'Dockerfile.CI'
dir '.ci'

registryCredentialsId 'dockerhub' // Supply credentials to avoid rate limit
image 'ubuntu:24.04'
// These arguments allow direct access to the NVIDIA NIC with ibverbs
args '--network=host --ulimit=memlock=-1 -e NVIDIA_MOFED=enabled -e NVIDIA_VISIBLE_DEVICES=all --runtime=nvidia'
}
Expand All @@ -42,26 +42,6 @@ pipeline {
}

stages {
stage('Install dependencies') {
steps {
// jq and iproute2 are used to find the interface IPv4 address
sh 'apt-get update && apt-get -y --no-install-recommends install libpython3-dev python3-venv jq iproute2'
sh 'python3 -m venv /venv'
sh '.ci/install-sys-pkgs.sh'
sh 'PATH="/venv/bin:$PATH" .ci/py-requirements.sh'
}
}

stage('Install Python package') {
steps {
sh '''PATH="/venv/bin:$PATH" pip install -v \
--config-settings=setup-args=--native-file=ci.ini \
--config-settings=setup-args=-Dibv=enabled \
--config-settings=setup-args=-Dibv_hw_rate_limit=enabled \
--config-settings=setup-args=-Dmlx5dv=enabled \
.'''
}
}

stage('Run tests') {
steps {
Expand Down
Loading