Shift a couple of stages from Jenkinsfile into Dockerfile

[james-smith-za]

The Jenkins job on our new deployment is running Docker containers with the option -u 1000:1000 instead of -u 0:0 which results in permissions issues when doing administrative things such as using apt-get to install stuff.

So this commit moves the install stages of the test run into a Dockerfile, so that the image which is all nicely configured can then be used to run the tests.

Relates to NGC-1447.

[james-smith-za]

This job is working on the Jenkins test instance barring the fact that the agent doesn't have multicast loopback (which I haven't configured yet on the test agent), so I guess it's a good sign?

[james-smith-za]

The test passes on old Jenkins though so that's helpful.

[bmerry]

The linting checks are failing on Github Actions.

[bmerry]

Suggested change

	# Copyright (c) 2024, National Research Foundation (SARAO)
	#
	# Copyright 2024 National Research Foundation (SARAO)

For consistency with other files.

[bmerry]

I think I'd prefer to just use the Dockerfile for the stuff that needs to be done as root. I feel like one gets better debuggability if the steps that work on the code under test are done in Jenkinsfile stages so that you can get a clear stage view of what happened, how long each stage took etc, rather than just "the Dockerfile failed to build".

Suggested change

	RUN apt-get update && apt-get -y --no-install-recommends install libpython3-dev python3-venv jq iproute2
	RUN python3 -m venv /venv
	RUN .ci/install-sys-pkgs.sh
	RUN PATH="/venv/bin:$PATH" .ci/py-requirements.sh
	# Install Python package
	RUN PATH="/venv/bin:$PATH" pip install -v \
	--config-settings=setup-args=--native-file=ci.ini \
	--config-settings=setup-args=-Dibv=enabled \
	--config-settings=setup-args=-Dibv_hw_rate_limit=enabled \
	--config-settings=setup-args=-Dmlx5dv=enabled \
	.
	RUN apt-get update && apt-get -y --no-install-recommends install libpython3-dev python3-venv jq iproute2
	RUN .ci/install-sys-pkgs.sh

Also change COPY . . to just copy install-sys-pkgs.sh. At present any code change will invalidate the entire Dockerfile, so you're not going to get any caching.

[bmerry]

If you take my suggestions above, the Dockerfile only needs the .ci directory, so it can move to .ci/Dockerfile and change this to

Suggested change

	filename 'Dockerfile.CI'
	dir '.ci'

[james-smith-za]

Fair points all round, and I have addressed your feedback as far as I am able. I am now running into issues with files not being found in meson:

22:19:11    + meson setup /var/jenkins/workspace/ead2_ngc-1447-jenkins-dockerfile /var/jenkins/workspace/ead2_ngc-1447-jenkins-dockerfile/.mesonpy-4tmu33hl -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md -Dtools=disabled -Dunit_test=disabled -Dcuda=disabled -Dgdrapi=disabled -Dcap=disabled -Dpython=true --native-file=ci.ini -Dibv=enabled -Dibv_hw_rate_limit=enabled -Dmlx5dv=enabled --native-file=/var/jenkins/workspace/ead2_ngc-1447-jenkins-dockerfile/.mesonpy-4tmu33hl/meson-python-native-file.ini
22:19:11    Could not find any valid candidate for native files: ci.ini
22:19:11  
22:19:11    ERROR: Cannot find specified native file: /var/jenkins/workspace/ead2_ngc-1447-jenkins-dockerfile/.mesonpy-4tmu33hl/meson-python-native-file.ini
22:19:11    error: subprocess-exited-with-error

see here.

I'm not quite clear as to which file is missing though: it looks as though it's complaining both about the ci.ini file and the /var/jenkins/..../meson-python-native-file.ini. The former looks to me as though it gets explicitly created somewhere but the latter I'm not sure about.

[bmerry]

I'll investigate.

[bmerry]

My guess would be that it's because install-sys-pkgs.sh is creating the ci.ini in $HOME, but $HOME during the docker build probably doesn't match the $HOME during the test.

I think let me see if I can rework things to write the ci.ini in the workspace rather than somewhere in $HOME, since I'm not sure we can rely on having a $HOME in the Jenkins case (particularly if the agent doesn't use UID 1000).

[bmerry]

Looks good, just some corrections to the copyright block. Feel free to merge after fixing those.