Standardize development dependencies / refactor GHA workflow (#153)

* Standardize development dependencies

There seems to be a preference towards using `requirements-dev.txt` vs. a `dev` extra for development dependencies, but right now neither `pip install -r requirements-dev.txt` nor `pip install -e ".[dev]" is enough to install all development dependencies.

I’ve added any dependencies listed in the `dev` extra to `requirements-dev.txt` and removed the `dev` extra so that we only have one list of dev dependencies to maintain.

This also pins the `moto` package (used to mock S3 in tests) to version 4.x as version 5 release a few days ago contains breaking changes.

* Remove unnecessary setup in GitHub Actions workflow

This seems to be left over from when we still used to run tests directly in GHA runners.

* Install servicelayer before running tests

Tests in `test_extensions.py` rely on `servicelayer` being installed as they ultimately read information from the egg info that is created during installation.

This is basically equivalent to the previous setup which executed `pip install -e .` outside of the container. As the entire directory is mounted into the container, the egg info subsequently was also available inside of the container. While that worked I found the fact that installing something outside of the container could make the tests fail or pass quite confusing. This should be a little more explicit.

* Run everything inside of the docker container

Running some steps outside and some inside of the container leads to weird behavior, e.g. when file permissions/owners do not match across steps.

* Slim down Docker image

This uses a pre-built Python image based on Debian as this should be much faster than building a custom image on top of a full-feature Ubuntu.

* Add separate Makefile target to run tests without starting a Docker container

.github/workflows/build.yml

@@ -8,34 +8,20 @@ jobs:
     steps:
     - uses: actions/checkout@v1
     - name: Show ref
+      run: echo "$GITHUB_REF"
+    - name: Build/pull Docker images
       run: |
-        echo "$GITHUB_REF"
-    - name: Set up Python
-      uses: actions/setup-python@v1
-      with:
-        python-version: '3.x'
-    - name: Install dependencies
-      env:
-        DEBIAN_FRONTEND: noninteractive
-        BOTO_CONFIG: /dev/null
-        AWS_ACCESS_KEY_ID: banana
-        AWS_SECRET_ACCESS_KEY: bananacake
-      run: |
-        sudo rm -f /etc/boto.cfg
-        sudo apt-get -qq update
-        sudo apt-get install -y libicu-dev
-        make dev
-        pip install -e ".[dev]"
+        docker compose build shell
+        docker compose pull rabbitmq
     - name: Run the code format check
-      run: make format-check
+      run: docker compose run --rm shell make format-check
     - name: Run the linter
-      run: make lint
+      run: docker compose run --rm shell make lint
     - name: Run the tests
-      run: |
-        make test
+      # Some tests rely on the package being installed
+      run: docker compose run --rm shell bash -c "make install && make test-local"
     - name: Build a distribution
-      run: |
-        python setup.py sdist bdist_wheel
+      run: docker compose run --rm shell python3 setup.py sdist bdist_wheel
     - name: Publish a Python distribution to PyPI
       if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
       uses: pypa/gh-action-pypi-publish@release/v1

Dockerfile

@@ -1,20 +1,10 @@
-FROM ubuntu:20.04
-ENV DEBIAN_FRONTEND noninteractive
+FROM python:3.12-slim
 
-RUN apt-get -qq -y update \
-    && apt-get -qq -y install python3-pip \
-    pkg-config libicu-dev \
-    && apt-get -qq -y autoremove \
-    && apt-get clean \
-    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-RUN pip3 install --no-binary=:pyicu: pyicu
-
-ENV LANG='en_US.UTF-8'
+RUN apt-get update
+RUN apt-get install make
 
 COPY . /opt/servicelayer
 WORKDIR /opt/servicelayer
-RUN pip3 install -q --no-cache-dir -e /opt/servicelayer[dev]
-RUN pip3 install -r requirements.txt
+RUN make dev
 
-CMD /bin/bash
+CMD /bin/bash

Makefile

@@ -8,15 +8,17 @@ build-docker:
 
 install:
 	pip install -q -e .
-	pip install -q twine coverage nose moto boto3
 
 dev:
 	python3 -m pip install --upgrade pip setuptools
 	python3 -m pip install -q -r requirements.txt
 	python3 -m pip install -q -r requirements-dev.txt
 
 test:
-	docker-compose run --rm shell pytest --cov=servicelayer
+	docker compose run --rm shell make test-local
+
+test-local:
+	pytest --cov=servicelayer
 
 lint:
 	ruff check .

requirements-dev.txt

@@ -5,3 +5,6 @@ pytest-env==1.1.3
 pytest-cov==5.0.0
 pytest-mock==3.14.0
 wheel==0.43.0
+twine==4.0.2
+moto==4.2.14
+boto3==1.34.32

setup.py

@@ -44,14 +44,6 @@
             "grpcio >= 1.32.0, <2.0.0",
             "google-cloud-storage >= 1.31.0, < 3.0.0",
         ],
-        "dev": [
-            "twine",
-            "moto < 5",
-            "boto3 >= 1.11.9, <2.0.0",
-            "pytest >= 3.6",
-            "coverage",
-            "pytest-cov",
-        ],
     },
     test_suite="tests",
     entry_points={

tests/test_extensions.py

@@ -5,6 +5,8 @@
 
 class ExtensionTest(TestCase):
     def test_extensions(self):
+        # This relies on the servicelayer package being installed as `get_extensions`
+        # ultimately reads entrypoints from the egg info
         exts = get_extensions("servicelayer.test")
         assert len(exts), exts
         assert get_extensions in exts, exts