Fix CI and the way dependencies are specified

test and dev are extras for installing tools, no multiple requirements files scattered across the repo. CI workflows should not be installing multiple distinct applications into a single venv and then trying to test. This muddies the waters and makes the purpose and requirements of each CI step unclear. Therefore, in addition to putting the requirement data where it really belongs, this fixes the workflows to specify a better isolated test configuration. Workflow jobs are used more thoughtfully to produce isolated VMs for distinct build and testing processes. New daily tests run Monday-through-Friday in the AM for US timezones, doing `safety check` and little else. Safety checking on dependencies is done separately for funcx-sdk and funcx-endpoint. The CI workflow for testing PRs and branches also does `safety check`, but importantly the hourly test does not. Notifications from the daily and hourly workflows are done in a separate dedicated job. This means that new jobs can be added upstream without needing distinct or rewritten notification logic. The absence of any pytest run for the funcx-sdk package is more obvious and identified properly as a problem. It is paired with the "import test" so that when enabled, the import test removal will be localized to the same area of the workflow file. Linting is a dedicated job which runs as part of the CI workflow and uses pre-commit on the full repo. No additional lint steps (e.g. one-off flake8 runs) are used or should be needed. Testing documentation is updated minimally to refer to the installation of extras, rather than requirements files. The doc site requirements for some (undocumented) reason are installing funcx-sdk test requirements. This has been removed. Any documentation requirements can continue to be specified in the doc directory, or under a new `[docs]` extra if necessary. It should not "inherit" the requirements of the testing process. The codecov integration does not deliver value in the absence of good unit testing and a test matrix of size greater than 1. Therefore, until such a time as codecov is useful and a worthwhile configuration can be identified, the integration has been removed. The funcx-endpoint setup.py file was autoformatted with `black` to simplify the formatting of the requirements data. No workflow steps are allowed to use `actions/checkout@master`. This is unnecessary and dangerous usage. `checkout@v2` is used -- as the GitHub Action's own documentation instructs users to do -- instead.
slateci · Oct 28, 2021 · 6e62716 · 6e62716
1 parent b772fcc
commit 6e62716
Show file tree

Hide file tree

Showing 13 changed files with 254 additions and 196 deletions.
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -9,73 +9,68 @@ on:
   pull_request:
 
 jobs:
-  test:
-    strategy:
-      matrix:
-        python-version: [3.7]
+  lint:
     runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v1
+      - name: install pre-commit
+        run: |
+          python -m pip install -U pip setuptools wheel
+          python -m pip install pre-commit
+      - name: run pre-commit
+        run: pre-commit run -a
 
+  test-sdk:
+    runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v1
-      with:
-        python-version: ${{ matrix.python-version }}
-    - name: Get latest pip version
-      run: |
-        python -m pip install --upgrade pip setuptools wheel
-    - name: Lint
-      run: |
-        pip install pre-commit
-        pre-commit run -a
-    - name: Install dependencies for funcx-sdk
-      run: |
-        python -m pip install -r funcx_sdk/requirements.txt
-        python -m pip install -r funcx_sdk/test-requirements.txt
-        pip list
-    - name: Check for vulnerabilities in libraries
-      run: |
-        pip install safety
-        pip freeze | safety check
-    - name: Test sdk by just importing
-      run: |
-        cd funcx_sdk
-        pip install .
-        python -c "from funcx.sdk.client import FuncXClient"
-        cd ..
-#    - name: Test with pytest
-#      run: |
-#        pytest
-    - name: Install dependencies for funcx-endpoint
-      run: |
-        python -m pip install -r funcx_endpoint/requirements.txt
-        python -m pip install -r funcx_endpoint/test-requirements.txt
-        pip list
-    - name: Check for vulnerabilities in libraries
-      run: |
-        pip install safety
-        pip freeze | safety check
-    - name: Test funcx-endpoint by just importing
-      run: |
-        cd funcx_endpoint
-        pip install .
-        python -c "from funcx_endpoint.version import VERSION"
-        funcx-endpoint -v
-        cd ..
-    - name: Lint with Flake8
-      run: |
-        flake8 funcx_endpoint
-    - name: Test with pytest
-      run: |
-        PYTHONPATH=funcx_endpoint python -m coverage run -m  pytest funcx_endpoint/tests/funcx_endpoint
-    - name: Report coverage with Codecov
-      run: |
-        codecov --token=${{ secrets.CODECOV_TOKEN }}
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v1
+        with:
+          python-version: 3.7
+      - name: install requirements
+        run: |
+          python -m pip install -U pip setuptools wheel
+          python -m pip install './funcx_sdk[test]'
+          pip install safety
+      - name: run safety check
+        run: safety check
+
+      # TODO: remove this test
+      # This is the weakest test which does anything, checking that the client can
+      # be imported. As soon as pytest is running again, remove this.
+      - name: check importable
+        run: python -c "from funcx.sdk.client import FuncXClient"
+      # - name: run pytest
+      #   run: |
+      #     cd funcx_sdk
+      #     pytest
+
+  test-endpoint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v1
+        with:
+          python-version: 3.7
+      - name: install requirements
+        run: |
+          python -m pip install -U pip setuptools wheel
+          python -m pip install './funcx_endpoint[test]'
+          pip install safety
+      - name: run safety check
+        run: safety check
+      - name: run pytest
+        run: |
+          PYTHONPATH=funcx_endpoint python -m coverage run -m pytest funcx_endpoint/tests/funcx_endpoint
 
   publish:
     # only trigger on pushes to the main repo (not forks, and not PRs)
     if: ${{ github.repository == 'funcx-faas/funcX' && github.event_name == 'push' }}
-    needs: test
+    needs:
+      - lint
+      - test-sdk
+      - test-endpoint
     runs-on: ubuntu-latest
     strategy:
       matrix:

diff --git a/.github/workflows/daily.yaml b/.github/workflows/daily.yaml
@@ -0,0 +1,57 @@
+name: daily
+on:
+  # build every weekday at 4:00 AM UTC
+  schedule:
+    - cron: '0 4 * * 1-5'
+  workflow_dispatch:
+
+jobs:
+  safety-check-sdk:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        ref: main
+    - uses: actions/setup-python@v1
+    - name: install requirements
+      run: |
+        python -m pip install --upgrade pip setuptools wheel
+        python -m pip install './funcx_sdk'
+        python -m pip install safety
+    - name: run safety check
+      run: safety check
+
+  safety-check-endpoint:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        ref: main
+    - uses: actions/setup-python@v1
+    - name: install requirements
+      run: |
+        python -m pip install --upgrade pip setuptools wheel
+        python -m pip install './funcx_endpoint'
+        python -m pip install safety
+    - name: run safety check
+      run: safety check
+
+  notify:
+    runs-on: ubuntu-latest
+    needs:
+      - safety-check-sdk
+      - safety-check-endpoint
+    if: failure()
+    steps:
+      # FIXME: make this send to a listhost or Slack
+      - name: Send mail
+        uses: dawidd6/action-send-mail@v3
+        with:
+          server_address: smtp.gmail.com
+          server_port: 465
+          username: ${{secrets.MAIL_USERNAME}}
+          password: ${{secrets.MAIL_PASSWORD}}
+          subject: ${{ github.repository }} - Daily Check ${{ job.status }}
+          to: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
+          from: funcX Tests # <[email protected]>
+          body: The daily ${{ github.repository }} workflow failed!
diff --git a/.github/workflows/hourly.yaml b/.github/workflows/hourly.yaml
@@ -10,7 +10,7 @@ on:
         description: "manual test"
 
 jobs:
-  tutorial_test:
+  smoke-test:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
@@ -22,25 +22,26 @@ jobs:
     - name: Install dependencies for funcx-sdk and test requirements
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip install ./funcx_sdk
-        python -m pip install -r funcx_sdk/test-requirements.txt
-    - name: Check for vulnerabilities in libraries
-      run: |
-        pip install safety
-        safety check
+        python -m pip install './funcx_sdk[test]'
+        python -m pip install safety
     - name: Run smoke tests to check liveness of hosted services
       run: |
         pytest -v funcx_endpoint/tests/smoke_tests --api-client-id ${{ secrets.API_CLIENT_ID }} --api-client-secret ${{ secrets.API_CLIENT_SECRET }}
-     # FIXME: make this send to a listhost or Slack
-    - name: Send mail
-      if: ${{ failure() }}
-      uses: dawidd6/action-send-mail@v3
-      with:
-        server_address: smtp.gmail.com
-        server_port: 465
-        username: ${{secrets.MAIL_USERNAME}}
-        password: ${{secrets.MAIL_PASSWORD}}
-        subject: ${{ github.repository }} - Tutorial test ${{ job.status }}
-        to: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
-        from: funcX Tests # <[email protected]>
-        body: The ${{ github.repository }} test ${{ github.workflow }} exited with status - ${{ job.status }}!
+
+  notify:
+    runs-on: ubuntu-latest
+    needs: [smoke-test]
+    if: failure()
+    steps:
+      # FIXME: make this send to a listhost or Slack
+      - name: Send mail
+        uses: dawidd6/action-send-mail@v3
+        with:
+          server_address: smtp.gmail.com
+          server_port: 465
+          username: ${{secrets.MAIL_USERNAME}}
+          password: ${{secrets.MAIL_PASSWORD}}
+          subject: ${{ github.repository }} - Tutorial test ${{ job.status }}
+          to: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
+          from: funcX Tests # <[email protected]>
+          body: The hourly ${{ github.repository }} workflow failed!
diff --git a/.github/workflows/smoke_test.yaml b/.github/workflows/smoke_test.yaml
@@ -21,14 +21,10 @@ jobs:
     - uses: actions/setup-python@v1
       with:
         python-version: 3.7
-    - name: Install dependencies for funcx-sdk and test requirements
+    - name: install requirements
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip install ./funcx_sdk
-        python -m pip install -r funcx_sdk/test-requirements.txt
-    - name: Test sdk by just importing
-      run: |
-        python -c "from funcx.sdk.client import FuncXClient"
+        python -m pip install './funcx_sdk[test]'
     - name: Run smoke tests to check liveness of hosted services
       run: |
         pytest -v funcx_endpoint/tests/smoke_tests --api-client-id ${{ secrets.API_CLIENT_ID }} --api-client-secret ${{ secrets.API_CLIENT_SECRET }}
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -30,3 +30,18 @@ After installing `pre-commit`, run
 in the repo to configure hooks.
 
 > NOTE: If necessary, you can always skip hooks with `git commit --no-verify`
+
+## Installing Testing Requirements
+
+Testing requirements for each of the two packages in this repository
+(funcx-sdk and funcx-endpoint) are specified as installable extras.
+
+To install the funcx-sdk test requirements
+
+    cd funcx_sdk
+    pip install '.[test]'
+
+To install the funcx-endpoint test requirements
+
+    cd funcx_endpoint
+    pip install '.[test]'
diff --git a/docs/doc-requirements.txt b/docs/doc-requirements.txt
@@ -1,4 +1,3 @@
--r ../funcx_sdk/requirements.txt
--r ../funcx_sdk/test-requirements.txt
+../funcx_sdk
 nbsphinx
 sphinx_rtd_theme
diff --git a/funcx_endpoint/requirements.txt b/funcx_endpoint/requirements.txt