Skip to content

Commit

Permalink
start of work to refactor submit (#28)
Browse files Browse the repository at this point in the history
* start of work to refactor submit

submit now uses the flux operator python sdk (non released version) to
create / submit / delete the minicluster, no need for external scripts
except for starting minikube. This is likely an improvement and I am going
to also test using the sdk for apply.

* refactor of apply

I still need to save the nodes/pods and other cluster info to
the .scripts directory, but the basic apply is now working JUST
using the Python SDK. When this is all done, we should be able
to remove the custom scripts that are running jobs, and update
and test all the cloud examples.

* finish up refactor of submit/apply

everything should now use the python sdk
* set min version of fluxoperator sdk
* test only apply
* add back helpers script

Signed-off-by: vsoch <[email protected]>
  • Loading branch information
vsoch authored Mar 6, 2023
1 parent 66c9a3e commit 97f2777
Show file tree
Hide file tree
Showing 123 changed files with 2,355 additions and 5,054 deletions.
85 changes: 81 additions & 4 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,74 @@ jobs:
pip install -r .github/dev-requirements.txt
pre-commit run --all-files
test-runs:
test-python:
runs-on: ubuntu-latest
steps:
- name: Clone the code
uses: actions/checkout@v3

- name: Install flux-cloud
run: |
conda create --quiet --name fc jinja2
export PATH="/usr/share/miniconda/bin:$PATH"
source activate fc
pip install .[all]
- name: Test Python
run: |
export PATH="/usr/share/miniconda/bin:$PATH"
source activate fc
export SHELL=/bin/bash
# This will bring MiniKube up/down
pytest -xs ./fluxcloud/tests/test_settings.py
pytest -xs ./fluxcloud/tests/test_utils.py
test-examples:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
test: ["lammps"]
test: ["test_minicluster_logging", "test_minicluster_volumes",
"test_minicluster_resources"]
steps:
- name: Clone the code
uses: actions/checkout@v3

- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: ^1.18

- name: Install flux-cloud
run: |
conda create --quiet --name fc jinja2
export PATH="/usr/share/miniconda/bin:$PATH"
source activate fc
pip install .[all]
pip install kubernetes
- name: Start minikube
uses: medyagh/setup-minikube@697f2b7aaed5f70bf2a94ee21a4ec3dde7b12f92 # v0.0.9

- name: Test Example
env:
test: ${{ matrix.test }}
run: |
export PATH="/usr/share/miniconda/bin:$PATH"
source activate fc
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
# We need to delete the minikube cluster to bring it up again
minikube delete
# This will bring MiniKube up/down
pytest -xs ./fluxcloud/tests/test_examples.py::${test}
test-runs:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
command: [["apply", "lammps"], ["submit", "./examples/minikube/basic"]]
steps:
- name: Clone the code
uses: actions/checkout@v3
Expand All @@ -50,9 +111,10 @@ jobs:
- name: Start minikube
uses: medyagh/setup-minikube@697f2b7aaed5f70bf2a94ee21a4ec3dde7b12f92 # v0.0.9

- name: Test ${{ matrix.test }}
- name: Test ${{ matrix.command }}
env:
name: ${{ matrix.test }}
name: ${{ matrix.command[1] }}
if: (matrix.command[0] == 'apply')
run: |
export PATH="/usr/share/miniconda/bin:$PATH"
source activate fc
Expand All @@ -61,3 +123,18 @@ jobs:
# We need to delete the minikube cluster to bring it up again
minikube delete
/bin/bash ./tests/test.sh ${name}
- name: Test ${{ matrix.command }}
env:
workdir: ${{ matrix.command[1] }}
if: (matrix.command[0] == 'submit')
run: |
export PATH="/usr/share/miniconda/bin:$PATH"
source activate fc
export SHELL=/bin/bash
eval $(minikube -p minikube docker-env)
minikube delete
cd ${workdir}
flux-cloud up --cloud minikube --force-cluster
flux-cloud --debug submit --non-interactive
flux-cloud down --cloud minikube
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,5 @@ env
.env
dist
__pycache__
examples/**/data
examples/**/_data
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ and **Merged pull requests**. Critical items to know are:
The versions coincide with releases on pip. Only major versions will be released as tags on Github.

## [0.0.x](https://github.com/converged-computing/flux-cloud/tree/main) (0.0.x)
- refactor flux submit and apply to use fluxoperator Python SDK (0.2.0)
- This reduces scripts in output folder, but is a good tradeoff for fewer errors
- remove "ui" command, flux-cloud is intended mostly for automation
- command and image will always be required.
- fix bash script bugs (0.1.19)
- support for node group level aws avail. zones, save times on each experiment apply (0.1.18)
- data should be namespaced by cloud type (so multiple experiments can be run alongside) (0.1.17)
Expand Down
5 changes: 0 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,6 @@ It will be expanded as needed.
Code is under development and likely to change!
In the meantime, for early documentation, see our ⭐️ [Documentation](https://converged-computing.github.io/flux-cloud/) ⭐️

## TODO

- test for list of experiments
- cloud-select could estimate the cost?
- run and add more cluster examples

## 😁️ Contributors 😁️

Expand Down
Loading

0 comments on commit 97f2777

Please sign in to comment.