-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
added a docker compose for a 'remote deployment' but locally
- Loading branch information
Showing
6 changed files
with
362 additions
and
30 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,18 +1,62 @@ | ||
| ## Usage | ||
|
|
||
| ### Development with remote deployment | ||
|
|
||
| ```bash | ||
| # re-deploy the EC2 instance stack | ||
| cd iac | ||
| make install | ||
| cdk deploy --app 'python app.py' --region us-west-2 --profile zenml | ||
|
|
||
| # auth with the deployed zenml server | ||
| zenml connect --server-url http://<ec2 ip>:8080 | ||
|
|
||
| # create our zenml components and stack | ||
| ./setup-zenml.sh | ||
| ``` | ||
|
|
||
| ### Clean it up so no one can hack us | ||
|
|
||
| ```bash | ||
| cdk destroy --app 'python app.py' --region us-west-2 --profile zenml | ||
| ``` | ||
|
|
||
| ### Local development | ||
|
|
||
| ```bash | ||
| docker-compose up | ||
| zenml connect --server-url http://localhost:8080 | ||
| ./setup-zenml.sh | ||
| ``` | ||
|
|
||
| ## ZenML OSS vs Metaflow OSS | ||
|
|
||
| | Feature | ZenML OSS | Metaflow OSS | | ||
| | --- | - | - | | ||
| | Preview Artifacts in the UI | ❌ | ❌ | | ||
| | Arbitrary visualizations in UI | | ✅ (metaflow cards) | | ||
| | Trigger DAGs from the UI | ❌ | ❌ (seems like not based on [this](https://github.com/Netflix/metaflow-ui/blob/master/README.mds)) | | ||
| | List/Preview Artifacts in the UI | ❌ | ❌ | | ||
| | Arbitrary visualizations in UI | ❌ | ✅ (metaflow cards) | | ||
| | Trigger DAGs from the UI | ❌ | ❌ (seems like no based on [this](https://github.com/Netflix/metaflow-ui/blob/master/README.mds)) | | ||
| | Deploy Models from the UI | ❌ | ❌ | | ||
| | Build and push docker images | ✅ | ❌ (possibly with netflix extensions) | | ||
| | Stream logs to UI during DAG run | ❌ | ✅ | | ||
| | Can abort DAG mid run from UI | ??? | ❌ | | ||
| | Build and push docker images for you, per step | ⚠️ (not during local development, only when using remote orchestrator) | ❌ (possibly with netflix extensions) | | ||
| | Easy switch from local to cloud | ✅ | ✅ | | ||
| | Run only certain steps on cloud, others locally | ❌ | ✅ | | ||
| | Favor running steps on prem, burst to cloud | ❌ (SkyPilot supports this, but ZenML's integration does not) | ✅ (with kubernetes) | | ||
| | Run DAGs on Airflow | ✅ | ✅ | | ||
| | Run DAGs on Step Functions + AWS Batch | ❌ | ✅ | | ||
| | No limit to number of collaborators | ??? | ✅ | | ||
| | Thin client | ❌ (55 deps) | ✅ (3 deps) | | ||
| | Simple control plane architecture | ✅ (zenml-server, mysql, simple IaC) | ❌ (postgres, backend, frontend, complex IaC) | | ||
| | supports proper Python packaging structure e.g. `pyproject.toml` for the DAG project | ??? | ❌ | | ||
|
|
||
|
|
||
|
|
||
| > The advantage of Skypilot is that it simply provisions a VM to execute the pipeline on your cloud provider. [Reference](https://docs.zenml.io/user-guide/production-guide/cloud-orchestration#starting-with-a-basic-cloud-stack) | ||
| - Does this mean that we CAN NOT have each step be deployed onto its own VM? | ||
|
|
||
| > When a pipeline is run with a remote orchestrator a Dockerfile is dynamically generated at runtime. | ||
| - So we can't have Docker images built and run one after another when developing locally? | ||
| - Does this also mean that we CAN NOT have a DAG where one step has pandas 1.1.1 and another has pandas 1.1.2? How can we have this if we can't have isolated environments per step *locally*? |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| services: | ||
| mysql: | ||
| image: mysql:8.0 | ||
| ports: | ||
| - 3306:3306 | ||
| environment: | ||
| MYSQL_ROOT_PASSWORD: password | ||
| volumes: | ||
| - ./volumes/mysql/data/:/var/lib/mysql | ||
| - ./volumes/mysql/backups/:/backups | ||
| # TODO: mount the mysql state as a volume | ||
|
|
||
| zenml: | ||
| image: zenmldocker/zenml-server | ||
| ports: | ||
| - "8080:8080" | ||
| environment: | ||
| ZENML_STORE_URL: mysql://root:password@mysql/zenml | ||
| ZENML_STORE_BACKUP_DIRECTORY: /backups | ||
| ZENML_STORE_BACKUP_STRATEGY: dump-file | ||
| ZENML_SERVER_AUTO_ACTIVATE: 1 | ||
| ZENML_DEFAULT_USER_NAME: default | ||
| ZENML_ENABLE_IMPLICIT_AUTH_METHODS: "true" | ||
| AWS_PROFILE: zenml | ||
| # ZENML_DEFAULT_USER_PASSWORD: The password to use for the initial admin user account. Defaults to an empty password value, if not set. | ||
| volumes: | ||
| - ~/.aws:/home/zenml/.aws | ||
| - ./volumes/mysql/data/:/var/lib/mysql | ||
| links: | ||
| - mysql | ||
| depends_on: | ||
| - mysql | ||
| # extra_hosts: | ||
| # - "host.docker.internal:host-gateway" | ||
| restart: on-failure |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| zenml[server] | ||
| sagemaker>=2.117.0 | ||
| kubernetes | ||
| aws-profile-manager |
Oops, something went wrong.