0.30.0rc1

This pre-release builds on top of 0.30.0rc0, specifically with a few bug fixes including fixing the quickstart

Note: Installing and using this will migrate your ZenML metadata database
tables without any way of downgrading to previous releases. Use at your own risk.

0.30.0rc0

This pre-release removes the ml-pipeline-sdk dependency and adds support for Python 3.10 as well as M1 macs.

Note: Installing and using this will migrate your ZenML metadata database
tables without any way of downgrading to previous releases. Use at your own risk.

0.23.0

This release comes with a brand-new Neptune integration to track your ML experiments

as well as lots of performance improvements!

Neptune integration

The new Neptune integration
includes a Neptune experiment tracker component that allows you to track your machine learning experiments
using Neptune.

Performance Optimization

The 0.20.0 release introduced our new server but brought with it a few performance and scalability issues.
Since then we've made many improvements to it, and this release is the final and biggest boost in performance. We reduced the amount of server calls needed for almost all CLI commands and greatly improved the speed of the dashboard as well.

PyArrow dependency removal

We've removed PyArrow as a dependency of the zenml python package.
As a consequence of that, our NumPy and Pandas materializer no
longer read and write their artifacts using PyArrow but instead use
native formats instead. If you still want to use PyArrow to serialize
your NumPy arrays and Pandas dataframes, you'll need to install it manually
like this: pip install pyarrow

In future releases we'll get rid of other unnecessary dependencies to
further slim down the zenml package.

Breaking Changes

The following changes introduces with this release mey require some manual
intervention to update your current installations:

• If your code calls some methods of our Client class, it might need to be
  updated to the new model classes introduced by the performance optimization changes
  explained above
• The CLI command to remove an attribute from a stack component now takes no more dashes
  in front of the attribute names:
  zenml stack-component remove-attribute <COMPONENT_NAME> <ATTRIBUTE_NAME>
• If you're using a custom stack component and have overridden the cleanup_step_run method,
  you'll need to update the method signature to include a step_failed parameter.

What's Changed

• Docs regarding roles and permissions by @AlexejPenner in #1081
• Add global config dir to zenml status by @schustmi in #1084
• Remove source pins and ignore source pins during step spec comparisons by @schustmi in #1083
• Docs/links for roles permissions by @AlexejPenner in #1091
• Bugfix/eng 1485 fix api docs build by @AlexejPenner in #1089
• fix bento builder step parameters to match bentoml by @safoinme in #1096
• Add bentoctl to BentoML docs and example by @safoinme in #1094
• Fix BaseParameters sample code in docs by @jcarlosgarcia in #1098
• zenml logs defaults to active stack without name_or_id by @AlexejPenner in #1101
• Fixed evidently docs by @htahir1 in #1111
• Update sagemaker default instance type by @schustmi in #1112
• The ultimate optimization for performance by @bcdurak in #1077
• Update stack exporting and importing by @schustmi in #1114
• Fix readme by @schustmi in #1116
• Remove Pyarrow dependency by @safoinme in #1109
• Bugfix for listing the runs filtered by a name by @bcdurak in #1118
• Neptune.ai integration by @AleksanderWWW in #1082
• Add YouTube video explaining Stack Components Settings vs Config by @dnth in #1120
• Add failed Status to component when step fails by @safoinme in #1115
• Add architecture diagrams to docs by @AlexejPenner in #1119
• Remove local orchestrator restriction from step operator docs by @schustmi in #1122
• Validate Stack Before Provision by @safoinme in #1110
• Bugfix/fix endpoints for dashboard development by @AlexejPenner in #1125
• Skip kubeflow UI daemon provisioning if a hostname is configured by @schustmi in #1126
• Update Neptune Example by @safoinme in #1124
• Add debugging guide to docs by @dnth in #1097
• Fix stack component attribute removal CLI command by @schustmi in #1127
• Improving error messages when fetching entitites by @bcdurak in #1117
• Introduce username and password to kubeflow for more native multi-tenant support by @htahir1 in #1123
• Add support for Label Studio OCR config generation by @shivalikasingh95 in #1062
• Misc doc updates by @schustmi in #1131
• Fix Neptune run cleanup by @safoinme in #1130

New Contributors

• @jcarlosgarcia made their first contribution in #1098
• @AleksanderWWW made their first contribution in #1082
• @shivalikasingh95 made their first contribution in #1062

Full Changelog: 0.22.0...0.23.0

0.22.0

This release comes with a new BentoML integration as well as a reworked Airflow orchestrator. Additionally, it greatly improves the server performance as well as other small fixes and updates to our docs!

BentoML integration

The new BentoML integration includes a BentoML model deployer component that allows you to deploy your models from any of the major machine learning frameworks on your local machine.

See example here.

Airflow orchestrator v2

The previous Airflow orchestrator was limited to running locally and had many additional unpleasant constraints that made it hard to work with. This release includes a completely rewritten, new version of the Airflow orchestrator that now relies on Docker images to run your pipelines and works both locally and with remote Airflow deployments.

See what changed in this video and check out the brand-new example here.

Notable bugfixes

• Further improvements to the synchronization that transfers pipeline run information from the MLMD database to the ZenML Server.
• The ZenML Label Studio integration can now be used with non-local (i.e. deployed) instances. For more information see the Label Studiodocs.
• The Spark example is fixed and now works again end-to-end.

Breaking Changes

The following changes introduces with this release mey require some manual
intervention to update your current installations:

• the Airflow orchestrator now requires a newer version of Airflow (run zenml integration install airflow to upgrade) and Docker installed
  to work.

What's Changed

• Fix bug when running non-local annotator instance. by @sheikhomar in #1045
• Introduce Permissions, Link Permissions to Roles, Restrict Access to endpoints based on Permission by @AlexejPenner in #1007
• Fix copy-pasted log message for annotator by @strickvl in #1049
• Add warning message for client server version mismatch by @schustmi in #1047
• Fix path to ingress values in ZenServer recipes by @wjayesh in #1053
• Prevent deletion/update of default entities by @stefannica in #1046
• Fix Publish API docs workflow by @AlexejPenner in #1054
• Fix multiple alembic heads warning by @fa9r in #1051
• Fix Null Step Configuration/Parameters Error by @fa9r in #1050
• Fix role permission migration by @schustmi in #1056
• Made role assignment/revokation possible through zen_server by @AlexejPenner in #1059
• Bugfix/make role assignment work with enum by @AlexejPenner in #1063
• Manually set scoped for each endpoint by @AlexejPenner in #1064
• Add run args to local docker orchestrator settings by @schustmi in #1060
• Docker ZenML deployment improvements and docs by @stefannica in #1061
• Bugfix Mlflow service cleanup configuration by @safoinme in #1067
• Rename DB Tables and Fix Foreign Keys by @fa9r in #1058
• Paginate secrets in AWSSecretsManager by @chiragjn in #1057
• Add explicit dashboard docs by @strickvl in #1052
• Added GA and Gitlab to envs by @htahir1 in #1068
• Add Inference Server Predictor to KServe and Seldon Docs by @safoinme in #1048
• Rename project table to workspace by @fa9r in #1073
• Airflow orchestrator v2 by @schustmi in #1042
• Add get_or_create_run() ZenStore method by @fa9r in #1070
• Fix the flaky fileio tests by @schustmi in #1072
• BentoML Deployer Integration by @safoinme in #1044
• Sync Speedup by @fa9r in #1055
• Fixed broken links in docs and examples. by @dnth in #1076
• Make additional stack component config options available as a setting by @schustmi in #1069
• Rename step_run_artifact table to step_run_input_artifact by @fa9r in #1075
• Update Spark Example to ZenML post 0.20.0 by @safoinme in #1071
• Always set caching to false for all Kubeflow based orchestrators by @schustmi in #1079
• Feature/eng 1402 consolidate stack sharing by @AlexejPenner in #1036

New Contributors

• @sheikhomar made their first contribution in #1045
• @chiragjn made their first contribution in #1057

Full Changelog: 0.21.1...0.22.0

0.21.1

This is an ad-hoc release to fix some bugs introduced the 0.21.0 release that
made the local ZenML dashboard unusable.

What's Changed

• Include latest (not oldest) three runs in HydratedPipelineModel by @schustmi in #1039
• Update docs to use pip install [server] by @strickvl in #1037
• Docs fix for Deepchecks by @strickvl in #1040
• Fix the pipeline run sync on sqlite and the --blocking zenml server deployment by @stefannica in #1041

Full Changelog: 0.21.0...0.21.1

0.21.0

This release primarily fixes a number of bugs that were introduced as part of
the 0.20.0 ZenServer release. These significantly improve the stability when
using ZenML with the ZenML Server.

Notable fixes include:

• Improved the synchronization that transfers pipeline run information from
  the MLMD database to the ZenML Server. This helps fix a number of issues with
  missing steps in the post-execution workflow, model deployment steps and other
  issues.
• The Label Studio example is fixed and now works again end-to-end.
• The ZenML Label Studio integration can now be used with non-local (i.e.
  deployed) instances. For more information see the Label Studiodocs.

New features and other improvements:

• ZenML now uses alembic for
  automated database migrations. The migrations happen automatically after every
  ZenML update.
• New zenml pipeline runs export / import / migrate CLI commands are now
  available to export, import and migrate pipeline runs from older, pre-0.20.0
  versions of ZenML. The ZenML server now also automatically picks up older
  pipeline runs that have been logged in the metadata store by ZenML prior to
  0.20.0.
• An MLMD gRPC service can now be deployed with the ZenML Helm chart to act
  as a proxy between clients, orchestrators and the MySQL database. This
  significantly reduces the time it takes to run pipelines locally.
• You can now specify affinity and tolerations and node selectors to all
  Kubernetes based orchestrators with the new Kubernetes Pod settings feature.

Breaking Changes

The following changes introduces with this release mey require some manual
intervention to update your current installations:

• the zenml server helm chart values.yaml file has been restructured to make
  it easier to configure and to clearly distinguish between the zenml server
  component and the newly introduced gRPC service component. Please update your
  values.yaml copies accordingly.
• the Azure integration dependency versions have been updated. Please run
  zenml integration install azure to update your current installation, if
  you're using Azure.

What's Changed

• Implement automatic alembic migration by @AlexejPenner in #990
• Fix GCP Artifact Store listdir empty path by @safoinme in #998
• Add flavors mini-video to docs by @strickvl in #999
• Remove the Client() warning when used inside a step by @stefannica in #1000
• Fix broken links caused by updated by @AlexejPenner in #1002
• Fix FileNotFoundError with remote path in HuggingFace Dataset materializer by @gabrielmbmb in #995
• Add zenml pipeline runs export / import / migrate CLI commands by @fa9r in #977
• Log message when activating a stack as part of registration by @schustmi in #1005
• Minor fixes in Migration to 0.20.0 documentation by @alvarobartt in #1009
• Doc updates by @htahir1 in #1006
• Fixing broken links in docs by @dnth in #1018
• Label Studio example fix by @strickvl in #1021
• Docs for using CUDA-enabled docker images by @strickvl in #1010
• Add social media heading on docs page by @dnth in #1020
• Add executing custom command for getting requirements by @gabrielmbmb in #1012
• Delay user instruction in dockerfile generation by @schustmi in #1004
• Update link checker configs for faster, more accurate checks by @dnth in #1022
• Add pip install zenml[server] to relevant examples by @dnth in #1027
• Add Tolerations and NodeAffinity to Kubernetes executor by @wefner in #994
• Support pydantic subclasses in BaseParameter attributes by @schustmi in #1023
• Unify run names across orchestrators by @schustmi in #1025
• Add gRPC metadata service to the ZenML helm chart by @stefannica in #1026
• Make the MLMD pipeline run information transfer synchronous by @stefannica in #1032
• Add console spinner back by @strickvl in #1034
• Fix Azure CLI auth problem by @wjayesh in #1035
• Allow non-local Label Studio instances for annotation by @strickvl in #1033
• Before deleting the global zen_server files, spin it down by @AlexejPenner in #1029
• Adding zenserver integration to stack recipe CLI by @wjayesh in #1017
• Add support for Azure ZenServer by @wjayesh in #1024
• Kubernetes Pod settings by @schustmi in #1008

New Contributors

• @alvarobartt made their first contribution in #1009
• @wefner made their first contribution in #994

Full Changelog: 0.20.5...0.21.0

0.20.5

ZenML 0.20.5 fixes another series of minor bugs, significantly improves the performance of the CLI, and adds an option to specify APT packages in Docker images.

What's Changed

• Fix accessing local zen store and artifact store in containers by @stefannica in #976
• K3d local registry pod spec updated by @wjayesh in #972
• Update readme page by @dnth in #985
• Remove beam dependency by @schustmi in #986
• Fix error message when registering secret without secrets manager by @schustmi in #981
• Update cheat sheet up to zenml==0.20.4 by @dnth in #987
• Example fixes (part 2) by @strickvl in #971
• Allow duplicate step classes inside a pipeline by @schustmi in #989
• Include deployment in azureml docker build by @schustmi in #984
• Automatically open browser upon zenml up command by @dnth in #978
• Add a just_mine flag for zenml stack list by @strickvl in #979
• Add option to specify apt packages by @schustmi in #982
• Replace old flavor references, fix the windows local ZenML server and other fixes by @stefannica in #988
• Improve docker and k8s detection by @schustmi in #991
• Update GH actions example by @schustmi in #993
• Update MissingStepParameterError exception message by @gabrielmbmb in #996
• Seprated code docs into core and integration docs by @AlexejPenner in #983
• Add docs/mkdocstrings_helper.py to format script sources by @fa9r in #997
• Further CLI optimization by @bcdurak in #992

Full Changelog: 0.20.4...0.20.5

0.20.4

This release fixes another series of minor bugs that were introduced in 0.20.0.

What's Changed

• Detect failed executions by @schustmi in #964
• Only build docker images for custom deployments by @schustmi in #960
• M1 Mac Installation Tutorial by @fa9r in #966
• Update ZenBytes links in docs by @fa9r in #968
• Fix the API docs builder by @stefannica in #967
• Fix gpu_limit condition in VertexOrchestrator by @gabrielmbmb in #963
• Add simple node affinitiy configurations by @schustmi in #973
• First iteration of the CLI optimization by @bcdurak in #962

Full Changelog: 0.20.3...0.20.4

0.20.3

This release fixes another series of minor bugs that were introduced in 0.20.0 and add full support docs for Kubeflow multi-tenancy deployments.

What's Changed

• Fixed GitHub/Colab JSON formatting error on quickstart. by @fa9r in #947
• Update YAML config template by @htahir1 in #952
• correct code from merge and fix import by @wjayesh in #950
• Check for active component using id instead of name by @schustmi in #956
• Tekton fix by @htahir1 in #955
• Improve zenml up/down UX and other fixes by @stefannica in #957
• Update kubeflow docs for multi-tenant deployments by @htahir1 in #958
• Update kubeflow.md by @abohmeed in #959
• Add additional stack validation for step operators by @schustmi in #954
• Fix pipeline run dashboard URL for unlisted runs by @fa9r in #951
• Support subclasses of registered types in recursive materialization by @fa9r in #953

New Contributors

• @abohmeed made their first contribution in #959

Full Changelog: 0.20.2...0.20.3

0.20.2

After a successful release of the new ZenML server and dashboard paradigm, we set to ironing out some bugs that slipped through.

What's Changed

• Capitalize all docs page titles. by @fa9r in #937
• Increase field sizes for docstrings and step parameters. by @fa9r in #940
• Fixing the bug in the registration of custom flavors by @bcdurak in #938
• Implemented docstring Attribute of StepModel by @fa9r in #936
• Fix shared stack emoji by @strickvl in #941
• Fix shared stacks not being allowed to be set as active. by @fa9r in #943
• Typo fix by @strickvl in #944
• Update Kubernetes Orchestrator Example by @fa9r in #942
• Add code and instructions to run quickstart on Colab. by @fa9r in #939
• Fixing the interaction in getting stacks/components by @bcdurak in #945
• Fix Kubeflow run name by @safoinme in #946
• VertexOrchestrator apply node selector constraint if gpu_limit > 0 by @gabrielmbmb in #935

Full Changelog: 0.20.1...0.20.2