Remove Pipelines - Prod

[wg102]

This is based on the issue we created for dev here: #1449

Note:

• Do we need to give a warning to the users before?
• Might need to ask permissions from people
• Might want to split this in multiple issues

It might include multiple steps.

• Double check resources to make sure no pipelines are currently in use

Repo: aaw-kubeflow-manifests : StatCan/aaw-kubeflow-manifests#291

• Remove the pipelines from the menu
  • kustomize/apps/centraldashboard/base/centraldashboard-config.yaml
• Remove the deployment form the aaw-kubeflow-manifest
  • Remove the pipeline folders pipeline/base

Repo: aaw-argocd-manifests https://github.com/StatCan/aaw-argocd-manifests/pull/240

• Remove the link between pipeline repo and argocd see https://github.com/StatCan/aaw-argocd-manifests/blob/aaw-dev-cc-00/kubeflow/kubeflow.jsonnet

Repo: aaw-kubeflow-profiles-controller and Charts

• Deal with pipelines-profiles-controller

  • Not too sure what is happening with the profile controller
    StatCan/aaw-kubeflow-profiles-controller@d8e4e31

  • Need to redo the StatCan/aaw-kubeflow-profiles-controller@d8e4e31

  • Then to update the chart https://github.com/StatCan/charts/pull/328/files#diff-f7efe3362b0689d66da5f3aa7e7db08402c3325139f52d40a992517c5ab1c1b1

  • Once the chart is updated need to update aaw-kubeflow-manifest with the right chart version

And if a newer version exists, so the chart won't be bumped to the version on file, but a newer one.

After the code is in prod

• Checkout if there are any resources left after (config maps, secrets, deployments, etc.)
• Create a deprecation script to clean up the resources
• Once in prod, test using Pipeline Removal Testing - Prod #1514

Possibly another task entirely:

• Create a script to remove the pipeline labels from the pods/notebooks - note: might need restart?

[wg102]

Note: #1498 needs to be completed first

[wg102]

Q: Do we need to give a warning to the users before?
A: No need if it doesn't disturb prod and create down time, which it should not. Kubeflow will not, and the controller modifications should not either. As far as removing the labels (sometime in the future) goes, it is undetermined as of now.

Q: Might need to ask permissions from people
A: Since it will not disturb their normal day, and as far as we know, from experience, from the current pipeline (or lack of) and from the survey, no one is using pipelines, therefore we can delete it.

Q: Might want to split this in multiple issues
A: As needed be.

Q: Why are we deprecating pipelines?
A: Mostly because of the maintenance burden, and i18n issues. On top of the fact that no user actively uses the component. it because too much effort for the low reward. Users will eventually be able to use another way such as (possibly) argo workflow instead.

[wg102]

Confirmation that no pipelines are currently in use:

⎈ aaw-prod-cc-00-aks~aaw$ kubectl get workflows -ALL
No resources found
⎈ aaw-prod-cc-00-aks~aaw$ kubectl get workflow -ALL
No resources found

[wg102]

Where do we update the SHA of prod? What about the other changes?

[wg102]

Order of pipelines to merge:

• https://github.com/StatCan/aaw-kubeflow-manifests/pull/291/files remove menu
• https://github.com/StatCan/aaw-kubeflow-manifests/pull/303/files update central dash
• https://github.com/StatCan/aaw-kubeflow-profiles-controller/pull/91/commits
• NEED TO BE CREATED AND UPDATED FIRST https://github.com/StatCan/charts/pull/328/files#diff-f7efe3362b0689d66da5f3aa7e7db08402c3325139f52d40a992517c5ab1c1b1
• Need to update the chart version and sha on https://github.com/StatCan/aaw-argocd-manifests/pull/240

[wg102]

Test to ensure the rolebindings don't get created following https://github.com/StatCan/aaw-kubeflow-profiles-controller/pull/91/files in dev.

Sync up argocd prod so the uI changes are in. kubeflow-aaw-central-dashboard config map
Merge the aaw-kubflow-manifest pull 303 (may need a sync as well/

[wg102]

Here is the plan on merging and testing it to Pipelines

• Test https://github.com/StatCan/aaw-kubeflow-profiles-controller/pull/91/commits in dev using argo-cd to manually update the profile controller sha. To ensure the rolebindings don't get created when creating a new notebook.
  Once this is confirmed, (it was tested once in dev, but new changes needed it to be reverted back) we can continue
• https://github.com/StatCan/aaw-kubeflow-manifests/pull/303/files to remove the Cards of pipeline in the home dashboard (might need sync
• Sync up argocd prod so the uI changes are in. kubeflow-aaw-central-dashboard config map
  This will create a restart of the central dashboard which should only take a few seconds
• Create a PR in charts repo with the new SHA and tag (1) see https://github.com/StatCan/charts/pull/328/files#diff-f7efe3362b0689d66da5f3aa7e7db08402c3325139f52d40a992517c5ab1c1b1 for example
• Update https://github.com/StatCan/aaw-argocd-manifests/pull/240/files with the data from point 1

At this point all the code will have been there.
And some of the resources are deleted when syncing

Some fast testing just to ensure that notebooks are functional can be done at this stage.

The following is cleanup and testing and should not affect availability

• Checkout if there are any resources left after (config maps, secrets, deployments, etc.)
• Create a deprecation script to clean up the resources https://github.com/StatCan/daaas-private/issues/55
• Once in prod, test using Pipeline Removal Testing - Prod #1514