kf-tools--pipelines ended in degraded in my first local quickstart

[cheese]

I am truly a newbie to Kubeflow and deployKF. I want to deploy my first Kubeflow with deployKF to get started. I followed https://www.deploykf.org/guides/local-quickstart to set up an instance of deployKF v0.1.4 (29ac97a) on my Fedora 38 machine with moby-engine 24.0.5 and k3d 5.6.0, but it ended up with kf-tools--pipelines in degraded state.

Four pods are failing to run:

1. ml-pipeline and ml-pipeline-ui pods error: secret "cloned--kubeflow-pipelines--backend-object-store-auth" not found
2. ml-pipeline-ui-artifact pods error: secret "mlpipeline-minio-artifact" not found

[thesuperzapper]

@cheese can you check the state AND logs of the Pods in the kyverno namespace, especially the kyverno-background-controller-xxxxxxx Pod?

After you save the background-controller logs (so you can post them), just delete the Pod:

kubectl delete pods -n kyverno -l "app.kubernetes.io/part-of=kyverno,app.kubernetes.io/component=background-controller"

After it comes back up, check that the following secrets were created:

• Secret/cloned--kubeflow-pipelines--backend-object-store-auth (in kubeflow namespace)
• Secret/mlpipeline-minio-artifact (in team-1 namespace)

[cheese]

Logs of background-controller are scrolling fast, I just cut the tail part.
kyverno-kyverno-background-controller-796f77c79f-hs267-1710737551788174593.log

I deleted the pod but the secrets were not created after it came back.

[thesuperzapper]

@cheese it really looks like something deleted the minio secrets which are the source of these clones:

• Secret/generated--kubeflow-pipelines--backend-object-store-auth (in the deploykf-minio namespace)
• Secret/generated--kubeflow-pipelines--profile-object-store-auth--team-1 (in the deploykf-minio namespace)
• Secret/generated--kubeflow-pipelines--profile-object-store-auth--team-1-prod (in the deploykf-minio namespace)

Did you delete these manaully?
(NOTE: they show as "need pruning" in the ArgoCD UI, which might be confusing because they should NOT be deleted, they aren't created by ArgoCD, but instead by a pre-install Job Pod)

Either way, you can trigger the job to recreate them by re-syncing the dkf-opt--deploykf-minio ArgoCD app manually, even though it will show as "in sync", you can do this in the ArgoCD WebUI.

If you need help accessing the ArgoCD WebUI, see the sync apps step of the getting started guide.

[cheese]

Thanks! I remember I did not delete them. After that, re-syncing dkf-opt--deploykf-minio just fixed the issue.

[thesuperzapper]

@cheese if possible, could you try and reproduce this situation in a new K3D cluster?

Because there is clearly some race condition (or bug) here with the dkf-opt--deploykf-minio app, and I want to figure out what it is.

[cheese]

I tried again and still had to re-sync dkf-opt--deploykf-minio after it announced healthy.

[thesuperzapper]

@cheese it is interesting that it's reproducible.

Can you tell me the version of the argocd CLI you are running? Run argocd version to check.

After that, run brew update and brew upgrade to update to the latest one.

Finally, can you please try with a new cluster and the latest version of ArgoCD and see if it still happens?
(I have updated the version of ArgoCD to 2.10.4 in main branch of deploykf/deploykf, so you should just be able to re-clone the repo following the prepare argocd step in the local quickstart again).