Kargo objects management in multi-cluster topology

[daftping]

A topology where one Argo CD instance (usually on a dedicated K8s cluster) manages multiple k8s clusters e.g. (qa1, dev1, dev2, prod1, prod2, ...) is pretty convenient (and probably common) for services running on multiple clusters. You can have a centralized view of all your Applications on all your clusters and group them in ArgoCD projects where you manage RBAC and other common settings.

In this topology sensitive settings like Argo CD Projects and ApplicationSets are managed centrally to set sensitive guardrails and control the RBAC. All Application configurations can be decentralized by ApplicationSet Git Generator especially now when templatePatch is implemented.

To incorporate Kargo into this topology we can install Kargo to the K8s cluster with Argo CD. Kargo will be able to manage all the Apps on multiple clusters however all the environment progression configurations (Stage, Warehouse, PromotionPolicy) must be created and managed on K8s clusters cluster with Kargo instance.

If I understand correctly, it is impossible to have Kargo CRD objects sourced from a different cluster other than Kargo deployed cluster.

Are there any plans to support scenarios where users can control environment progression configs (CRDs) in their cluster where apps are deployed?

The only solution I see right now is to have Application(Sets) deployed to Kargo K8s cluster and source Kargo CRDs from client's Git repository.

[krancour]

To incorporate Kargo into this topology we can install Kargo to the K8s cluster with Argo CD. Kargo will be able to manage all the Apps on multiple clusters

Correct.

however all the environment progression configurations (Stage, Warehouse, PromotionPolicy) must be created and managed on K8s clusters cluster with Kargo instance.

I am not certain what you are saying or asking here. Can you please clarify?

[daftping]

I'd like users to deploy and manage environment progression configurations (Stage, Warehouse, PromotionPolicy) on their own clusters (qa*, dev*, prod*) where Kargo is not installed.

[krancour]

Ok... I thought that may have been what you were asking. Thank you for clarifying.

That's not possible. Resources like warehouses and stages link together to form a pipeline that may span many clusters (and may even span multiple Argo CD instances). The close relationships between Kargo resources necessitates that they all live in one central, Kargo control plane.

[krancour]

Thinking about this more, I should have made note of the security angle here as well.

There would be security and scalability implications to having a single Kargo controller reconciling resources in multiple remote clusters.

Although it's a tangent, the hub and spoke model you're using with Argo CD suffers from the same problems. Our experience with Argo CD has led us to consciously avoid that topology with Kargo.

[daftping]

Thank you for your prompt response.

[jessesuen]

@daftping is your use case that you want developers to self-service their Stages, Warehouses, etc... by giving them access to a namespace where they are free to create those resources themselves, but they cannot do this on the Kargo controlplane? If so why not?

[krancour]

That's an insightful question @jessesuen.

The Kargo control plane doesn't have to run in the same cluster as your centralized Argo CD control plane. A Kargo controller running in some other cluster can talk to a remote Argo CD control plane.

[daftping]

In the current implementation, developers don't have access to the control plane cluster but there are no technical limitations of providing such access, it is just easier from permission and operations point of view.
Having Kargo on a different cluster than Argo CD is actually can be beneficial for us.

Thank you for the responses.

[pww217]

I'm sorry to revive this thread, but I just want to make sure I understand correctly.

Does the Kargo control plane and the CRD resources like warehouses all need to exist on a single cluster, or can you have the control plane on one and CRDs on another?

It sounds to me like it can manage CD on multiple clusters but the control plane + CRDs must all be on the same cluster? Is that correct? And that it's not a planned feature either?

[krancour]

The control plane is centralized, as control planes tend to be. The Kargo CRDs and instances of them (CRs) live in that control plane. Kargo controllers can run outside that control plane, typically deployed 1:1 and "side by side" with an Argo CD control plane. Each controller talks back to the Kargo control plane. So to recap, each Kargo controller connects to, potentially, two different clusters -- one being the one hosting the Kargo control plane where all Kargo data is centralized and the other being the one hosting a (possibly one of many) Argo CD control plane.