Handling Helm+Kustomize in your rendering

[asafhm]

The purpose of this thread is to see how users of Kargo who also prefer to render their applications using a combination of Helm+Kustomize do so.
I chose to focus on cases where we don't manage the underlying chart files, much like 3rd party charts, since this is where I found things become tricky.

To get it out of the way first, I believe combining Helm with Kustomize is an extremely powerful technique, as none of these tools alone quite brought me all the way to the finish line satisfactorily.

To me, here's what I'd expect out of the ideal solution:

1. Lets me use both Helm and Kustomize conveniently, thus allowing me to override resources originated in the chart, or add my own.
2. Lets me define a version for the chart, which Kargo progressively promotes across my stages
3. Lets me override the helm values per stage
4. Lets me follow the Rendered Manifests Pattern without compromise

Let's assume this is our repo layout:

.
├── cert-manager/
│   ├── base/
│   └── stages/
│       ├── dev/
│       ├── staging/
│       └── prod/
└── kube-prometheus-stack/
    ├── base/
    └── stages/
        ├── dev/
        ├── staging/
        └── prod/

Honestly, the only reasonable candidate I found that will let me harness Helm and Kustomize together is the helmCharts inflator in Kustomize. The question is where to place it... here are 3 possible options:

1. <addon>/base/kustomization.yaml:

e.g.

# cert-manager/base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: cert-manager
  includeCRDs: true
  releaseName: cert-manager
  version: x.x.x
  repo: ...
  valuesFile: ./values.yaml
    ...

resources:
- secret.yaml

The problem is that you no longer retain the ability to override the values of that chart in your stages' kustomize files, and that's sadly because of how that directive is implemented. This is off the table then.

2. <addon>/stages/<stage>/kustomization.yaml:

e.g.

# cert-manager/stages/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: cert-manager
  includeCRDs: true
  releaseName: cert-manager
  version: x.x.x
  repo: ...
  valuesFile: ../../base/values.yaml # I can store my base values in a shared location like so
  additionalValuesFiles:
  - ./values.yaml # my overrides

resources:
- secret.yaml

This approach requires me to put the helmCharts block in each of my stages' files. That's obviously not DRY.
Moreover, promotion of charts' versions becomes a whole thing now, since the helmCharts declaration is managed in each stage individually, they don't necessarily share the same version code-wise. That makes me have to awkwardly progress each on its own in the kustomization.yaml files, without benefiting from the progressive process Kargo has for these cases.

3. Like the first option, but with Kargo involved...

Full disclosure, I didn't try this, but in theory this could work -
We use the first approach, but we make sure to add a step in our Kargo Stage to surgically inject the additionalValuesFiles part into our kustomization.yaml before we run kustomize-build.

Something like this:

- uses: yaml-update
  config:
    path: ./src/cert-manager/base/kustomization.yaml
    updates:
    - key: helmCharts[0].additionalValuesFiles[0]
      value: ../stages/prod/values.yaml

That can let us overcome helmCharts shortcoming of overriding values when using more than one kustomization.yaml file, and also share the chart version across multiple stages, as the helmCharts definition is written once, in the base directory.
The only downside I see for this approach is that it's not trivial for kustomize users, as now part of the logic for rendering these manifests sits elsewhere, in Kargo files, which perhaps are stored in a different repository altogether.

Please share your thoughts and how you approach this.

[krancour]

@asafhm thank you so much for the level of effort and detail that you've put into exploring this topic.

I think you're on the right track with the third option, and I've got some small ideas to improve it further and to mitigate your valid concern:

it's not trivial for kustomize users, as now part of the logic for rendering these manifests sits elsewhere, in Kargo files, which perhaps are stored in a different repository altogether

I'll follow up on this in the coming week.

Thanks again for starting this convo!

[asafhm]

With pleasure! Can't wait to hear what ideas you came up with :)

[asafhm]

Another caveat for those thinking of going with the third option - being dependent on Kargo for rendering your apps can pose a challenge when attempting to implement ephemeral/preview environments, as you now need to take care of setting up Warehouse/Stage resources for these temporary environments just to be able to render your apps. Also, you're now responsible for these resources' cleanup once their environment is gone.