[kube-prometheus-stack] - no default dashboard or data source

[fjellvannet]

Describe the bug a clear and concise description of what the bug is.

I installed kube-prometheus-stack into the monitoring-namespace in my bare-metal kubeadm-cluster.

However, when I log into grafana, there are no dashboards visible, and prometheus also does not appear as the default datasource.

What's your helm version?

version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

What's your kubectl version?

Client Version: v1.32.1 Kustomize Version: v5.5.0 Server Version: v1.32.1

Which chart?

prometheus-community/kube-prometheus-stack

What's the chart version?

68.3.0

What happened?

I tried to install the kube-prometheus-stack using the following command:
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

All the pods / services etc are deployed properly, and I can log into grafana using a port-forward or ingress.

However - there are no dashboards in grafana, and also there is no datasource (prometheus is not automatically added).

What you expected to happen?

I would expect the default dashboards for the different kubernetes components like etcd etc. to be visible and populated with data. Instead, Grafana appears like a clean instance with nothing configured.

How to reproduce it?

Set up a bare-metal kubeadm cluster on ubuntu nodes, I now have 3 workers and 3 masters. For block- and other persistent storage, rook-ceph is configured on the worker nodes. Additionally, serverTLSBootstrap is enabled in the kubelets as described here for metrics-server to work. Cert-manager is also running in the cluster. Other than that it is pretty bare-metal, and I have no clue why I don't see this data.

Enter the changed values of values.yaml?

NONE

I tried enabling persistent storage for grafana, enabled an ingress for grafana, set the url of the prometheus-datasource in grafana to this value: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/, but it all made no difference. I see no dashboards and no datasource, even though they are all true in the values. While testing I have also not only helm upgraded, but also helm uninstall and then again helm install to be sure that everything is reconfigured.

I purposely created the issue with NONE as the modified values, because I think it would make sense have the default values work more or less out of the box.

Enter the command that you execute and failing/misfunctioning.

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

Anything else we need to know?

I have tried to follow the different hints I found in the related issues #873 and #3800. See changed values of values.yaml to see what I tried to modify. As both of these issues were old and opened and closed multiple times, I decided to open a new clean issue to get this fixed.