Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(helm): Add health probes to long-lived containers #15359

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
+661 −75
Open

feat(helm): Add health probes to long-lived containers #15359

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
9df3b42
fix(helm): Fix health probe helper templates
lindhe Dec 12, 2024
393a271
feat(helm): Add health probes to gateway
lindhe Dec 11, 2024
e31c65c
feat(helm): Add health probes to admin-api
lindhe Dec 11, 2024
aa3a08e
feat(helm): Add startupProbe to loki-sc-rules
lindhe Dec 12, 2024
20fac46
feat(helm): Add health probes to loki in backend
lindhe Dec 12, 2024
bc95253
feat(helm): Add health probes to bloom-builder
lindhe Dec 12, 2024
28099f8
feat(helm): Add health probes to bloom-gateway
lindhe Dec 12, 2024
5327e7f
feat(helm): Add health probes to bloom-planner
lindhe Dec 12, 2024
9fa4d2e
feat(helm): Add health probes to memcached-exporter
lindhe Dec 12, 2024
17e9550
feat(helm): Add health probes to memcached
lindhe Dec 12, 2024
259a8aa
feat(helm): Add startupProbe to compactor
lindhe Dec 12, 2024
ed6fc04
feat(helm): Add startupProbe to distributor
lindhe Dec 12, 2024
d49c60f
feat(helm): Add health probes to enterprise-gateway
lindhe Dec 12, 2024
4e01303
feat(helm): Add startupProbe to index-gateway
lindhe Dec 12, 2024
b72f592
feat(helm): Add startupProbe to ingester
lindhe Dec 12, 2024
d4a6286
feat(helm): Parameterize readinessProbe in canary
lindhe Dec 12, 2024
3be90b2
feat(helm): Add health probes to loki-canary
lindhe Dec 12, 2024
5aef046
feat(helm): Add health probes to pattern-ingester
lindhe Dec 12, 2024
68739ef
feat(helm): Add startupProbe to querier
lindhe Dec 12, 2024
e32c7ab
feat(helm): Add health probes to query-frontend
lindhe Dec 12, 2024
4bd3969
feat(helm): Add startupProbe to query-scheduler
lindhe Dec 12, 2024
4292176
feat(helm): Add health probes to read
lindhe Dec 12, 2024
45660ea
feat(helm): Add health probes to ruler
lindhe Dec 12, 2024
4d9ab1c
feat(helm): Add startupProbe to loki-sc-rules
lindhe Dec 12, 2024
ed590a7
feat(helm): Add startupProbe to table-manager
lindhe Dec 12, 2024
0a7b6c7
feat(helm): Add health probes to write
lindhe Dec 12, 2024
0cc8969
chore(helm): Update reference.md
lindhe Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
225 changes: 225 additions & 0 deletions docs/sources/setup/install/helm/reference.md
View file
Open in desktop

Large diffs are not rendered by default.

26 changes: 26 additions & 0 deletions production/helm/loki/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,32 @@ Entries should include a reference to the pull request that introduced the chang
- [BUGFIX] Disables the Helm test pod when `test.enabled=false`.
- [BUGFIX] Fix `enterprise.image.tag` to `3.3.0`
- [ENHANCEMENT] Bump Loki version to 3.3.2
- [BUGFIX] Fix broken `readinessProbe` and `livenessProbe` helper template definitons.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` for the nginx container in the gateway pods.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` for admin-api.
- [ENHANCEMENT] Add configurable `startupProbe` to the loki-sc-rules sidecar container in the backend pods.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to the loki container in the backend pods.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to bloom-builder.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to bloom-gateway.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to bloom-planner.
- [ENHANCEMENT] Add configurable `livenessProbe`, `readinessProbe` and `startupProbe` to the exporter container in the memcached pods.
- [ENHANCEMENT] Add configurable `livenessProbe`, `readinessProbe` and `startupProbe` to the memcached container in the memcached pods.
- [ENHANCEMENT] Add configurable `startupProbe` to compactor.
- [ENHANCEMENT] Add configurable `startupProbe` to distributor.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` for the gateway container in the enterprise-gateway pods.
- [ENHANCEMENT] Add configurable `startupProbe` to index-gateway.
- [ENHANCEMENT] Add configurable `startupProbe` to ingester.
- [ENHANCEMENT] Make `readinessProbe` configurable in loki-canary.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to loki-canary.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to pattern-ingester.
- [ENHANCEMENT] Add configurable `startupProbe` to querier.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to query-frontend.
- [ENHANCEMENT] Add configurable `startupProbe` to query-scheduler.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to the loki container in the read pods.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to ruler.
- [ENHANCEMENT] Add configurable `startupProbe` to the loki-sc-rules sidecar container in the single-binary pods.
- [ENHANCEMENT] Add configurable `startupProbe` to table-manager.
- [ENHANCEMENT] Add configurable `livenessProbe` and `startupProbe` to write.

## 6.23.0

Expand Down
12 changes: 11 additions & 1 deletion production/helm/loki/templates/admin-api/deployment-admin-api.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,18 @@ spec:
- name: http-memberlist
containerPort: 7946
protocol: TCP
{{- with .Values.adminApi.livenessProbe }}
livenessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.adminApi.readinessProbe }}
readinessProbe:
{{- toYaml .Values.adminApi.readinessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.adminApi.startupProbe }}
startupProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
resources:
{{- toYaml .Values.adminApi.resources | nindent 12 }}
securityContext:
Expand Down
16 changes: 15 additions & 1 deletion production/helm/loki/templates/backend/statefulset-backend.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,10 @@ spec:
readinessProbe:
{{- toYaml .Values.sidecar.readinessProbe | nindent 12 }}
{{- end }}
{{- if .Values.sidecar.startupProbe }}
startupProbe:
{{- toYaml .Values.sidecar.startupProbe | nindent 12 }}
{{- end }}
{{- if .Values.sidecar.resources }}
resources:
{{- toYaml .Values.sidecar.resources | nindent 12 }}
Expand Down Expand Up @@ -183,8 +187,18 @@ spec:
{{- end }}
securityContext:
{{- toYaml .Values.loki.containerSecurityContext | nindent 12 }}
{{- with .Values.loki.livenessProbe }}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it's better to split liveness probes for backend/read/write components in SSM mode. The liveness URLs/probes could be different for those components, isn't it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean SSD mode? I heard that was recently deprecated, so maybe best to not make any changes there? 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But in general, I agree with your sentiment! I just used .Values.loki because other things used it…

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simple Scalable Mode with 3 components (backend, read and write), maybe it's also known as SSD, not sure. I haven't heard it's deprecated. We just have a weird issue now with hanging read component after some backend unavailability and we are thinking it's good idea to use liveness probe on read component to restart it when it's "hanging". But this "liveness" URL (I use lokiread/loki/api/v1/labels for example) will be unique for reader only and it's better to configure it separately. Any way, thank you for your PR, I will be really happy if it will be merged soon!

Copy link

@rainchik rainchik Feb 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just used .Values.loki because other things used it

Yes, I've seen the same pattern for readiness probes in main, I'm not sure it's the right way to use common settings for all components

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grafana Labs is moving away from recommending SSD mode, so while it's not officially deprecated, it's no longer recommended as a deployment mode because a microservice deployment lets you better scale individual components.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the info, @JStickler .
We will check for transition to microservice approach then.
@lindhe I think my suggestion is not actual now, we can close this thread.

livenessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.loki.readinessProbe }}
readinessProbe:
{{- toYaml .Values.loki.readinessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.loki.startupProbe }}
startupProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: config
mountPath: /etc/loki/config
Expand Down
12 changes: 11 additions & 1 deletion production/helm/loki/templates/bloom-builder/deployment-bloom-builder.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,18 @@ spec:
{{- end }}
securityContext:
{{- toYaml .Values.loki.containerSecurityContext | nindent 12 }}
{{- with .Values.loki.livenessProbe }}
livenessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.loki.readinessProbe }}
readinessProbe:
{{- toYaml .Values.loki.readinessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.loki.startupProbe }}
startupProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: config
mountPath: /etc/loki/config
Expand Down
38 changes: 31 additions & 7 deletions production/helm/loki/templates/bloom-gateway/_helpers-bloom-gateway.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,44 @@ bloom gateway selector labels
app.kubernetes.io/component: bloom-gateway
{{- end }}

{{/*
bloom gateway livenessProbe
*/}}
{{- define "loki.bloomGateway.livenessProbe" }}
{{- if .Values.bloomGateway.livenessProbe }}
livenessProbe:
{{- toYaml .Values.bloomGateway.livenessProbe | nindent 2 }}
{{- else if .Values.loki.livenessProbe }}
livenessProbe:
{{- toYaml .Values.loki.livenessProbe | nindent 2 }}
{{- end }}
{{- end }}

{{/*
bloom gateway readinessProbe
*/}}
{{- define "loki.bloomGateway.readinessProbe" -}}
{{- with .Values.bloomGateway.readinessProbe }}
{{- define "loki.bloomGateway.readinessProbe" }}
{{- if .Values.bloomGateway.readinessProbe }}
readinessProbe:
{{- toYaml . | nindent 2 }}
{{- else }}
{{- with .Values.loki.readinessProbe }}
{{- toYaml .Values.bloomGateway.readinessProbe | nindent 2 }}
{{- else if .Values.loki.readinessProbe }}
readinessProbe:
{{- toYaml . | nindent 2 }}
{{- toYaml .Values.loki.readinessProbe | nindent 2 }}
{{- end }}
{{- end }}

{{/*
bloom gateway startupProbe
*/}}
{{- define "loki.bloomGateway.startupProbe" }}
{{- if .Values.bloomGateway.startupProbe }}
startupProbe:
{{- toYaml .Values.bloomGateway.startupProbe | nindent 2 }}
{{- else if .Values.loki.startupProbe }}
startupProbe:
{{- toYaml .Values.loki.startupProbe | nindent 2 }}
{{- end }}
{{- end }}
{{- end -}}

{{/*
bloom gateway priority class name
Expand Down
2 changes: 2 additions & 0 deletions production/helm/loki/templates/bloom-gateway/statefulset-bloom-gateway.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,9 @@ spec:
{{- end }}
securityContext:
{{- toYaml .Values.loki.containerSecurityContext | nindent 12 }}
{{- include "loki.bloomGateway.livenessProbe" . | nindent 10 }}
{{- include "loki.bloomGateway.readinessProbe" . | nindent 10 }}
{{- include "loki.bloomGateway.startupProbe" . | nindent 10 }}
volumeMounts:
- name: temp
mountPath: /tmp
Expand Down
38 changes: 31 additions & 7 deletions production/helm/loki/templates/bloom-planner/_helpers-bloom-planner.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,44 @@ bloom planner selector labels
app.kubernetes.io/component: bloom-planner
{{- end }}

{{/*
bloom planner livenessProbe
*/}}
{{- define "loki.bloomPlanner.livenessProbe" }}
{{- if .Values.bloomPlanner.livenessProbe }}
livenessProbe:
{{- toYaml .Values.bloomPlanner.livenessProbe | nindent 2 }}
{{- else if .Values.loki.livenessProbe }}
livenessProbe:
{{- toYaml .Values.loki.livenessProbe | nindent 2 }}
{{- end }}
{{- end }}

{{/*
bloom planner readinessProbe
*/}}
{{- define "loki.bloomPlanner.readinessProbe" -}}
{{- with .Values.bloomPlanner.readinessProbe }}
{{- define "loki.bloomPlanner.readinessProbe" }}
{{- if .Values.bloomPlanner.readinessProbe }}
readinessProbe:
{{- toYaml . | nindent 2 }}
{{- else }}
{{- with .Values.loki.readinessProbe }}
{{- toYaml .Values.bloomPlanner.readinessProbe | nindent 2 }}
{{- else if .Values.loki.readinessProbe }}
readinessProbe:
{{- toYaml . | nindent 2 }}
{{- toYaml .Values.loki.readinessProbe | nindent 2 }}
{{- end }}
{{- end }}

{{/*
bloom planner startupProbe
*/}}
{{- define "loki.bloomPlanner.startupProbe" }}
{{- if .Values.bloomPlanner.startupProbe }}
startupProbe:
{{- toYaml .Values.bloomPlanner.startupProbe | nindent 2 }}
{{- else if .Values.loki.startupProbe }}
startupProbe:
{{- toYaml .Values.loki.startupProbe | nindent 2 }}
{{- end }}
{{- end }}
{{- end -}}

{{/*
bloom planner priority class name
Expand Down
2 changes: 2 additions & 0 deletions production/helm/loki/templates/bloom-planner/statefulset-bloom-planner.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,9 @@ spec:
{{- end }}
securityContext:
{{- toYaml .Values.loki.containerSecurityContext | nindent 12 }}
{{- include "loki.bloomPlanner.livenessProbe" . | nindent 10 }}
{{- include "loki.bloomPlanner.readinessProbe" . | nindent 10 }}
{{- include "loki.bloomPlanner.startupProbe" . | nindent 10 }}
volumeMounts:
- name: temp
mountPath: /tmp
Expand Down
37 changes: 23 additions & 14 deletions production/helm/loki/templates/compactor/_helpers-compactor.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,32 +32,41 @@ compactor image
{{/*
compactor readinessProbe
*/}}
{{- define "loki.compactor.readinessProbe" -}}
{{- with .Values.compactor.readinessProbe }}
{{- define "loki.compactor.readinessProbe" }}
{{- if .Values.compactor.readinessProbe }}
readinessProbe:
{{- toYaml . | nindent 2 }}
{{- else }}
{{- with .Values.loki.readinessProbe }}
{{- toYaml .Values.compactor.readinessProbe | nindent 2 }}
{{- else if .Values.loki.readinessProbe }}
readinessProbe:
{{- toYaml . | nindent 2 }}
{{- toYaml .Values.loki.readinessProbe | nindent 2 }}
{{- end }}
{{- end }}
{{- end -}}

{{/*
compactor livenessProbe
*/}}
{{- define "loki.compactor.livenessProbe" -}}
{{- with .Values.compactor.livenessProbe }}
{{- define "loki.compactor.livenessProbe" }}
{{- if .Values.compactor.livenessProbe }}
livenessProbe:
{{- toYaml . | nindent 2 }}
{{- else }}
{{- with .Values.loki.livenessProbe }}
{{- toYaml .Values.compactor.livenessProbe | nindent 2 }}
{{- else if .Values.loki.livenessProbe }}
livenessProbe:
{{- toYaml . | nindent 2 }}
{{- toYaml .Values.loki.livenessProbe | nindent 2 }}
{{- end }}
{{- end }}

{{/*
compactor startupProbe
*/}}
{{- define "loki.compactor.startupProbe" }}
{{- if .Values.compactor.startupProbe }}
startupProbe:
{{- toYaml .Values.compactor.startupProbe | nindent 2 }}
{{- else if .Values.loki.startupProbe }}
startupProbe:
{{- toYaml .Values.loki.startupProbe | nindent 2 }}
{{- end }}
{{- end }}
{{- end -}}

{{/*
compactor priority class name
Expand Down
1 change: 1 addition & 0 deletions production/helm/loki/templates/compactor/statefulset-compactor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ spec:
{{- toYaml .Values.loki.containerSecurityContext | nindent 12 }}
{{- include "loki.compactor.readinessProbe" . | nindent 10 }}
{{- include "loki.compactor.livenessProbe" . | nindent 10 }}
{{- include "loki.compactor.startupProbe" . | nindent 10 }}
volumeMounts:
- name: temp
mountPath: /tmp
Expand Down
12 changes: 10 additions & 2 deletions production/helm/loki/templates/distributor/deployment-distributor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,18 @@ spec:
{{- end }}
securityContext:
{{- toYaml .Values.loki.containerSecurityContext | nindent 12 }}
{{- with .Values.loki.readinessProbe }}
readinessProbe:
{{- toYaml .Values.loki.readinessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.loki.livenessProbe }}
livenessProbe:
{{- toYaml .Values.loki.livenessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.loki.startupProbe }}
startupProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: config
mountPath: /etc/loki/config
Expand Down
14 changes: 12 additions & 2 deletions production/helm/loki/templates/gateway/deployment-gateway-enterprise.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ spec:
topologySpreadConstraints:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
{{- end }}
serviceAccountName: {{ template "loki.serviceAccountName" . }}
{{- if .Values.enterpriseGateway.priorityClassName }}
priorityClassName: {{ .Values.enterpriseGateway.priorityClassName }}
Expand Down Expand Up @@ -110,8 +110,18 @@ spec:
- name: http-metrics
containerPort: 3100
protocol: TCP
{{- with .Values.enterpriseGateway.livenessProbe }}
livenessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.enterpriseGateway.readinessProbe }}
readinessProbe:
{{- toYaml .Values.enterpriseGateway.readinessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.enterpriseGateway.startupProbe }}
startupProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
resources:
{{- toYaml .Values.enterpriseGateway.resources | nindent 12 }}
securityContext:
Expand Down
12 changes: 11 additions & 1 deletion production/helm/loki/templates/gateway/deployment-gateway-nginx.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,18 @@ spec:
envFrom:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.gateway.livenessProbe }}
livenessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.gateway.readinessProbe }}
readinessProbe:
{{- toYaml .Values.gateway.readinessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.gateway.startupProbe }}
startupProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
securityContext:
{{- toYaml .Values.gateway.containerSecurityContext | nindent 12 }}
{{- with .Values.gateway.lifecycle }}
Expand Down
12 changes: 10 additions & 2 deletions production/helm/loki/templates/index-gateway/statefulset-index-gateway.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,10 +102,18 @@ spec:
{{- end }}
securityContext:
{{- toYaml .Values.loki.containerSecurityContext | nindent 12 }}
{{- with .Values.loki.readinessProbe }}
readinessProbe:
{{- toYaml .Values.loki.readinessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.loki.livenessProbe }}
livenessProbe:
{{- toYaml .Values.loki.livenessProbe | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.loki.startupProbe }}
startupProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: config
mountPath: /etc/loki/config
Expand Down
Loading
Loading