Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Operator: unhealthy ingesters not leaving the ring #15702

Open
aleert opened this issue Jan 11, 2025 · 1 comment · May be fixed by #15703
Open

Operator: unhealthy ingesters not leaving the ring #15702

aleert opened this issue Jan 11, 2025 · 1 comment · May be fixed by #15703
Assignees
@xperimental
Labels
sig/operator

Comments

@aleert
Copy link
Contributor

aleert commented Jan 11, 2025

Describe the bug
After network issues within our clusters we found, that ingesters were not able to join ring so we had to manually remove them.
There are multiple issues, describing this behavior, eg #8615 and #14847 .

Suggested fix would be to add autoforget_unhealthy flag to ingester config by default, as there seems to be no downsides for it.

Expected behavior
Unhealthy ingester leaving their ring after a timeout.

Environment:
Kubernetes 1.27
@aleert aleert linked a pull request Jan 11, 2025 that will close this issue
Open
6 tasks
@xperimental
Copy link
Collaborator

Hi @aleert ,

Can you provide more information on how to reproduce issue?

I have recently tried to reproduce a very similar report, but for me the ingesters instantly became healthy again after the network issues were removed.

@xperimental xperimental self-assigned this Jan 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xperimental xperimental

Labels
sig/operator
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(operator): autoforget unhealthy ingesters aleert/loki
2 participants
@xperimental @aleert