Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turndown scaledown not works ( EKS) #42

Open
billmetangmo opened this issue May 25, 2022 · 0 comments
Open

Turndown scaledown not works ( EKS) #42

billmetangmo opened this issue May 25, 2022 · 0 comments
Labels
bug Something isn't working cost savings support

Comments

@billmetangmo
Copy link

billmetangmo commented May 25, 2022

Hello,

I wanted to use turndown to stop my dev cluster during off hours. It's an EKS cluster.
I succeeded to create the example schedule .
However when the schedule is supposed to happen, it failed with the following error :

E0525 11:47:00.907323       1 schedulecontroller.go:190] TurndownSchedule 'example-schedule' in work queue no longer exists
I0525 11:47:14.319434       1 namedlogger.go:24] [TurndownScheduler] Schedule Created: &{Current:scaledown ScaleDownID:1adae7c3-6f07-4837-baec-a5ed8a8efad0 ScaleDownTime:2022-05-25 11:55:00 +0000 UTC ScaleDownMetadata:map[repeat:daily type:scaledown] ScaleUpID:afcc2f9c-4324-4a29-880d-a532546857ef ScaleUpTime:2022-05-25 12:45:00 +0000 UTC ScaleUpMetadata:map[repeat:daily type:scaleup]}
I0525 11:47:14.339321       1 event.go:282] Event(v1.ObjectReference{Kind:"TurndownSchedule", Namespace:"", Name:"example-schedule", UID:"a6b0311e-8430-423a-ae1f-26935b7279ee", APIVersion:"kubecost.k8s.io/v1alpha1", ResourceVersion:"1476199", FieldPath:""}): type: 'Normal' reason: 'ScheduleTurndownSuccess' Successfully scheduled turndown
I0525 11:55:00.000175       1 turndownscheduler.go:404] -- Scale Down --
I0525 11:55:00.009267       1 namedlogger.go:24] [TurndownScheduler] Turndown Pod does not exist on expected host node. Preparing environment...
I0525 11:55:00.009306       1 namedlogger.go:24] [Turndown] Creating or Getting the Target Host Node...
I0525 11:55:00.419785       1 namedlogger.go:24] [MasterlessStrategy] Finite node backed cluster. Creating singleton nodepool for turndown.
I0525 11:55:00.665213       1 namedlogger.go:48] [Error] Failed to prepare current turndown environment. Cancelling. Err=AccessDeniedException: 
	status code: 403, request id: e5777622-11c8-4beb-aed1-1c1073c1925d
I0525 11:55:00.665544       1 scheduler.go:188] Job was cancelled: afcc2f9c-4324-4a29-880d-a532546857ef
I0525 11:55:00.685926       1 namedlogger.go:24] [TurndownScheduler] Turndown Schedule Successfully Cancelled

The error seems to be an "Access Denied" however i created the credentials with the suggested policy https://github.com/kubecost/cluster-turndown#eks--aws-kops-setup
Do you have any idea ?

Regards

@billmetangmo billmetangmo changed the title Turndown scaledown not works Turndown scaledown not works ( EKS) May 25, 2022
@Adam-Stack-PM Adam-Stack-PM added enhancement New feature or request needs-triage and removed enhancement New feature or request labels Jun 20, 2022
@michaelmdresser michaelmdresser added the bug Something isn't working label Aug 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cost savings support
Projects
None yet
Development

No branches or pull requests

4 participants