Lifecycle hook to properly drain instances

[nitrocode]

It would be handy to have a lifecycle hook on the autoscaling group to safely spin down instances when updating the AMI. The hook could pause the ec2 termination, wait for spacelift to finish any remaining jobs on its agents, drain the instance, and continue the termination.

references

• https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_lifecycle_hook
• https://circleci.com/blog/graceful-shutdown-using-aws/
• Add support for instance refresh #22

[xophere]

Yes please.

[lorengordon]

Yeah, this would be very handy. I have a spacelift stack that is actually managing this worker pool, and the pool is linked to the stack. A little self-referential I guess. Which is of course the problem. But, as it is, when I modify the launch template, the instance refresh kicks off, which terminates the runner that the stack run is on, crashing the worker as far as spacelift sees and often leaving the state locked and requiring manual intervention! Some kind of lifecycle hook to properly drain instances would resolve the problem without complicating the architecture.

[zhimsel]

👍🏻 this. I've been speaking to our Spacelift reps to try and get this feature worked on. 🤞🏻