| title | description | author | image | marp | theme | url | class | video_embed |
|---|---|---|---|---|---|---|---|---|
Proactive cluster autoscaling in Kubernetes |
Scaling nodes in a Kubernetes cluster could take several minutes with the default settings. In this webinar, Chris will demo live how you can configure Pod Priorities and a placeholder pod to pre-warm node instances for quicker scaling. By the end of the session, you will master how to align the Horizontal Pod Autoscaler and Cluster Autoscaler for blazing fast scaling even with occasional traffic spikes. |
Chris Nesbitt-Smith |
true |
themes/esynergy |
lead |
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/o1U9k-Ng8D0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> |
- Learnk8s - Instructor+consultant
- Crown Prosecution Service (UK gov) - Consultant
- Opensource
- Memory Utilization
- CPU Utilization
- Pending pods
Memory UtilizationCPU Utilization- Pending pods
---
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
<style scoped> iframe { width: 100%; height: 100%; } section { padding:0; } </style> <iframe src="https://blackfriday.appvia.io/" title="BlackFriday" frameborder="0"></iframe>
- Don't scale!
- (Pre) scale
- Don't scale!
- (Pre) scale
<style scoped> iframe { width: 105%; height: 105%; transform-origin: top left; transform: scale(0.95); } section { padding:0; } </style> <iframe src="https://learnk8s.io/kubernetes-instance-calculator" title="instance calc" frameborder="0"></iframe>
<style scoped> iframe { width: 100%; height: 100%; } section { padding: 0; } </style> <iframe src="http://localhost:8001/static" title="" frameborder="0"></iframe>
<style scoped> section { padding:0; font-size: 0.01em /* display: initial; */ } </style>
default 0s Normal ScalingReplicaSet deployment/podinfo Scaled up replica set podinfo-8558cfcd5d to 5
default 0s Normal Scheduled pod/podinfo-8558cfcd5d-h4c5j Successfully assigned default/podinfo-8558cfcd5d-h4c5j to lke74013-115226-6331fbd57928
default 0s Normal SuccessfulCreate replicaset/podinfo-8558cfcd5d Created pod: podinfo-8558cfcd5d-h4c5j
default 0s Normal Scheduled pod/podinfo-8558cfcd5d-z9hvq Successfully assigned default/podinfo-8558cfcd5d-z9hvq to lke74013-115226-6331fbd57928
default 0s Normal Scheduled pod/podinfo-8558cfcd5d-d5khs Successfully assigned default/podinfo-8558cfcd5d-d5khs to lke74013-115226-6331fbd57928
default 0s Normal SuccessfulCreate replicaset/podinfo-8558cfcd5d Created pod: podinfo-8558cfcd5d-z9hvq
default 0s Normal SuccessfulCreate replicaset/podinfo-8558cfcd5d Created pod: podinfo-8558cfcd5d-d5khs
default 0s Warning FailedScheduling pod/podinfo-8558cfcd5d-5x2v7 0/2 nodes are available: 1 Insufficient memory, 1 node(s) didn't match Pod's node affinity/selector.
default 0s Normal Pulling pod/podinfo-8558cfcd5d-d5khs Pulling image "stefanprodan/podinfo"
default 0s Normal Pulling pod/podinfo-8558cfcd5d-h4c5j Pulling image "stefanprodan/podinfo"
default 0s Normal Pulling pod/podinfo-8558cfcd5d-z9hvq Pulling image "stefanprodan/podinfo"
default 0s Normal Pulled pod/podinfo-8558cfcd5d-d5khs Successfully pulled image "stefanprodan/podinfo" in 1.708158076s
default 0s Normal Created pod/podinfo-8558cfcd5d-d5khs Created container podinfo
default 0s Normal Started pod/podinfo-8558cfcd5d-d5khs Started container podinfo
default 0s Normal Pulled pod/podinfo-8558cfcd5d-h4c5j Successfully pulled image "stefanprodan/podinfo" in 3.104773992s
default 0s Normal Created pod/podinfo-8558cfcd5d-h4c5j Created container podinfo
default 0s Normal Started pod/podinfo-8558cfcd5d-h4c5j Started container podinfo
default 0s Normal NotTriggerScaleUp pod/podinfo-8558cfcd5d-5x2v7 pod didn't trigger scale-up: 1 node(s) didn't match Pod's node affinity/selector
default 0s Normal Pulled pod/podinfo-8558cfcd5d-z9hvq Successfully pulled image "stefanprodan/podinfo" in 8.442357379s
default 0s Normal Created pod/podinfo-8558cfcd5d-z9hvq Created container podinfo
default 0s Normal Started pod/podinfo-8558cfcd5d-z9hvq Started container podinfo
default 0s Warning FailedScheduling pod/podinfo-8558cfcd5d-5x2v7 0/2 nodes are available: 1 Insufficient memory, 1 node(s) didn't match Pod's node affinity/selector.
default 1s Normal NodeHasSufficientMemory node/lke74013-115226-6332129f84dd Node lke74013-115226-6332129f84dd status is now: NodeHasSufficientMemory
default 1s Normal NodeHasNoDiskPressure node/lke74013-115226-6332129f84dd Node lke74013-115226-6332129f84dd status is now: NodeHasNoDiskPressure
kube-system 0s Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-tpxpn
kube-system 0s Normal Scheduled pod/csi-linode-node-fvk46 Successfully assigned kube-system/csi-linode-node-fvk46 to lke74013-115226-6332129f84dd
kube-system 0s Normal Scheduled pod/calico-node-tpxpn Successfully assigned kube-system/calico-node-tpxpn to lke74013-115226-6332129f84dd
kube-system 0s Normal SuccessfulCreate daemonset/csi-linode-node Created pod: csi-linode-node-fvk46
kube-system 0s Normal Scheduled pod/kube-proxy-ds24f Successfully assigned kube-system/kube-proxy-ds24f to lke74013-115226-6332129f84dd
kube-system 0s Normal SuccessfulCreate daemonset/kube-proxy Created pod: kube-proxy-ds24f
default 0s Normal Synced node/lke74013-115226-6332129f84dd Node synced successfully
default 0s Normal Starting node/lke74013-115226-6332129f84dd Starting kubelet.
default 0s Normal NodeAllocatableEnforced node/lke74013-115226-6332129f84dd Updated Node Allocatable limit across pods
default 0s Normal NodeHasSufficientMemory node/lke74013-115226-6332129f84dd Node lke74013-115226-6332129f84dd status is now: NodeHasSufficientMemory
default 0s Normal NodeHasNoDiskPressure node/lke74013-115226-6332129f84dd Node lke74013-115226-6332129f84dd status is now: NodeHasNoDiskPressure
default 0s Normal NodeHasSufficientPID node/lke74013-115226-6332129f84dd Node lke74013-115226-6332129f84dd status is now: NodeHasSufficientPID
default 0s Normal RegisteredNode node/lke74013-115226-6332129f84dd Node lke74013-115226-6332129f84dd event: Registered Node lke74013-115226-6332129f84dd in Controller
kube-system 1s Normal Pulling pod/kube-proxy-ds24f Pulling image "linode/kube-proxy-amd64:v1.23.10"
kube-system 1s Normal Pulling pod/calico-node-tpxpn Pulling image "docker.io/calico/cni:v3.22.1"
kube-system 0s Normal Pulling pod/csi-linode-node-fvk46 Pulling image "bitnami/kubectl:1.16.3-debian-10-r36"
kube-system 0s Normal Pulled pod/kube-proxy-ds24f Successfully pulled image "linode/kube-proxy-amd64:v1.23.10" in 3.544455967s
kube-system 0s Normal Created pod/kube-proxy-ds24f Created container kube-proxy
kube-system 0s Normal Started pod/kube-proxy-ds24f Started container kube-proxy
default 0s Normal Starting node/lke74013-115226-6332129f84dd
kube-system 0s Normal Pulled pod/calico-node-tpxpn Successfully pulled image "docker.io/calico/cni:v3.22.1" in 8.06147771s
kube-system 1s Normal Created pod/calico-node-tpxpn Created container upgrade-ipam
kube-system 0s Normal Started pod/calico-node-tpxpn Started container upgrade-ipam
kube-system 0s Normal Pulled pod/calico-node-tpxpn Container image "docker.io/calico/cni:v3.22.1" already present on machine
kube-system 0s Normal Created pod/calico-node-tpxpn Created container install-cni
kube-system 0s Normal Started pod/calico-node-tpxpn Started container install-cni
kube-system 0s Normal Pulling pod/calico-node-tpxpn Pulling image "docker.io/calico/pod2daemon-flexvol:v3.22.1"
kube-system 0s Normal Pulled pod/csi-linode-node-fvk46 Successfully pulled image "bitnami/kubectl:1.16.3-debian-10-r36" in 11.657565568s
kube-system 0s Normal Created pod/csi-linode-node-fvk46 Created container init
kube-system 0s Normal Started pod/csi-linode-node-fvk46 Started container init
kube-system 0s Normal Pulling pod/csi-linode-node-fvk46 Pulling image "linode/csi-node-driver-registrar:v1.3.0"
default 0s Warning FailedScheduling pod/podinfo-8558cfcd5d-5x2v7 0/2 nodes are available: 1 Insufficient memory, 1 node(s) didn't match Pod's node affinity/selector.
kube-system 0s Normal Pulled pod/calico-node-tpxpn Successfully pulled image "docker.io/calico/pod2daemon-flexvol:v3.22.1" in 5.534993769s
kube-system 0s Normal Created pod/calico-node-tpxpn Created container flexvol-driver
kube-system 0s Normal Started pod/calico-node-tpxpn Started container flexvol-driver
kube-system 0s Normal Pulling pod/calico-node-tpxpn Pulling image "docker.io/calico/node:v3.22.1"
default 0s Normal NodeReady node/lke74013-115226-6332129f84dd Node lke74013-115226-6332129f84dd status is now: NodeReady
kube-system 0s Normal Pulled pod/csi-linode-node-fvk46 Successfully pulled image "linode/csi-node-driver-registrar:v1.3.0" in 4.741448054s
kube-system 0s Normal Created pod/csi-linode-node-fvk46 Created container csi-node-driver-registrar
kube-system 0s Normal Started pod/csi-linode-node-fvk46 Started container csi-node-driver-registrar
kube-system 0s Normal Pulling pod/csi-linode-node-fvk46 Pulling image "linode/linode-blockstorage-csi-driver:v0.5.0"
default 0s Normal Scheduled pod/podinfo-8558cfcd5d-5x2v7 Successfully assigned default/podinfo-8558cfcd5d-5x2v7 to lke74013-115226-6332129f84dd
default 0s Warning FailedCreatePodSandBox pod/podinfo-8558cfcd5d-5x2v7 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4d21d0493b5ac49eb7a173b9946fb379f9e0d98b1d55ab13547ad0826842bc6d" network for pod "podinfo-8558cfcd5d-5x2v7": networkPlugin cni failed to set up pod "podinfo-8558cfcd5d-5x2v7_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
default 0s Normal TaintManagerEviction pod/podinfo-8558cfcd5d-5x2v7 Cancelling deletion of Pod default/podinfo-8558cfcd5d-5x2v7
default 0s Normal SandboxChanged pod/podinfo-8558cfcd5d-5x2v7 Pod sandbox changed, it will be killed and re-created.
default 0s Warning FailedCreatePodSandBox pod/podinfo-8558cfcd5d-5x2v7 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5d02b7c46e44913d4fabb9e02dc77044a3163f976c0a46f47c398108ac3fa000" network for pod "podinfo-8558cfcd5d-5x2v7": networkPlugin cni failed to set up pod "podinfo-8558cfcd5d-5x2v7_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
default 0s Normal SandboxChanged pod/podinfo-8558cfcd5d-5x2v7 Pod sandbox changed, it will be killed and re-created.
default 0s Warning FailedCreatePodSandBox pod/podinfo-8558cfcd5d-5x2v7 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3447a34aeda5501828a9c476e0d6d7077eb7f895c8fbcd7022af1643865241c6" network for pod "podinfo-8558cfcd5d-5x2v7": networkPlugin cni failed to set up pod "podinfo-8558cfcd5d-5x2v7_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
default 0s Normal SandboxChanged pod/podinfo-8558cfcd5d-5x2v7 Pod sandbox changed, it will be killed and re-created.
default 0s Warning FailedCreatePodSandBox pod/podinfo-8558cfcd5d-5x2v7 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "77a8281777afde238f38cddc9141ed2e52c78540ccb4ad7b296da3fc8c20a6ef" network for pod "podinfo-8558cfcd5d-5x2v7": networkPlugin cni failed to set up pod "podinfo-8558cfcd5d-5x2v7_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
default 0s Normal SandboxChanged pod/podinfo-8558cfcd5d-5x2v7 Pod sandbox changed, it will be killed and re-created.
kube-system 0s Normal Pulled pod/calico-node-tpxpn Successfully pulled image "docker.io/calico/node:v3.22.1" in 6.353089879s
kube-system 0s Normal Created pod/calico-node-tpxpn Created container calico-node
kube-system 0s Normal Started pod/calico-node-tpxpn Started container calico-node
default 0s Warning FailedCreatePodSandBox pod/podinfo-8558cfcd5d-5x2v7 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0fc48fe077b786da531ebc656016468d3ae3f41e6cf520e27b638dbfbab580e2" network for pod "podinfo-8558cfcd5d-5x2v7": networkPlugin cni failed to set up pod "podinfo-8558cfcd5d-5x2v7_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
default 0s Normal SandboxChanged pod/podinfo-8558cfcd5d-5x2v7 Pod sandbox changed, it will be killed and re-created.
kube-system 0s Warning Unhealthy pod/calico-node-tpxpn Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/bird/bird.ctl: connect: no such file or directory
default 0s Normal Pulling pod/podinfo-8558cfcd5d-5x2v7 Pulling image "stefanprodan/podinfo"
kube-system 0s Warning Unhealthy pod/calico-node-tpxpn Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
kube-system 0s Normal Pulled pod/csi-linode-node-fvk46 Successfully pulled image "linode/linode-blockstorage-csi-driver:v0.5.0" in 6.754070918s
kube-system 0s Normal Created pod/csi-linode-node-fvk46 Created container csi-linode-plugin
kube-system 0s Normal Started pod/csi-linode-node-fvk46 Started container csi-linode-plugin
kube-system 0s Warning Unhealthy pod/calico-node-tpxpn Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused
kube-system 0s Warning Unhealthy pod/calico-node-tpxpn Readiness probe failed: 2022-09-26 21:01:13.334 [INFO][249] confd/health.go 180: Number of node(s) with BGP peering established = 1...
default 0s Normal Pulled pod/podinfo-8558cfcd5d-5x2v7 Successfully pulled image "stefanprodan/podinfo" in 4.44920297s
default 0s Normal Created pod/podinfo-8558cfcd5d-5x2v7 Created container podinfo
default 0s Normal Started pod/podinfo-8558cfcd5d-5x2v7 Started container podinfo
default 0s Normal ScalingReplicaSet deployment/podinfo Scaled down replica set podinfo-8558cfcd5d to 1
default 0s Normal Killing pod/podinfo-8558cfcd5d-d5khs Stopping container podinfo
default 0s Normal Killing pod/podinfo-8558cfcd5d-z9hvq Stopping container podinfo
default 0s Normal Killing pod/podinfo-8558cfcd5d-rxbmc Stopping container podinfo
default 0s Normal Killing pod/podinfo-8558cfcd5d-h4c5j Stopping container podinfo
default 0s Normal ScalingReplicaSet deployment/overprovisioning Scaled up replica set overprovisioning-7d874998c to 1
default 0s Normal Scheduled pod/overprovisioning-7d874998c-rgxf9 Successfully assigned default/overprovisioning-7d874998c-rgxf9 to lke74013-115226-6331fbd57928
default 0s Normal SuccessfulCreate replicaset/overprovisioning-7d874998c Created pod: overprovisioning-7d874998c-rgxf9
default 0s Normal Pulling pod/overprovisioning-7d874998c-rgxf9 Pulling image "k8s.gcr.io/pause"
default 0s Normal Pulled pod/overprovisioning-7d874998c-rgxf9 Successfully pulled image "k8s.gcr.io/pause" in 11.25774753s
default 0s Normal Created pod/overprovisioning-7d874998c-rgxf9 Created container pause
default 0s Normal Started pod/overprovisioning-7d874998c-rgxf9 Started container pause
default 0s Normal ScalingReplicaSet deployment/podinfo Scaled up replica set podinfo-8558cfcd5d to 5
default 0s Normal Scheduled pod/podinfo-8558cfcd5d-8m87n Successfully assigned default/podinfo-8558cfcd5d-8m87n to lke74013-115226-6332129f84dd
default 0s Normal Scheduled pod/podinfo-8558cfcd5d-hlrd8 Successfully assigned default/podinfo-8558cfcd5d-hlrd8 to lke74013-115226-6332129f84dd
default 0s Normal Scheduled pod/podinfo-8558cfcd5d-4mvj4 Successfully assigned default/podinfo-8558cfcd5d-4mvj4 to lke74013-115226-6332129f84dd
default 0s Warning FailedScheduling pod/podinfo-8558cfcd5d-mh96r 0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 Insufficient memory.
default 0s Normal Preempted pod/overprovisioning-7d874998c-rgxf9 Preempted by default/podinfo-8558cfcd5d-mh96r on node lke74013-115226-6331fbd57928
default 0s Normal Killing pod/overprovisioning-7d874998c-rgxf9 Stopping container pause
default 0s Warning FailedScheduling pod/overprovisioning-7d874998c-hm26l 0/3 nodes are available: 1 Insufficient cpu, 3 Insufficient memory.
default 0s Normal SuccessfulCreate replicaset/overprovisioning-7d874998c Created pod: overprovisioning-7d874998c-hm26l
default 0s Normal Pulling pod/podinfo-8558cfcd5d-8m87n Pulling image "stefanprodan/podinfo"
default 0s Normal Pulling pod/podinfo-8558cfcd5d-4mvj4 Pulling image "stefanprodan/podinfo"
default 0s Normal Pulling pod/podinfo-8558cfcd5d-hlrd8 Pulling image "stefanprodan/podinfo"
default 0s Normal Killing pod/overprovisioning-7d874998c-rgxf9 Stopping container pause
default 0s Warning FailedKillPod pod/overprovisioning-7d874998c-rgxf9 error killing pod: failed to "KillContainer" for "pause" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: 4fd559419e945392981b536fefe272c2bac1349002cb29d6d33ea3fbdb1bf98c"
default 0s Warning FailedScheduling pod/overprovisioning-7d874998c-hm26l 0/3 nodes are available: 3 Insufficient memory.
default 0s Normal Scheduled pod/podinfo-8558cfcd5d-mh96r Successfully assigned default/podinfo-8558cfcd5d-mh96r to lke74013-115226-6331fbd57928
default 0s Normal Pulled pod/podinfo-8558cfcd5d-8m87n Successfully pulled image "stefanprodan/podinfo" in 1.019384148s
default 0s Normal Created pod/podinfo-8558cfcd5d-8m87n Created container podinfo
default 0s Normal Started pod/podinfo-8558cfcd5d-8m87n Started container podinfo
default 0s Normal Pulling pod/podinfo-8558cfcd5d-mh96r Pulling image "stefanprodan/podinfo"
default 0s Normal Pulled pod/podinfo-8558cfcd5d-4mvj4 Successfully pulled image "stefanprodan/podinfo" in 2.018256028s
default 0s Normal Created pod/podinfo-8558cfcd5d-4mvj4 Created container podinfo
default 0s Normal Started pod/podinfo-8558cfcd5d-4mvj4 Started container podinfo
default 0s Normal Pulled pod/podinfo-8558cfcd5d-hlrd8 Successfully pulled image "stefanprodan/podinfo" in 2.996429351s
default 0s Normal Created pod/podinfo-8558cfcd5d-hlrd8 Created container podinfo
default 0s Normal Pulled pod/podinfo-8558cfcd5d-mh96r Successfully pulled image "stefanprodan/podinfo" in 1.446868994s
default 0s Normal Started pod/podinfo-8558cfcd5d-hlrd8 Started container podinfo
default 0s Normal Created pod/podinfo-8558cfcd5d-mh96r Created container podinfo
default 0s Normal Started pod/podinfo-8558cfcd5d-mh96r Started container podinfo
<style scoped> iframe { width: 100%; height: 100%; } section { padding:0; } </style> <iframe src="https://www.youtube-nocookie.com/embed/stNgDdSX_gE?modestbranding=1&rel=0&iv_load_policy=3&fs=0" title="scaling without placeholder" frameborder="0"></iframe>
<style scoped> iframe { width: 100%; height: 100%; } section { padding:0; } </style> <iframe src="https://www.youtube-nocookie.com/embed/i2EBF_wgCIo?modestbranding=1&rel=0&iv_load_policy=3&fs=0" title="scaling with placeholder" frameborder="0"></iframe>
<style scoped> pre { width: 45%; transform: scale(2); transform-origin: 0 0; } </style>
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: overprovisioning
value: -1
globalDefault: false<style scoped> iframe { width: 100%; height: 100%; } section { padding: 0; } </style> <iframe src="http://localhost:8001/static" title="" frameborder="0"></iframe>
<style scoped> iframe { width: 100%; height: 100%; } section { padding:0; } </style> <iframe src="https://www.youtube-nocookie.com/embed/kyi2UCzrENA?modestbranding=1&rel=0&iv_load_policy=3&fs=0" title="hpa reactive" frameborder="0"></iframe>
<style scoped> iframe { width: 100%; height: 100%; } section { padding:0; } </style> <iframe src="https://www.youtube-nocookie.com/embed/0uDj5Nqnlxc?modestbranding=1&rel=0&iv_load_policy=3&fs=0" title="hpa proactive" frameborder="0"></iframe>
<style scoped> pre { width: 45%; transform: scale(2); transform-origin: 0 0; } </style>
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass<style scoped> h2 { position: absolute; bottom: 1ch; left: 2vw; width: 95% } </style>
- cns.me
- talks.cns.me
- github.com/chrisns
- learnk8s.io
cns.me
cns.me