ACI provider : panic: runtime error: invalid memory address or nil pointer dereference

[clebifradf]

Describe the Issue

ACI Connector is crashing at start.

Steps To Reproduce

The error returned by new NewACIProvider is unchecked :

azure-aci/cmd/virtual-kubelet/main.go

Line 222 in 50c56a0

p, err := azproviderv2.NewACIProvider(ctx, cfgPath, azConfig, azACIAPIs, cfg,

Expected behavior

ACI starts or returns an explicit error.

Virtual-kubelet version

Version 1.6.0

Kubernetes version

1.27

Additional context

Stacktrace :

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x110 pc=0x1c7ba23]
goroutine 1 [running]:
github.com/virtual-kubelet/azure-aci/pkg/provider.(*ACIProvider).capacity(0x0)
/workspace/pkg/provider/vk_node.go:42 +0x43
github.com/virtual-kubelet/azure-aci/pkg/provider.(*ACIProvider).ConfigureNode(0x0, {0xc0002bd0e0?, 0x22abca9?}, 0xc0001c0a90)
/workspace/pkg/provider/vk_node.go:21 +0x27
main.main.func7.1({{0x25af858, 0xc0001a27f0}, {0x25af718, 0xc0001a2820}, {0x25af8d0, 0xc0001a2850}, {0x25af920, 0xc0001a2880}, 0xc0001c0a90})
/workspace/cmd/virtual-kubelet/main.go:225 +0x246
github.com/virtual-kubelet/virtual-kubelet/node/nodeutil.NewNode({0x7ffebcf10273, 0x16}, 0xc00080faf0, {0xc00080fb20, 0x7, 0x0?})
/go/pkg/mod/github.com/virtual-kubelet/[email protected]/node/nodeutil/controller.go:341 +0x6c7
main.main.func7({0x25c3f68?, 0xc0003a5c20})
/workspace/cmd/virtual-kubelet/main.go:213 +0x285
main.main.func8(0xc00034e000, {0x2271a07?, 0x9?, 0x9?})
/workspace/cmd/virtual-kubelet/main.go:275 +0x1c6
github.com/spf13/cobra.(*Command).execute(0xc00034e000, {0xc000050150, 0x9, 0x9})
/go/pkg/mod/github.com/spf13/[email protected]/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0xc00034e000)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:1040 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:961
main.main()
/workspace/cmd/virtual-kubelet/main.go:331 +0xdfc

[ameetkonnur]

Same issue here. Added the ACI add-on and give me the below error for the aci-connector-linux Pod.
AKS Version ; 1.27.7
Similar issue faced by my customer as well.

[ameetkonnur]

Disabled the add-on and re-installed and it seems to have fixed the issue.
@clebifradf were you able to figure out the issue ?

[HouzuoGuo]

Same issue here:

⋊> howard@ws ⋊> ~ k -n kube-system logs -f pod/aci-connector-linux-c6cbb46d4-rmpxh
Defaulted container "aci-connector-linux" out of: aci-connector-linux, init-validation (init)
Flag --provider has been deprecated, only one provider is supported
time="2024-01-06T16:31:04Z" level=info msg="features [init-container confidential-compute events] enabled"
I0106 16:31:04.831412       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca::/etc/kubernetes/certs/ca.crt"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x110 pc=0x167bb20]

goroutine 1 [running]:
github.com/virtual-kubelet/azure-aci/pkg/provider.(*ACIProvider).capacity(0x0)
        /workspace/pkg/provider/vk_node.go:42 +0x30
github.com/virtual-kubelet/azure-aci/pkg/provider.(*ACIProvider).ConfigureNode(0x0, {0x4000102990?, 0x1fbb678?}, 0x4000161890)
        /workspace/pkg/provider/vk_node.go:21 +0x24
main.main.func7.1({{0x1fbb7b8, 0x4000102990}, {0x1fbb678, 0x40001029c0}, {0x1fbb830, 0x40001029f0}, {0x1fbb880, 0x4000102a20}, 0x4000161890})
        /workspace/cmd/virtual-kubelet/main.go:225 +0x1b8
github.com/virtual-kubelet/virtual-kubelet/node/nodeutil.NewNode({0xffffd4415825, 0x16}, 0x400068fad0, {0x400068fb00, 0x7, 0x1c284?})
        /go/pkg/mod/github.com/virtual-kubelet/[email protected]/node/nodeutil/controller.go:341 +0x528
main.main.func7({0x1fcff98?, 0x4000619f80})
        /workspace/cmd/virtual-kubelet/main.go:213 +0x1e8
main.main.func8(0x400004f200, {0x1c7e747?, 0x9?, 0x9?})
        /workspace/cmd/virtual-kubelet/main.go:275 +0x1b0
github.com/spf13/cobra.(*Command).execute(0x400004f200, {0x400004c150, 0x9, 0x9})
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:920 +0x5ac
github.com/spf13/cobra.(*Command).ExecuteC(0x400004f200)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:1040 +0x338
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:961
main.main()
        /workspace/cmd/virtual-kubelet/main.go:331 +0xc14
⋊> howard@ws ⋊> ~ k -n kube-system describe pod/aci-connector-linux-c6cbb46d4-rmpxh
Name:                 aci-connector-linux-c6cbb46d4-rmpxh
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      aci-connector-linux
Node:                 aks-syspool-15722634-vmss000000/10.224.0.4
Start Time:           Sat, 06 Jan 2024 16:29:36 +0000
Labels:               app=aci-connector-linux
                      kubernetes.azure.com/managedby=aks
                      pod-template-hash=c6cbb46d4
Annotations:          checksum/cloud-provider-config: 7d9f21c3c5a3634934ec96cafb0ead9561c2c6f4e6f9b70d8a10197f599f230c
                      cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:               Running
IP:                   10.224.0.96
IPs:
  IP:           10.224.0.96
Controlled By:  ReplicaSet/aci-connector-linux-c6cbb46d4
Init Containers:
  init-validation:
    Container ID:   containerd://d2c689c2188f42cfa01be4c7e2533110516d0fbaac836762e3b6fceba749080f
    Image:          mcr.microsoft.com/oss/virtual-kubelet/init-validation:0.2.0
    Image ID:       mcr.microsoft.com/oss/virtual-kubelet/init-validation@sha256:b45031b51cf67bc02d59bd1a8e615d6f618688a58c872c61c16cf208d95ae237
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 06 Jan 2024 16:29:37 +0000
      Finished:     Sat, 06 Jan 2024 16:29:37 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      AKS_CREDENTIAL_LOCATION:             /etc/aks/azure.json
      AZURE_CLIENT_SECRET:                 <set to the key 'clientSecret' in secret 'aci-connector-linux'>  Optional: false
      ACI_EXTRA_USER_AGENT:                add-on/aks
      ACI_SUBNET_NAME:                     virtual-node-aci
      KUBE_DNS_IP:                         10.0.0.10
      VIRTUALNODE_USER_IDENTITY_CLIENTID:  6517668a-fc46-4a50-944e-45e7e13f3db2
      POD_NAME:                            aci-connector-linux-c6cbb46d4-rmpxh (v1:metadata.name)
      NAMESPACE:                           kube-system (v1:metadata.namespace)
    Mounts:
      /etc/aks/azure.json from aks-credential (rw)
      /etc/virtual-kubelet from credentials (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2kgb4 (ro)
Containers:
  aci-connector-linux:
    Container ID:  containerd://4fd2f0d28199e90c61d81454ab40f2f8356dff0552b4708349a503fa18846965
    Image:         mcr.microsoft.com/oss/virtual-kubelet/virtual-kubelet:1.6.0
    Image ID:      mcr.microsoft.com/oss/virtual-kubelet/virtual-kubelet@sha256:629e904504cff5c551abfab568927e46c27e66da935b19d020de4ad6005ce60d
    Port:          <none>
    Host Port:     <none>
    Command:
      virtual-kubelet
    Args:
      --provider
      azure
      --nodename
      virtual-node-aci-linux
      --os
      Linux
      --authentication-token-webhook=true
      --no-verify-clients=false
      --client-verify-ca=/etc/kubernetes/certs/ca.crt
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Sat, 06 Jan 2024 16:31:04 +0000
      Finished:     Sat, 06 Jan 2024 16:31:04 +0000
    Ready:          False
    Restart Count:  4
    Environment:
      AKS_CREDENTIAL_LOCATION:             /etc/acs/azure.json
      KUBERNETES_PORT:                     tcp://bettersup-k8s-dns-izcxjrow.hcp.westeurope.azmk8s.io:443
      KUBE_DNS_IP:                         10.0.0.10
      ENABLE_REAL_TIME_METRICS:            true
      KUBELET_PORT:                        10250
      AZURE_CLIENT_SECRET:                 <set to the key 'clientSecret' in secret 'aci-connector-linux'>  Optional: false
      APISERVER_CERT_LOCATION:             /etc/virtual-kubelet/cert.pem
      APISERVER_KEY_LOCATION:              /etc/virtual-kubelet/key.pem
      MASTER_URI:                          https://bettersup-k8s-dns-izcxjrow.hcp.westeurope.azmk8s.io
      CLUSTER_CIDR:                        10.224.0.0/16
      KUBERNETES_PORT_443_TCP:             tcp://bettersup-k8s-dns-izcxjrow.hcp.westeurope.azmk8s.io:443
      VKUBELET_POD_IP:                      (v1:status.podIP)
      ACI_EXTRA_USER_AGENT:                add-on/aks
      ACI_SUBNET_NAME:                     virtual-node-aci
      VIRTUALNODE_USER_IDENTITY_CLIENTID:  6517668a-fc46-4a50-944e-45e7e13f3db2
      KUBERNETES_PORT_443_TCP_ADDR:        bettersup-k8s-dns-izcxjrow.hcp.westeurope.azmk8s.io
      KUBERNETES_SERVICE_HOST:             bettersup-k8s-dns-izcxjrow.hcp.westeurope.azmk8s.io
    Mounts:
      /etc/acs/azure.json from aks-credential (rw)
      /etc/kubernetes/certs from certificates (ro)
      /etc/virtual-kubelet from credentials (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2kgb4 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/certs
    HostPathType:
  credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  aci-connector-linux
    Optional:    false
  aks-credential:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  kube-api-access-2kgb4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m20s                default-scheduler  Successfully assigned kube-system/aci-connector-linux-c6cbb46d4-rmpxh to aks-syspool-15722634-vmss000000
  Normal   Pulled     2m19s                kubelet            Container image "mcr.microsoft.com/oss/virtual-kubelet/init-validation:0.2.0" already present on machine
  Normal   Created    2m19s                kubelet            Created container init-validation
  Normal   Started    2m19s                kubelet            Started container init-validation
  Normal   Pulled     52s (x5 over 2m18s)  kubelet            Container image "mcr.microsoft.com/oss/virtual-kubelet/virtual-kubelet:1.6.0" already present on machine
  Normal   Created    52s (x5 over 2m18s)  kubelet            Created container aci-connector-linux
  Normal   Started    52s (x5 over 2m18s)  kubelet            Started container aci-connector-linux
  Warning  BackOff    51s (x7 over 2m16s)  kubelet            Back-off restarting failed container aci-connector-linux in pod aci-connector-linux-c6cbb46d4-rmpxh_kube-system(dd5ee92b-637a-476e-b422-ff00c4f5a94f)

[smritidahal653]

Addressed in #582