Integrate kubelogin for Azure AD Authentication to AKS Clusters

[marianheinsen]

What problem are you facing?

We are provisioning Azure AKS Kubernetes Clusters using the Azure Provider by Upbound. After provisioning, we install software inside of the clusters using this Helm Provider as well as the related Kubernetes Provider. When using Local Accounts for authentication with the clusters API servers, this works out of the box. However, we would like to switch to authentication based on Azure AD Service Principals and Managed Identities, as this is the best practice and recommended by Azure. The Kubernetes Go Client which is used by this provider does not support this authentication method by default, so currently we can't use this provider with AKS clusters configured with Azure AD Authentication. As the same problem arises with standard kubectl usage and Terraform providers as well (see e.g. Azure/kubelogin#114), Azure provides the kubelogin client-go credential plugin which can be used to retrieve the user credentials and pass it to the Kubernetes Go Client.

How could Crossplane help solve your problem?

I propose to integrate the official kubelogin Go package into this provider (as well as the Kubernetes Provider), so that it's possible to use it with AKS clusters configured with Azure AD Authentication. I'm open to provide an implementation for this and open a PR. Also, I would be happy to hear the maintainer's thoughts about this. Are you open to such contribution? @turkenh @negz

[marianheinsen]

Related issue in Kubernetes Provider repo: crossplane-contrib/provider-kubernetes#105

[haarchri]

I'm encountering the same issue when trying to connect to an AKS cluster with authentication based on Azure AD Service Principals and Managed Identities. The error message indicates that the kubelogin executable is not found.

Error Details:

Type     Reason                   Age                 From                                Message
  ----     ------                   ----                ----                                -------
  Warning  CannotConnectToProvider  23m (x10 over 27m)  managed/release.helm.crossplane.io  cannot create new Kubernetes client: cannot create Kubernetes client: Get "https://test123-aks-23c35881-9a9e-4ee3-aef9-2b4cc02d7d00-rq3nt806.hcp.westus.azmk8s.io:443/api?timeout=32s": getting credentials: exec: executable kubelogin not found

It looks like you are trying to use a client-go credential plugin that is not installed.

To learn more about this feature, consult the documentation available at:
      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

kubelogin is not installed which is required to connect to AAD enabled cluster.

To learn more, please go to https://aka.ms/aks/kubelogin

Provided Kubeconfig:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xyz==
    server: https://test123-aks-23c35881-9a9e-4ee3-aef9-2b4cc02d7d00-rq3nt806.hcp.westus.azmk8s.io:443
  name: test123-aks-j5wsb-f2b4q
contexts:
- context:
    cluster: test123-aks-j5wsb-f2b4q
    user: clusterUser_start123-rg_test123-aks-j5wsb-f2b4q
  name: test123-aks-j5wsb-f2b4q
current-context: test123-aks-j5wsb-f2b4q
kind: Config
preferences: {}
users:
- name: clusterUser_start123-rg_test123-aks-j5wsb-f2b4q
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --environment
      - AzurePublicCloud
      - --server-id
      - 6dae42f8-1234-4678-1234-3960e28e3630
      - --client-id
      - 80faf920-1234-4b52-1234-a8e7bedfc67a
      - --tenant-id
      - b9925bc4-1234-4c37-1234-fa456d1bb1c7
      - --login
      - devicecode
      command: kubelogin
      env: null
      installHint: |2
        kubelogin is not installed which is required to connect to AAD enabled cluster.
        To learn more, please go to
        https://aka.ms/aks/kubelogin

[haarchri]

using the following manifest for kubernetesCluster:

apiVersion: containerservice.azure.upbound.io/v1beta1
kind: KubernetesCluster
metadata:
  name: test123-aks-j5wsb-wdh8r
spec:
  deletionPolicy: Delete
  forProvider:
    apiServerAccessProfile:
    - vnetIntegrationEnabled: true
    autoScalerProfile:
    - emptyBulkDeleteMax: "10"
      expander: random
      maxGracefulTerminationSec: "600"
      maxNodeProvisioningTime: 15m
      maxUnreadyNodes: 3
      maxUnreadyPercentage: 45
      newPodScaleUpDelay: 0s
      scaleDownDelayAfterAdd: 10m
      scaleDownDelayAfterDelete: 10s
      scaleDownDelayAfterFailure: 3m
      scaleDownUnneeded: 10m
      scaleDownUnready: 20m
      scaleDownUtilizationThreshold: "0.5"
      scanInterval: 10s
      skipNodesWithSystemPods: true
    azureActiveDirectoryRoleBasedAccessControl:
    - adminGroupObjectIds:
      - 6f49479a-1234-480b-1234-741b51a973a1
      azureRbacEnabled: true
      managed: true
      tenantId: b9925bc4-1234-4c37-1234-fa456d1bb1c7
    azurePolicyEnabled: true
    defaultNodePool:
    - customCaTrustEnabled: false
      enableAutoScaling: true
      enableHostEncryption: false
      enableNodePublicIp: false
      fipsEnabled: false
      kubeletDiskType: OS
      maxCount: 10
      maxPods: 30
      minCount: 1
      name: nodepool1
      nodeCount: 1
      onlyCriticalAddonsEnabled: false
      orchestratorVersion: 1.27.3
      osDiskSizeGb: 128
      osDiskType: Ephemeral
      osSku: Ubuntu
      scaleDownMode: Delete
      type: VirtualMachineScaleSets
      ultraSsdEnabled: false
      vmSize: Standard_DS3_v2
      workloadRuntime: OCIContainer
    dnsPrefix: test123-aks-d679da55-b462-415e-8b04-7487ac5bbd3d
    identity:
    - identityIds:
      - /subscriptions/038f2b7c-1234-43b8-1234-c9ad5da610a8/resourceGroups/start123-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test123-aks-j5wsb-q46cg
      type: UserAssigned
    imageCleanerIntervalHours: 48
    kubernetesVersion: 1.27.3
    location: westus
    networkProfile:
    - networkPlugin: azure
    nodeResourceGroup: MC_start123-rg_test123-aks-j5wsb-wdh8r_westus
    oidcIssuerEnabled: true
    omsAgent:
    - logAnalyticsWorkspaceId: /subscriptions/038f2b7c-1234-43b8-1234-c9ad5da610a8/resourceGroups/start123-rg/providers/Microsoft.OperationalInsights/workspaces/test123-aks-j5wsb-clzf4
      msiAuthForMonitoringEnabled: true
    publicNetworkAccessEnabled: true
    resourceGroupName: start123-rg
    roleBasedAccessControlEnabled: true
    runCommandEnabled: true
    skuTier: Standard
    tags:
      app: test123
      env: dev
      namespace: start123
    windowsProfile:
    - adminUsername: azureuser
    workloadIdentityEnabled: true
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
  writeConnectionSecretToRef:
    name: 7867facd-54cd-42ed-8402-bff9af903ba0-akscluster
    namespace: upbound-system

[haarchri]

I was playing around with the missing kubelogin - my first idea was to ship the kubelogin simply with the Dockerfile. The result is the following:

kubectl logs -n upbound-system crossplane-contrib-provider-helm-1e553bfdff7a-5f968cb5d8-49kvm
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory

The next thing I tried was to add the Azure CLI, which relies on a lot of Python. For that reason, I switched the base image and needed to install many other tools. But the result is still as follows:

kubectl logs -n upbound-system crossplane-contrib-provider-helm-4a269aae5ec5-6cd44f9464-sj8h7  
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.

What I tried then was to exec into the provider and did an az login with device authentication. After that, the Helm release was synced, and ready true.
I think adding the right credentials could do the trick, but overall, this way is not the best because the Docker image is more than 650MB, and we rely on some CLI and binary tools. Therefore, we should try to use the suggested kubelogin Go library for better integration.

for reference:

# Use Alpine as the base image
FROM alpine

ARG TARGETOS
ARG TARGETARCH

ENV HELM_CACHE_HOME /tmp/helm-cache
ENV HELM_CONFIG_HOME /tmp/helm-config
ENV HELM_DATA_HOME /tmp/helm-data

# Add necessary binaries
ADD bin/$TARGETOS\_$TARGETARCH/provider /usr/local/bin/provider-helm

# Download and unzip kubelogin binary
ADD https://github.com/Azure/kubelogin/releases/download/v0.0.33/kubelogin-${TARGETOS}-${TARGETARCH}.zip /tmp/kubelogin.zip
RUN unzip /tmp/kubelogin.zip -d /tmp && \
    mv /tmp/bin/$TARGETOS\_$TARGETARCH/kubelogin /usr/local/bin/ && \
    rm -rf /tmp/kubelogin.zip /tmp/bin

# Install Azure CLI
RUN apk add --no-cache --update python3 py3-pip && \
    apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && \
    pip3 install --no-cache-dir --prefer-binary azure-cli && \
    apk del build

# Create .azure directory
RUN mkdir -p /home/65532/.azure && chown -R 65532:65532 /home/65532/.azure

USER 65532
ENTRYPOINT ["provider-helm"]

[haarchri]

crossplane-contrib/provider-upjet-azure#459

[turkenh]

Thanks a lot @haarchri for your investigations. I agree that we should better fix this in the go code rather than introducing binaries and other dependencies as you've find out.

I believe we can come up with a similar API and solution as #109 here, and started looking the code at https://github.com/Azure/kubelogin. Based on my observations so far, the code there is not friendly to be imported as a library with private methods and inflexible options in the exposed methods. I will dig further.