conflicts with "FxVersion" when running terraform plan against manifest HCL

[chrisadkin-zz]

I have the following HCL:

provider "kubernetes-alpha" {
  config_path = "~/.kube/config"
}

resource "kubernetes_manifest" "arc" {
  provider = kubernetes-alpha

  manifest = {
    "apiVersion" = "arcdata.microsoft.com/v1alpha1"

    "kind" = "datacontroller"
    "metadata" = {
      "generation" = 1
      "name" = "arc"
      "namespace" = "arc"
    }
    "spec" = {
      "credentials" = {
        "controllerAdmin" = "controller-login-secret"
        "serviceAccount" = "sa-mssql-controller"
      }
      "docker" = {
        "imagePullPolicy" = "Always"
        "imageTag" = "public-preview-dec-2020"
        "registry" = "mcr.microsoft.com"
        "repository" = "arcdata"
      }
      "security" = {
        "allowDumps" = true
        "allowNodeMetricsCollection" = true
        "allowPodMetricsCollection" = true
        "allowRunAsRoot" = false
      }
      "services" = [
        {
          "name" = "controller"
          "port" = 30080
          "serviceType" = "LoadBalancer"
        },
        {
          "name" = "serviceProxy"
          "port" = 30777
          "serviceType" = "LoadBalancer"
        },
      ]
      "settings" = {
        "ElasticSearch" = {
          "vm.max_map_count" = "-1"
        }
        "azure" = {
          "connectionMode" = "Indirect"
          "location" = "eastus"
          "resourceGroup" = "myresourcegroup"
          "subscription" = "c82c901a-129a-435d-86e4-cc6b294590ae"
        }
        "controller" = {
          "displayName" = "arc"
          "enableBilling" = "True"
          "logs.rotation.days" = "7"
          "logs.rotation.size" = "5000"
        }
      }
      "storage" = {
        "data" = {
          "accessMode" = "ReadWriteOnce"
          "className" = "default"
          "size" = "15Gi"
        }
        "logs" = {
          "accessMode" = "ReadWriteOnce"
          "className" = "default"
          "size" = "10Gi"
        }
      }
    }
  }
}

When I run a terraform apply against this I get:

[Warning: This custom resource does not have an associated OpenAPI schema.
│
│   with kubernetes_manifest.arc,
│   on main.tf line 52, in resource "kubernetes_manifest" "arc":
│   52: resource "kubernetes_manifest" "arc" {
│
│ We could not find an OpenAPI schema for this custom resource. Updates to this resource will cause a forced replacement.
╵
╷
│ Error: Dry-run failed for non-structured resource
│
│   with kubernetes_manifest.arc,
│   on main.tf line 52, in resource "kubernetes_manifest" "arc":
│   52: resource "kubernetes_manifest" "arc" {
│
│ A dry-run apply was performed for this resource but was unsuccessful: Apply failed with 7 conflicts: conflicts with "FxVersion" using arcdata.microsoft.com/v1alpha1:
│ - .spec.services
│ conflicts with "OpenAPI-Generator" using arcdata.microsoft.com/v1alpha1:
│ - .spec.docker.imageTag
│ - .spec.settings.azure.subscription
│ - .spec.settings.azure.subscription
│ - .spec.settings.azure.subscription
│ - .spec.storage.data.className
│ - .spec.storage.logs.className](url)

I'm using:

• Kubernetes 1.20.7
• Terraform v0.15.4
• kubernete-alpha provider v0.4.0

Candidly I don't know where to begin with troubleshooting this, I generated the HCL using the yamldecode function per this article https://www.hashicorp.com/blog/deploy-any-resource-with-the-new-kubernetes-provider-for-hashicorp-terraform.

Is this a bug, limitation in the provider or something incorrect I have done that is at play here ?

[jrhouston]

@chrisadkin We're going to have to reproduce this, can you tell us if this CRD is part of the cluster you're using or something you've had to install?

Do you get the same error if you run the YAML manifest through kubectl apply --dry-run?

[alexsomesan]

TL;DR; The next provider release should work as expected with these resources.

It looks like this is a case of non-structural CRDs. I was able to install them using these instructions: https://docs.microsoft.com/en-us/azure/azure-arc/data/create-data-controller-using-kubernetes-native-tools

The provider changes needed to manage these resources are already in master, but not yet released.

I was able to apply the example resource you provided, minus the generation attribute in metadata - it should not be set manually as it's managed by the API server itself.

[chrisadkin-zz]

Many thanks for picking this up, my apologies for not responding sooner, but I have been on vacation, I presume that by "Non-structural" you are referring to the CRD's schema ?, second question are there any timeframes for the release of the provider update ?.

[alexsomesan]

@chrisadkin We just released 0.5.0 which contains improvements relevant to your issue. Can you give it a try and share your results with us?