kubernetes_manifest resource warns OpenAPI schema not recognised and updates will force recreation

[Doug-North]

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.0.5
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.55.0
+ provider registry.terraform.io/hashicorp/helm v2.2.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.4.1
+ provider registry.terraform.io/hashicorp/vault v2.23.0

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_manifest" "kafka_users" {
  for_each = local.kafka_users
  provider = kubernetes.kafka
  manifest = {
    "apiVersion" = "kafka.strimzi.io/v1beta1"
    "kind"       = "KafkaUser"
    "metadata" = {
      "labels" = {
        "app"                          = "kafka-user"
      }
      "name"      = each.key
      "namespace" = "kafka"
    }... 

It all checks out - just not the OpenAPI warning.

Debug Output

│ Warning: This custom resource does not have an associated OpenAPI schema.
│ 
│   with kubernetes_manifest.monese_dev_kafka_users["test-user"],
│   on new-kafka-users.tf line 50, in resource "kubernetes_manifest" "kafka_users":
│   50: resource "kubernetes_manifest" "kafka_users" {
│ 
│ We could not find an OpenAPI schema for this custom resource. Updates to
│ this resource will cause a forced replacement.
│ 
│ (and one more similar warning elsewhere)
╵

Steps to Reproduce

1. terraform apply -->

Expected Behavior

No warning - and for the OpenApi schema defined in CRD to be recognised

Actual Behavior

Warning, and a resource that for me is not usable.

Important Factoids

• Issue was on previous "kubernetes-alpha" provider too.
• CRD created by Strimzi v0.22.0
• CRD uses OpenApi Schema

References

• No valid OpenAPI definition warning terraform-provider-kubernetes-alpha#243

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ls-sajad-sameti]

any news on this issue? we are facing the same problem with IstioOperator CRD.

[warwick-mitchell1]

@ls-sajad-sameti I'm pretty sure this is an issue with Istio not providing the Schema in the CRD correctly.
Unfortunately had no luck in asking the Istio team to add: istio/istio#33535

[ghost]

We experienced the same with Longhorn or it's CRD altough it is installed in our cluster we get this warning.
Anyone working on this issue and can share some status?

[jbg]

@scbay It's not about the CRD being missing from your cluster (if it was, you would not be able to create the object at all). You receive this warning because some CRD authors unfortunately leave the schema unspecified in the CRD definition (so it's basically just a freeform object), which prevents validation of the object you create with kubernetes_manifest and prevents proper diffing on changes (so the object has to be replaced whenever it is changed).

The "proper" solution is to ask your vendor to please fix their CRDs to follow best practices. Unfortunately many vendors can't or won't produce good quality CRDs.

If the warning is accompanied by some other problem (@Doug-North said "Actual behaviour: Warning, and a resource that for me is not usable" but didn't expand on what "not usable" meant), the other problem is likely unrelated to the warning.

I'd love if there was a way to silence the warning. We have some vendor CRDs which have this problem and the persistent warning is annoying. Something like kubectl_manifest's force_new = true so that you could explicitly opt-in to the replace-on-change behaviour which could also silence this warning.

[ecerulm]

I have the same problem with the eniconfigs.crd.k8s.amazonaws.com CRDs, terraform complains about This custom resource does not have an associated OpenAPI schema, but the CRDs seems to have an openAPIV3Schema see source code here.

# Source: aws-vpc-cni/templates/customresourcedefinition.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: eniconfigs.crd.k8s.amazonaws.com
  labels:
    app.kubernetes.io/name: aws-node
    app.kubernetes.io/instance: aws-vpc-cni
    k8s-app: aws-node
    app.kubernetes.io/version: "v1.11.4"
spec:
  scope: Cluster
  group: crd.k8s.amazonaws.com
  preserveUnknownFields: false
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          x-kubernetes-preserve-unknown-fields: true
  names:
    plural: eniconfigs
    singular: eniconfig
    kind: ENIConfig

I'm not familiar with Open API Schema so I don't know what's wrong with this openAPIV3Schema, is terraform right or wrong when it says that there is no OpenAPI schema for this CRD?

[jbg]

There is no schema in that CRD.

[ecerulm]

The CRD has a '.spec.versions[0].schema.openAPIV3Schema', is not that the schema?
I mean the

   schema:
        openAPIV3Schema:
          type: object
          x-kubernetes-preserve-unknown-fields: true

seems like the one of the examples that they give in the Kubernetes Documentation for OpenAPI v3 validation schema.

I though that "openAPIV3Schema" meant that the value must be an object, and nothing else. Do you mean that is not right kind of schema?

Is there any good example of an actual CRD that can be managed with kubernetes_manifest so that I can look to the source code and see what is an actual valid "OpenAPI schema"?

[jbg]

This is not a schema. All it says is that the object is an object. All objects are objects! x-kubernetes-preserve-unknown-fields: true is a hack to avoid having to specify a schema (the fields of the object and their types).

Is there any good example of an actual CRD that can be managed with kubernetes_manifest so that I can look to the source code and see what is an actual valid "OpenAPI schema"?

Do note that you can manage this CRD with kubernetes_manifest. You can also manage the related custom resources with kubernetes_manifest, with the caveat that modifications to them will cause a delete-and-create rather than an in-place update.

Here's an example of the openAPIV3Schema for a CRD that has a schema: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/install/v2_2_4_full.yaml#L37

[alexsomesan]

As jbg explained, the CRD examples quoted here don't include valid schema information. What ecerulm quoted as schema is actually just the placeholder attribute for the schema, but it's contents are empty (x-kubernetes-preserve-unknown-fields: true is a wildcard to mark the absence of schema).

Whenever we encounter this kind of CRDs, we have no other choice but to force re-creation on updates since we cannot guarantee the consistency of the resource schema across updates, which is a hard requirement by Terraform (checked and enforced after each apply).

I'm going to close this issue, but if anyone thinks the error message is not descriptive enough, please speak up.