nsxt_policy_tier1_gateway: Changing to locale_service from basic edge_cluster_path removes/readds Tier1 gateway

[ens-scmeeu]

Describe the bug

NOTE: I am filing this as a bug because we are able to do this in the GUI without issue (changes Tier 1, doesn't remove/readd), so there must be an NSX API to do this in a more efficient manner.

When changing an existing Tier 1 gateway to use locale_service with specified preferred edges, the TF provider wants to remove the Tier 1 and then readd it instead of simply changing it in place. If there are any child objects such as firewall policy or service interfaces, the destroy/recreate fails entirely (it might work if they are all part of the same Terraform project, but in our case they are in a different project which references this tier 1 as a data reference).

UPDATE: It appears that static routes alone are not enough to cause the teardown to fail, and when those static routes had been added manually, they are lost entirely after the Tier 1 is recreated.

Reproduction steps

UPDATE: Swapping steps 1 and 2 (doing the inverse) once using locale_service is even worse. It complains it can't find the locale_service object to remove it and the TF apply aborts, so we have no good way to go back to plain edge_cluster_path either.

1. Create tier 1 using terraform using simple edge_cluster_path:

   resource "nsxt_policy_tier1_gateway" "MY_TIER1" {
       display_name = "MY_TIER1"
       failover_mode = "NON_PREEMPTIVE"
       default_rule_logging = true
       enable_firewall = true
       enable_standby_relocation = false
       tier0_path = nsxt_policy_tier0_gateway.MY_TIER0.path
       route_advertisement_types = [
           "TIER1_STATIC_ROUTES",
           "TIER1_CONNECTED",
           "TIER1_LB_VIP",
           "TIER1_LB_SNAT",
           "TIER1_IPSEC_LOCAL_ENDPOINT",
       ]
       edge_cluster_path = data.nsxt_policy_edge_cluster.MY_EDGE_CLUSTER.path
   }

2. Change HCL removing edge_cluster_path and adding locale_service with specific edge nodes:

   resource "nsxt_policy_tier1_gateway" "MY_TIER1" {
       display_name = "MY_TIER1"
       failover_mode = "NON_PREEMPTIVE"
       default_rule_logging = true
       enable_firewall = true
       enable_standby_relocation = false
       tier0_path = nsxt_policy_tier0_gateway.MY_TIER0.path
       route_advertisement_types = [
           "TIER1_STATIC_ROUTES",
           "TIER1_CONNECTED",
           "TIER1_LB_VIP",
           "TIER1_LB_SNAT",
           "TIER1_IPSEC_LOCAL_ENDPOINT",
       ]
   
       locale_service {
           edge_cluster_path = data.nsxt_policy_edge_cluster.MY_EDGE_CLUSTER.path
           preferred_edge_paths = [
               data.nsxt_policy_edge_node.MY_EDGE_NODE1.path,
               data.nsxt_policy_edge_node.MY_EDGE_NODE2.path,
           ]
       }
   }

3. You should see TF plan suggest removing the Tier 1 and readding (NOT changing in place as we would expect)

Expected behavior

I would expect it to change the Tier 1 in place. After creating the Tier 1 as above in step 1, we are able to modify the Tier 1 using the NSX GUI with the same change and it does it in place without impact.

Additional context

This is a blocker for our organization as we have need to migrate to specific edge node allocations without a major impact for our clients.

[annakhm]

Thanks for spotting this.

[ens-scmeeu]

Thanks for the quick patch. Will this patch also fix the Tier 1 recreate when locale_service is removed? (the inverse situation)