users.xml strange nifikop behavior of file filling

[TimMilton]

What steps will reproduce the bug?

After updating the CRD and NiFiKop to the latest version 1.11.4, I noticed strange behavior with the users.xml file. We have a cluster running with a node recycling scheduler on AWS, as well on spot instances.
And after another nodes restart all nifi nodes lost their groups and users, including the NiFi nodes themselves, even though they are specified in the nificluster and present in nifiusers CRD. The file was populated only on the primary node and only with its own identity, but without group.

Primary nifi node users.xml was:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
    <groups/>
    <users>
        <user identifier="123" identity="nifi-controller"/>
        <user identifier="123" identity="nifi-0-node.nifi-headless.nifi.svc.cluster.local"/>
    </users>
</tenants>

Should be something like:

<tenants>
    <groups>
        <group identifier="ggggg-gggg1" name="nifi-nifi.managed-readers"/>
        <group identifier="ggggg-gggg2" name="nifi-nifi.managed-nodes">
            <user identifier="nifi-1-node-id-uuuu-123"/>
            <user identifier="nifi-0-node-id-uuuu-123"/>
        </group>
        <group identifier="ggggg-gggg3" name="nifi-nifi.managed-admins">
            <user identifier="nifi-controller-id-uuuuu-123"/>
            <user identifier="user-id-1-uuuuu-123"/>
        </group>
    </groups>
    <users>
        <user identifier="nifi-0-node-id-uuuu-123" identity="nifi-0-node.nifi-headless.nifi.svc.cluster.local"/>
        <user identifier="nifi-controller-id-uuuuu-123" identity="nifi-controller"/>
        <user identifier="nifi-1-node-id-uuuu-123" identity="nifi-1-node.nifi-headless.nifi.svc.cluster.local"/>
        <user identifier="user-id-1-uuuuu-123" identity="user1"/>
    </users>
</tenants>

So far, I`ve able to reproduce this issue, if it can be considered a bug, by simply deleting the users.xml file from nodes and restarting them.

Logs:

{"level":"error","time":"2025-01-03T13:55:43.444Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-controller","namespace":"nifi"},"namespace":"nifi","name":"nifi-controller","reconcileID":"123","error":"could not connect to nifi nodes: nifi-headless.nifi.svc.cluster.local:8443: 404 response from NiFi cluster","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","time":"2025-01-03T13:55:44.647Z","logger":"nifi_client","caller":"nificlient/common.go:20","msg":"404 response from nifi node: ","statusCode":"404 Not Found","stacktrace":"github.com/konpyutaika/nifikop/pkg/nificlient.errorGetOperation\n\t/workspace/pkg/nificlient/common.go:20\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).DescribeCluster\n\t/workspace/pkg/nificlient/system.go:17\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).Build\n\t/workspace/pkg/nificlient/client.go:182\ngithub.com/konpyutaika/nifikop/pkg/nificlient.NewFromConfig\n\t/workspace/pkg/nificlient/client.go:203\ngithub.com/konpyutaika/nifikop/pkg/common.NewClusterConnection\n\t/workspace/pkg/common/common.go:56\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/usergroup.SyncUserGroup\n\t/workspace/pkg/clientwrappers/usergroup/usergroup.go:62\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiUserGroupReconciler).Reconcile\n\t/workspace/internal/controller/nifiusergroup_controller.go:292\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2025-01-03T13:55:44.647Z","logger":"controller.NifiUserGroup","caller":"controller/controller_common.go:35","msg":"failed to sync NifiUserGroup nifi.managed-admins"}
{"level":"error","time":"2025-01-03T13:55:44.647Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiusergroup","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUserGroup","nifiUserGroup":{"name":"nifi.managed-admins","namespace":"nifi"},"namespace":"nifi","name":"nifi.managed-admins","reconcileID":"123","error":"could not connect to nifi nodes: nifi-headless.nifi.svc.cluster.local:8443: 404 response from NiFi cluster","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","time":"2025-01-03T13:55:45.632Z","logger":"nifi_client","caller":"nificlient/common.go:20","msg":"404 response from nifi node: ","statusCode":"404 Not Found","stacktrace":"github.com/konpyutaika/nifikop/pkg/nificlient.errorGetOperation\n\t/workspace/pkg/nificlient/common.go:20\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).DescribeCluster\n\t/workspace/pkg/nificlient/system.go:17\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).Build\n\t/workspace/pkg/nificlient/client.go:182\ngithub.com/konpyutaika/nifikop/pkg/nificlient.NewFromConfig\n\t/workspace/pkg/nificlient/client.go:203\ngithub.com/konpyutaika/nifikop/pkg/common.NewClusterConnection\n\t/workspace/pkg/common/common.go:56\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/usergroup.SyncUserGroup\n\t/workspace/pkg/clientwrappers/usergroup/usergroup.go:62\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiUserGroupReconciler).Reconcile\n\t/workspace/internal/controller/nifiusergroup_controller.go:292\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2025-01-03T13:55:45.632Z","logger":"controller.NifiUserGroup","caller":"controller/controller_common.go:35","msg":"failed to sync NifiUserGroup nifi.managed-admins"}
{"level":"error","time":"2025-01-03T13:55:45.632Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiusergroup","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUserGroup","nifiUserGroup":{"name":"nifi.managed-admins","namespace":"nifi"},"namespace":"nifi","name":"nifi.managed-admins","reconcileID":"123","error":"could not connect to nifi nodes: nifi-headless.nifi.svc.cluster.local:8443: 404 response from NiFi cluster","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

Manual deleting:

{"level":"error","time":"2025-01-02T16:25:56.202Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi.user1","namespace":"nifi"},"namespace":"nifi","name":"nifi.user1","reconcileID":"123","error":"could not connect to nifi nodes: nifi-headless.nifi.svc.cluster.local:8443: 404 response from NiFi cluster","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","time":"2025-01-02T16:25:56.294Z","logger":"nifi_client","caller":"nificlient/common.go:20","msg":"404 response from nifi node: ","statusCode":"404 Not Found","stacktrace":"github.com/konpyutaika/nifikop/pkg/nificlient.errorGetOperation\n\t/workspace/pkg/nificlient/common.go:20\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).DescribeCluster\n\t/workspace/pkg/nificlient/system.go:17\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).Build\n\t/workspace/pkg/nificlient/client.go:182\ngithub.com/konpyutaika/nifikop/pkg/nificlient.NewFromConfig\n\t/workspace/pkg/nificlient/client.go:203\ngithub.com/konpyutaika/nifikop/pkg/common.NewClusterConnection\n\t/workspace/pkg/common/common.go:56\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/user.ExistUser\n\t/workspace/pkg/clientwrappers/user/user.go:22\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiUserReconciler).Reconcile\n\t/workspace/internal/controller/nifiuser_controller.go:299\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2025-01-02T16:25:56.294Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failure checking for existing user nifi.user2"}
{"level":"error","time":"2025-01-02T16:25:56.294Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi.user2","namespace":"nifi"},"namespace":"nifi","name":"nifi.user2","reconcileID":"123","error":"could not connect to nifi nodes: nifi-headless.nifi.svc.cluster.local:8443: 404 response from NiFi cluster","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","time":"2025-01-02T16:25:56.324Z","logger":"nifi_client","caller":"nificlient/common.go:26","msg":"Non 200 response from nifi node","statusCode":"403 Forbidden","body":"Unable to view the controller. Contact the system administrator.","stacktrace":"github.com/konpyutaika/nifikop/pkg/nificlient.errorGetOperation\n\t/workspace/pkg/nificlient/common.go:26\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).DescribeCluster\n\t/workspace/pkg/nificlient/system.go:17\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).Build\n\t/workspace/pkg/nificlient/client.go:182\ngithub.com/konpyutaika/nifikop/pkg/nificlient.NewFromConfig\n\t/workspace/pkg/nificlient/client.go:203\ngithub.com/konpyutaika/nifikop/pkg/common.NewClusterConnection\n\t/workspace/pkg/common/common.go:56\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/user.ExistUser\n\t/workspace/pkg/clientwrappers/user/user.go:22\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiUserReconciler).Reconcile\n\t/workspace/internal/controller/nifiuser_controller.go:299\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2025-01-02T16:25:56.324Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failure checking for existing user nifi.user3"}
{"level":"error","time":"2025-01-02T16:25:56.324Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi.user3","namespace":"nifi"},"namespace":"nifi","name":"nifi.user3","reconcileID":"123","error":"could not connect to nifi nodes: nifi-headless.nifi.svc.cluster.local:8443: non 200 response from NiFi cluster","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","time":"2025-01-02T16:25:56.561Z","logger":"nifi_client","caller":"nificlient/common.go:20","msg":"404 response from nifi node: ","statusCode":"404 Not Found","stacktrace":"github.com/konpyutaika/nifikop/pkg/nificlient.errorGetOperation\n\t/workspace/pkg/nificlient/common.go:20\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).DescribeCluster\n\t/workspace/pkg/nificlient/system.go:17\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).Build\n\t/workspace/pkg/nificlient/client.go:182\ngithub.com/konpyutaika/nifikop/pkg/nificlient.NewFromConfig\n\t/workspace/pkg/nificlient/client.go:203\ngithub.com/konpyutaika/nifikop/pkg/common.NewClusterConnection\n\t/workspace/pkg/common/common.go:56\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/user.ExistUser\n\t/workspace/pkg/clientwrappers/user/user.go:22\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiUserReconciler).Reconcile\n\t/workspace/internal/controller/nifiuser_controller.go:299\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2025-01-02T16:25:56.562Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failure checking for existing user nifi-0-node.nifi-headless.nifi.svc.cluster.local"}
{"level":"error","time":"2025-01-02T16:25:56.562Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-0-node.nifi-headless.nifi.svc.cluster.local","namespace":"nifi"},"namespace":"nifi","name":"nifi-0-node.nifi-headless.nifi.svc.cluster.local","reconcileID":"123","error":"could not connect to nifi nodes: nifi-headless.nifi.svc.cluster.local:8443: 404 response from NiFi cluster","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","time":"2025-01-02T16:25:56.691Z","logger":"nifi_client","caller":"nificlient/common.go:20","msg":"404 response from nifi node: ","statusCode":"404 Not Found","stacktrace":"github.com/konpyutaika/nifikop/pkg/nificlient.errorGetOperation\n\t/workspace/pkg/nificlient/common.go:20\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).DescribeCluster\n\t/workspace/pkg/nificlient/system.go:17\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).Build\n\t/workspace/pkg/nificlient/client.go:182\ngithub.com/konpyutaika/nifikop/pkg/nificlient.NewFromConfig\n\t/workspace/pkg/nificlient/client.go:203\ngithub.com/konpyutaika/nifikop/pkg/common.NewClusterConnection\n\t/workspace/pkg/common/common.go:56\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/user.ExistUser\n\t/workspace/pkg/clientwrappers/user/user.go:22\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiUserReconciler).Reconcile\n\t/workspace/internal/controller/nifiuser_controller.go:299\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2025-01-02T16:25:56.691Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failure checking for existing user nifi-1-node.nifi-headless.nifi.svc.cluster.local"}
{"level":"error","time":"2025-01-02T16:25:56.691Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-1-node.nifi-headless.nifi.svc.cluster.local","namespace":"nifi"},"namespace":"nifi","name":"nifi-1-node.nifi-headless.nifi.svc.cluster.local","reconcileID":"123","error":"could not connect to nifi nodes: nifi-headless.nifi.svc.cluster.local:8443: 404 response from NiFi cluster","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/

What is the expected behavior?

Correct update of the users.xml file

What do you see instead?

Empty users.xml file, except on primary node

Possible solution

No response

NiFiKop version

v1.11.4-release

Golang version

Kubernetes version

Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.7-eks-56e63d8

NiFi version

1.25.0, 1.28.0

Additional context

No response