The karmada propagates resource very slowly when karmada-controller-manager is just restarted

[wenhuwang]

Please provide an in-depth description of the question you have:
The karmada propagates new networkpolicy resource to member cluster very slowly when karmada-controller-manager is just restarted, it lasts about 20 minutes.

workqueue depth metrics is as follow:

the list of resources managed by karmada is as follow:

• cluster: 7
• namespace: 1100
• workspace: 105
• ciliumnetworkpolicy: 2500
• networkpolicy: 3800

What do you think about this question?:
Through the log, it can be roughly located that during the controller startup process, when the cache in the Informer has not been synchronized, the old resource events are treated as Create event types, resulting in the controller receiving a large number of Create events, and i am working on this problem.

Environment:

• Karmada version: v1.7.2
• Kubernetes version: v1.22.14

[wenhuwang]

/assign

[liangyuanpeng]

Can this be replicated in a higher version of karmada. version 1.7 may too old.

[wenhuwang]

This problem arises because the controller-runtime library is designed this way by default. the informer startup and cache synchronization process are asynchronous

And i will also try to reproduce this issue with the latest version

[wenhuwang]

I reproduced it locally using version v1.12.2, and add predicateFunc to the resourcebinding controller to determine the event type.
The number of resources managed by karmada is 175

# kubectl get networkpolicies.networking.k8s.io | wc -l
176

When the karmada-controller-manager restarted the resourcebinding controller reconcile times is 175, and the event type received is all of Create

# kubectl -n karmada-system-demo logs  karmada-controller-manager-bfcb477d6-dgm79  | grep " Sync work of resourceBinding(" | wc -l
175
# kubectl -n karmada-system-demo logs  karmada-controller-manager-bfcb477d6-dgm79  | grep "Create event for ResourceBinding " | wc -l
175

[wenhuwang]

I compared the performance of karmada versions v1.7.2 and v1.12.2. The test data is as follows：

• cluster : 2
• workspace: 7
• namespace: 10
• networkpolicy: 6000

karmada v1.7.2
When the kamrada controller is just restarted, a new network policy is synchronized to the member cluster in about 8 minutes.
workqueue depth metrics

karmada v1.12.2
When the kamrada controller is just restarted, a new network policy is synchronized to the member cluster in about 4 minutes.
workqueue depth metrics

From the test results, the performance of the new version is about 50% higher than that of v1.7.2.

[RainbowMango]

Thanks for sharing the report with us.

Just sharing some info here, we are building a team dedicated to addressing performance issues, and #6031 tracks the efforts planned in release-1.13. Welcome to join us!