[fix] [broker] topics failed to delete after remove cluster from replicated clusters set and caused OOM

[poorbarcode]

Motivation

Background

• Users may start 2 clusters with the Geo-Replication feature with Global ZK
• Users want to hire one cluster, so remove the cluster from namespace-level replicatedClusters
  • Broker will remove the topics of the cluster removed automatically.
• Issue occurs:
  • The system topic __change_events being deleted first.
  • User topics can not be deleted anymore, because the operation delete topic-level policies can not be executed successfully anymore.
  • The task topic.checkReplication and other operations will retry again and again, as a result, the broker will crash of OOM

---

---

Modifications

• Skip removing topic-level policies if the system topic __change_event has been deleted.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

Please check the review comments.

[lhotari]

LGTM, good work @poorbarcode

[lhotari]

Triggering CI with fix #23431

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.36%. Comparing base (bbc6224) to head (5c2e2b4).
Report is 663 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23360      +/-   ##
============================================
+ Coverage     73.57%   74.36%   +0.78%     
- Complexity    32624    34983    +2359     
============================================
  Files          1877     1953      +76     
  Lines        139502   147162    +7660     
  Branches      15299    16199     +900     
============================================
+ Hits         102638   109430    +6792     
- Misses        28908    29305     +397     
- Partials       7956     8427     +471     

Flag	Coverage Δ
inttests	27.31% <62.50%> (+2.73%)	⬆️
systests	24.32% <62.50%> (-0.01%)	⬇️
unittests	73.72% <100.00%> (+0.88%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../service/SystemTopicBasedTopicPoliciesService.java	74.42% <100.00%> (+0.23%)	⬆️
...er/systopic/NamespaceEventsSystemTopicFactory.java	100.00% <100.00%> (ø)

... and 633 files with indirect coverage changes

[lhotari]

I wonder if this is causing #23474 which currently blocks Pulsar CI. There's PR #23478 to disable OneWayReplicatorTestBase tests the problem is addressed. @poorbarcode do you have a chance to fix the problem?

[poorbarcode]

I wonder if this is causing #23474 which currently blocks Pulsar CI. There's PR #23478 to disable OneWayReplicatorTestBase tests the problem is addressed. @poorbarcode do you have a chance to fix the problem?

Self-assigned #23474, I will fix it next week

[lhotari]

@poorbarcode Please also handle cherry-picking to branch-3.0 and branch-3.3

[heesung-sn]

It appears that this PR fails the test in branch-3.3.

https://github.com/heesung-sn/pulsar/actions/runs/11535492430/job/32111441443

  Error:  Tests run: 8, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 782.877 s <<< FAILURE! - in org.apache.pulsar.broker.service.OneWayReplicatorUsingGlobalZKTest
  Error:  org.apache.pulsar.broker.service.OneWayReplicatorUsingGlobalZKTest.testRemoveCluster  Time elapsed: 300.528 s  <<< FAILURE!
  org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.broker.service.OneWayReplicatorUsingGlobalZKTest.testRemoveCluster() didn't finish within the time-out 300000
  	at java.base/java.lang.Thread.sleep0(Native Method)
  	at java.base/java.lang.Thread.sleep(Thread.java:509)
  	at org.apache.pulsar.broker.service.OneWayReplicatorUsingGlobalZKTest.testRemoveCluster(OneWayReplicatorUsingGlobalZKTest.java:211)
  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)

[poorbarcode]

@heesung-sn

It appears that this PR fails the test in branch-3.3.

Sure, it will be fixed by the PR: #23522

[lhotari]

@poorbarcode Do you have a chance to take a look at #23543 ? It's a deadlock in SystemTopicBasedTopicPoliciesService which happens also after #23522 has been merged. The deadlock might be completely unrelated to the recent changes, but since you know SystemTopicBasedTopicPoliciesService well, I'd appreciate it if you can take a look at the thread dump and the deadlocks.