[fix][broker] Fix deadlock in Key_Shared PIP-379 implementation

[lhotari]

Fixes #23848

Motivation

The Pulsar 4.0 Key_Shared PIP-379 implementation could deadlock and this problem was first captured in test environment heapdump where the thread stack information included in the heapdump showed multiple threads in blocked state, hinting a deadlock.
With the help of @pdolif, the problem was reproduced in isolation by simply adding this type of test code in a KeySharedSubscriptionTest test case.

// test code that triggered the deadlock issue
new Thread(() -> {
    while(true) { pulsar.getBrokerService().updateRates(); }
}).start();

In the test case, this triggered a similar pattern of blocked threads and therefore could be considered a reproduction of the previously faced dead lock issue in a test environment.

threaddump with deadlock:

• in jstack.review https://jstack.review/?https://gist.github.com/lhotari/75eebb84687333bd7b133f2cbf3b6af6#tda_1_dump
• deadlock details: https://gist.github.com/lhotari/75eebb84687333bd7b133f2cbf3b6af6#file-deadlock-txt-L2244

Modifications

• add test case to reproduce the dead lock case which will prevent future regressions in the same area
• fix the dead lock issue by replacing synchronization with read-write locks
  • locks are for minimum "scope"
  • callbacks are called outside of a lock scope to reduce the chances of potential other deadlocks
• use volatile fields for counters in DrainingHashEntry so that entry methods can be used without separate locks

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete