[improve][pip] PIP-366: Support to specify different config for Confi…

…guration and Local Metadata Store (apache#23033)

pip/pip-366.md

@@ -0,0 +1,70 @@
+# PIP-366: Support to specify different config for Configuration and Local Metadata Store
+
+# Background knowledge
+
+Pulsar metadata store maintains all the metadata, configuration, and coordination of a Pulsar cluster, such as topic metadata, schema, broker load data, and so on.
+
+The metadata store of each Pulsar instance should contain the following two components:
+
+- A local metadata store ensemble (`metadataStoreUrl`) that stores cluster-specific configuration and coordination, such as which brokers are responsible for which topics as well as ownership metadata, broker load reports, and BookKeeper ledger metadata.
+- A configuration store quorum (`configurationMetadataStoreUrl`) stores configuration for clusters, tenants, namespaces, topics, and other entities that need to be globally consistent.
+
+# Motivation
+
+When using Geo-Replication and global configuration store for configuration global consistency, the configuration store's config may be different from the local metadata store's config. For example, the configuration store may have a different set of ZooKeeper servers than the local metadata store.
+
+The global configuration store may deploy in a different data center, and the local metadata store may be deployed in the same data center as the Pulsar broker. In this case, the global configuration store may need to use TLS and authentication to protect the connection to metadata store server, while the local metadata store may not need to use TLS and authentication.
+
+However, the current implementation of Pulsar only supports configuring different metadata store url for the local metadata store and the configuration store. This limitation makes it impossible to support the above scenario.
+
+# Goals
+
+## In Scope
+
+- Support specifying different configurations for the local metadata store and the configuration store.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+Pulsar support `metadataStoreConfigPath` configuration, but it only supports for `RocksdbMetadataStore`, and it is not able to specify different configuration for Configuration Metadata Store.
+
+```java
+    @FieldContext(
+            category = CATEGORY_SERVER,
+            doc = "Configuration file path for local metadata store. It's supported by RocksdbMetadataStore for now."
+    )
+    private String metadataStoreConfigPath = null;
+```
+
+Therefore, we need to add a new configuration `configurationStoreConfigPath` for `ConfigurationMetadataStore`, and the `metadataStoreConfigPath` will be still use for `LocalMetadataStore`.
+
+```java
+    @FieldContext(
+            category = CATEGORY_SERVER,
+            doc = "Configuration file path for configuration metadata store."
+    )
+    private String configurationStoreConfigPath = null;
+```
+
+When the `configurationStoreConfigPath` are not set, the `metadataStoreConfigPath` will be used as the configuration file path for the configuration store.
+
+For each metadata store implementation, we need pass the corresponding configuration file path to the metadata store. For example, for ZKMetadataStore, we can specify config when create the Zookeeper client.
+
+```java
+    protected ZooKeeper createZooKeeper() throws IOException {
+        return new ZooKeeper(connectString, sessionTimeoutMs, watcherManager, allowReadOnlyMode, /** Add the config here **/ new ZKClientConfig(configPath));
+    }
+```
+
+# Backward & Forward Compatibility
+
+Fully compatible.
+
+# Links
+
+<!--
+Updated afterwards
+-->
+* Mailing List discussion thread: https://lists.apache.org/thread/98ggo1zg1k7dbyx8wr9bc8onm10p16c6
+* Mailing List voting thread: https://lists.apache.org/thread/wm30dy9bkhxxmmcb0v9ftb56ckpknrfr