[Bug] Multiple Hoodie writer commit conflict

[zymap]

Describe the bug
When we run the hudi sink with multiple instances, then using it sink a partitioned topic with failover subscription mode. Both instances will consume the message from the topic. Hudi supports the concurrency mode to support multiple writers.
When we enable this feature, the hudi writer will throw fileAlreadyExists exception and failed the commit.

org.apache.hudi.exception.HoodieIOException: Failed to create file file:/tmp/integration/hudi/.hoodie/20220517094051766.commit
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:745) ~[hudi-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:560) ~[hudi-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:536) ~[hudi-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.saveAsComplete(HoodieActiveTimeline.java:183) ~[hudi-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:270) ~[hudi-client-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:234) ~[hudi-client-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.client.HoodieJavaWriteClient.commit(HoodieJavaWriteClient.java:88) ~[hudi-java-client-0.11.0.jar:0.11.0]
	at org.apache.hudi.client.HoodieJavaWriteClient.commit(HoodieJavaWriteClient.java:51) ~[hudi-java-client-0.11.0.jar:0.11.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:206) ~[hudi-client-common-0.11.0.jar:0.11.0]
	at org.apache.pulsar.ecosystem.io.sink.hudi.BufferedConnectWriter.flushRecords(BufferedConnectWriter.java:82) ~[PqY5lYEJSWPWMDq7E5HC2Q/:?]
	at org.apache.pulsar.ecosystem.io.sink.hudi.HoodieWriter.flush(HoodieWriter.java:85) ~[PqY5lYEJSWPWMDq7E5HC2Q/:?]
	at org.apache.pulsar.ecosystem.io.sink.SinkWriter.commitIfNeed(SinkWriter.java:128) ~[PqY5lYEJSWPWMDq7E5HC2Q/:?]
	at org.apache.pulsar.ecosystem.io.sink.SinkWriter.run(SinkWriter.java:113) [PqY5lYEJSWPWMDq7E5HC2Q/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.77.Final.jar:4.1.77.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/tmp/integration/hudi/.hoodie/20220517094051766.commit
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:315) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:353) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:403) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:466) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:445) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1105) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:994) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$create$2(HoodieWrapperFileSystem.java:222) ~[hudi-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:101) ~[hudi-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.common.fs.HoodieWrapperFileSystem.create(HoodieWrapperFileSystem.java:221) ~[hudi-common-0.11.0.jar:0.11.0]
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:740) ~[hudi-common-0.11.0.jar:0.11.0]
	... 16 more

To Reproduce
Steps to reproduce the behavior:

1. Create a partitioned topic pio
2. submit a hudi writer with following configuration:

    {
  "tenant": "public",
  "namespace": "default",
  "name": "lakehouse1",
  "topicName": "pio",
  "parallelism": 2,
  "sourceSubscriptionName": "sub",
  "processingGuarantees": "EFFECTIVELY_ONCE",
  "subscriptionType": "Failover",
  "type": "hudi",
  "inputs": [
    "pio"
  ],
  "archive": "/Volumes/work/github.com/streamnative/pulsar-io-lakehouse/target/pulsar-io-lakehouse-2.9.2.0-SNAPSHOT.nar",
  "className": "org.apache.pulsar.ecosystem.io.SinkConnector",
  "configs":
  {
    "type": "hudi",
    "hoodie.table.name": "hudi-connector-test",
    "hoodie.table.type": "COPY_ON_WRITE",
    "hoodie.base.path": "file:///tmp/integration/hudi",
    "hoodie.clean.async": "true",
    "hoodie.write.concurrency.mode": "optimistic_concurrency_control",
    "hoodie.cleaner.policy.failed.writes": "LAZY",
    "hoodie.write.lock.provider": "org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider",
    "hoodie.write.lock.zookeeper.url": "localhost",
    "hoodie.write.lock.zookeeper.port": "2181",
    "hoodie.write.lock.zookeeper.lock_key": "pulsar_hudi",
    "hoodie.write.lock.zookeeper.base_path": "/hudi",
    "hoodie.datasource.write.recordkey.field": "id",
    "hoodie.datasource.write.partitionpath.field": "id",
    "maxRecordsPerCommit": 10
  }
}

3. produce message to the pio
4. See error

Expected behavior
Both the commit should success

Screenshots
If applicable, add screenshots to help explain your problem.

Environment

• OS: [e.g. Ubuntu]
• Pulsar version: [e.g. 2.7.0]
• Deployment: [e.g. standalone]
• Connector/offloader/protocol handler/... version: [e.g. 2.7.0]

Additional context
Add any other context about the problem here.