[FLINK-35196] [Connector / Pulsar] Fix bouncycastle class not found

[wenbingshen]

Purpose of the change

1. The bouncycastle dependency was removed in this PR [FLINK-30430][Connector/Pulsar] Remove all the useless dependencies, drop mock runtime. #9 , which was correct at the time, and there was no table api at that time

2. When we introduced table api, dependencies started to show errors because bouncycastle was referenced in table api

2024-04-22 14:45:48
java.lang.NoClassDefFoundError: org/apache/pulsar/shade/org/bouncycastle/util/Arrays
	at org.apache.flink.connector.pulsar.table.sink.PulsarWritableMetadata.readMetadata(PulsarWritableMetadata.java:67)
	at org.apache.flink.connector.pulsar.table.sink.PulsarWritableMetadata.applyWritableMetadataInMessage(PulsarWritableMetadata.java:55)
	at org.apache.flink.connector.pulsar.table.sink.PulsarTableSerializationSchema.serialize(PulsarTableSerializationSchema.java:106)
	at org.apache.flink.connector.pulsar.table.sink.PulsarTableSerializationSchema.serialize(PulsarTableSerializationSchema.java:40)
	at org.apache.flink.connector.pulsar.sink.writer.PulsarWriter.write(PulsarWriter.java:143)
	at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:160)
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:542)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
	at java.lang.Thread.run(Thread.java:748)

[boring-cyborg]

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

[wenbingshen]

@syhily @snuyanzin @tsreaper PTAL. Thanks.

[syhily]

Thanks for your commit. I think jdk15on should be dropped by in favor of the jdk18on.

[wenbingshen]

Thanks for your commit. I think jdk15on should be dropped by in favor of the jdk18on.

@syhily Thanks review. I have upgraded jdk15on to jdk18on. PTAL.

[syhily]

The pulsar-client-all should be upgrade to the 3.0.4 in the mean time.

[wenbingshen]

The pulsar-client-all should be upgrade to the 3.0.4 in the mean time.

@syhily sure. I have addressed.

[wenbingshen]

The pulsar-client-all should be upgrade to the 3.0.4 in the mean time.

@syhily After pulsar-client-all was updated to 3.0.4, the unit test failed. I took a brief look and found that publishTimeStopCursor seemed to be connected to the consumer and then seek position, so consumer will be disconnect.Then after reconnecting the consumer, an exception of Exclusive consumer is already connected occurred. I think this requires more detailed investigation shows that this may be due to some changes in the internal implementation of pulsar-client-all due to version upgrades, and troubleshooting is not that easy.

So I think it is worth using a separate PR to upgrade pulsar-client-all to version 3.0.4 and solve the problems caused by the related upgrade.

This PR only solves the problem that the table api cannot find the bouncycastle class.

Related error logs:
https://github.com/apache/flink-connector-pulsar/actions/runs/8832359197/job/24249517876?pr=91

PTAL. Thanks.

[syhily]

The pulsar-client-all should be upgrade to the 3.0.4 in the mean time.

@syhily After pulsar-client-all was updated to 3.0.4, the unit test failed. I took a brief look and found that publishTimeStopCursor seemed to be connected to the consumer and then seek position, so consumer will be disconnect.Then after reconnecting the consumer, an exception of Exclusive consumer is already connected occurred. I think this requires more detailed investigation shows that this may be due to some changes in the internal implementation of pulsar-client-all due to version upgrades, and troubleshooting is not that easy.

So I think it is worth using a separate PR to upgrade pulsar-client-all to version 3.0.4 and solve the problems caused by the related upgrade.

This PR only solves the problem that the table api cannot find the bouncycastle class.

Related error logs: apache/flink-connector-pulsar/actions/runs/8832359197/job/24249517876?pr=91

PTAL. Thanks.

Thanks for your works. I think we can divide the work into two steps. Bump pulsar-client-all do require a lot of works. But since the pulsar-client-all depends on the legacy 15on. I think we can exclude these dependencies first in this PR and use 18on.

[wenbingshen]

@syhily ignore this problem. Thanks.

[syhily]

OK, feel free to reopen this PR if you need.