[improve][fn]: Added logLevel to change the log level of Function/Source/Sink Pod/Process to desired Level

[mukesh154]

Master Issue: #23783

Motivation:

The pulsar.log.level property inkubernetes_instance_log4j2.xml & java_instance_log4j2.xml files in the Pulsar Functions runtime is responsible for configuring the log settings for function pods when using the Java runtime in Kubernetes StatefulSets & for process runtime respectively. Currently, this configuration file is hardcoded to use the INFO log level, which restricts flexibility when more granular logging (such as DEBUG level) is needed for troubleshooting or deeper monitoring. This limitation is particularly impactful when debugging issues related to sources, sinks, or function executions, where more detailed logs could be beneficial.

Presently, adjusting the log level requires manual changes to the kubernetes_instance_log4j2.xml or java_instance_log4j2.xml file, which is inefficient and error-prone, especially in dynamic environments. There is no built-in way to modify the log level dynamically without altering the configuration file directly.

This PR introduces a new argument (logLevel) to override the JVM property -Dpulsar.log.level=info at runtime for functions, sources, and sinks. This change provides a more flexible and dynamic approach to adjust log verbosity based on specific needs, such as enabling DEBUG or ERROR logs for better diagnostics during troubleshooting.

Modifications:

• Introduction of logLevel Argument: Added a new logLevel argument for Pulsar Functions, Sources, and Sinks, which allows the user to override the default JVM property -Dpulsar.log.level=info. This enables setting custom log levels (e.g., DEBUG, ERROR) dynamically at runtime without modifying the kubernetes_instance_log4j2.xml or java_instance_log4j2.xml file.

• JVM Property Override: The new logLevel argument directly overrides the -Dpulsar.log.level property, giving users control over the logging level without needing to edit the underlying configuration file.

• Backward Compatibility: The default log level remains INFO unless the new logLevel argument is provided. This ensures that existing deployments continue to function as before.

This update enhances the troubleshooting and monitoring capabilities of Pulsar Functions, offering greater flexibility while preserving the simplicity and stability of existing configurations.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): no
• The public API: no
• The schema: no
• The default values of configurations: no
• The wire protocol: no
• The rest endpoints: no
• The admin cli options: yes. Added a cli arg logLevel for creating/updating function/source/sink
• Anything that affects deployment: no

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

• doc-required

  (If you need help on updating docs, create a doc issue)

• no-need-doc

  (Please explain why)

• doc

  (If this PR contains doc changes)

[lhotari]

there are more valid levels in Log4J 2: https://logging.apache.org/log4j/2.x/manual/customloglevels.html . I'm not sure if others are relevant, but perhaps it would make sense to allow any valid Log4J log level?

[mukesh154]

Added all valid levels

[lhotari]

@mukesh154 Thanks for the contribution, this looks very useful.

[lhotari]

One minor typo in review comments

[lhotari]

Suggested change

	@Option(names = "--log-level", description = "Log level at which the logs of a Pulsar Function are produced"
	+ " #Java")
	@Option(names = "--log-level", description = "Log level at which the logs of a Pulsar Function are produced")

[mukesh154]

The current changes is limited to Java runtime only.