Add default TLS client cert and key paths for Prometheus input and receiver

[musa-asad]

Description of the issue

The Prometheus input and receiver within the CloudWatch Agent do not have default TLS client certificate and key file paths set. Without these default paths, the agent does not know which certificate to use for mutual TLS authentication with the Target Allocator server. This prevents the agent from properly connecting to the Target Allocator, as the agent cannot locate the required TLS certificates.

Description of changes

Important

Co-PRs: 1) aws-observability/helm-charts#163; 2) aws/amazon-cloudwatch-agent-operator#284

• Added client certificate and key variables with default file paths to Prometheus input and receiver.
• Assigned client certificate and key variables variables to Target Allocator TLS settings.
• Updated unit test to test for new variables.
• Fixed linting issues.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

1. Created EKS cluster using custom agent, operator, and target allocator images (Implement mTLS resources and configuration for Target Allocator server amazon-cloudwatch-agent-operator#284).
2. Ran helm upgrade --install --debug amazon-cloudwatch-observability helm-charts/charts/amazon-cloudwatch-observability --set clusterName=<cluster_name> --set region=us-west-2 --namespace amazon-cloudwatch --create-namespace with custom helm charts (Implement mTLS resources for CloudWatch Agent client aws-observability/helm-charts#163) and edited values.yaml with a custom agent and prometheus configuration.
3. Set up sample application: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Sample-Workloads-memcached.html.

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

[lisguo]

I know that we already have amazon-cloudwatch-observability-agent-cert used in a couple of places. Is the outbound-cert new? Is it only for TA?

[musa-asad]

The outbound-cert is only for when the agent behaves as a client in mTLS. Right now, it's only being used for when the agent acts as a client to the TA server, but it could be used against other servers too.

• amazon-cloudwatch-observability-agent-client-cert -> client cert and key for fluent-bit
• amazon-cloudwatch-observability-agent-server-cert -> server cert and key for agent (technically can be used in this place, but avoided due to separation of concerns)
• amazon-cloudwatch-observability-agent-cert -> server cert and key for dcgm, neuron, and ta
• amazon-cloudwatch-observability-agent-outbound-cert -> client cert and key for agent