[SYNPY-1304] Introduction of OpenTelemetry (#1007)

* Added to integration tests, several code paths, and CLI as an optional parameter

CONTRIBUTING.md

@@ -15,8 +15,12 @@ By contributing, you are agreeing that we may redistribute your work under this
 - [The development life cycle](#the-development-life-cycle)
    * [Development](#development)
    * [Testing](#testing)
-      + [Integration testing agaisnt the `dev` synapse server](#integration-testing-agaisnt-the-dev-synapse-server)
+      + [Integration testing against the dev synapse server](#integration-testing-against-the-dev-synapse-server)
+      + [Running OpenTelemetry in Integration Tests](#running-opentelemetry-in-integration-tests)
    * [Code style](#code-style)
+   * [OpenTelemetry](#opentelemetry)
+      + [Attributes within traces](#attributes-within-traces)
+      + [Adding new spans](#adding-new-spans)
    * [Repository Admins](#repository-admins)
 
 ## I don't want to read this whole thing I just have a question!
@@ -180,8 +184,11 @@ authEndpoint=https://repo-dev.dev.sagebase.org/auth/v1
 fileHandleEndpoint=https://repo-dev.dev.sagebase.org/file/v1
 ```
 
+#### Running OpenTelemetry in Integration Tests
+`tests/integration/conftest.py` is where we defining which trace provider to use. Set the `SYNAPSE_OTEL_INTEGRATION_TEST_PROVIDER` environment variable to `otlp` or `console` depending on your use case.
+
 #### Integration testing for external collaborators
-As an external collaborator you will not have access to a development account and environment to run the integration tests agaisnt. Either request that a Sage Bionetworks staff member run your integration tests via a pull request, or, contact us via the [Service Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9) to requisition a development account for integration testing only.
+As an external collaborator you will not have access to a development account and environment to run the integration tests against. Either request that a Sage Bionetworks staff member run your integration tests via a pull request, or, contact us via the [Service Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9) to requisition a development account for integration testing only.
 
 ### Code style
 
@@ -197,6 +204,23 @@ pip install flake8
 flake8
 ```
 
+### OpenTelemetry
+[OpenTelemetry](https://opentelemetry.io/) helps support the analysis of traces and spans which can provide insights into latency, errors, and other performance metrics. During development it may prove useful to collect information about the execution of your code. The following is meant to be a starting point for additions to the current traces being collected.
+
+To learn more about how to modify trace collection read the documentation [here](https://opentelemetry.io/docs/instrumentation/python/manual/)
+
+#### Attributes within traces
+Attributes that are collected within traces should not contain any sensitive data. We should follow the [Common specification concepts](https://opentelemetry.io/docs/specs/otel/common/) defined by OpenTelemetry when it comes to naming attribute keys.
+
+All synapse related attributes should live within the `synapse` namespace, for example: `synapse.id`, `synapse.parent_id`.
+
+#### Adding new spans
+1. All new integration tests should create a new span for the test.
+1. New spans within the Synapse Python Client should only be added if it will bring value to those looking at the spans. Some initial questions to ask yourself are:
+    * "Will this information help someone in the future to review an error in the code?"
+    * "Is this an external call or an entry point into the python client?"
+    * "Is it useful to know how long this section of code takes to execute?"
+
 ### Repository Admins
 
 - [Release process](https://sagebionetworks.jira.com/wiki/spaces/SYNPY/pages/643498030/Synapse+Python+Client+Staging+Validation+Production)

README.md

@@ -153,6 +153,61 @@ The purpose of synapseutils is to create a space filled with convenience functio
         print(dirname)
         print(filename)
 
+OpenTelemetry (OTEL)
+--------------------------------
+[OpenTelemetry](https://opentelemetry.io/) helps support the analysis of traces and spans which can provide insights into latency, errors, and other performance metrics. The synapseclient is ready to provide traces should you want them. The Synapse Python client supports OTLP Exports and can be configured via environment variables as defined [here](https://opentelemetry-python.readthedocs.io/en/stable/exporter/otlp/otlp.html).
+
+Read more about OpenTelemetry in Python [here](https://opentelemetry.io/docs/instrumentation/python/)
+
+### Quick-start
+The following shows an example of setting up [jaegertracing](https://www.jaegertracing.io/docs/1.50/deployment/#all-in-one) via docker and executing a simple python script that implements the Synapse Python client.
+
+#### Running the jaeger docker container
+Start a docker container with the following options:
+```
+docker run --name jaeger \
+  -e COLLECTOR_OTLP_ENABLED=true \
+  -p 16686:16686 \
+  -p 4318:4318 \
+  jaegertracing/all-in-one:latest
+```
+Explanation of ports:
+* `4318` HTTP
+* `16686` Jaeger UI
+
+Once the docker container is running you can access the Jaeger UI via: `http://localhost:16686`
+
+#### Example
+By default the OTEL exporter sends trace data to `http://localhost:4318/v1/traces`, however you may override this by setting the `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` environment variable.
+
+```
+import synapseclient
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.sdk.resources import SERVICE_NAME, Resource
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+
+trace.set_tracer_provider(
+    TracerProvider(
+        resource=Resource(attributes={SERVICE_NAME: "my_own_code_above_synapse_client"})
+    )
+)
+trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
+tracer = trace.get_tracer("my_tracer")
+
+@tracer.start_as_current_span("my_span_name")
+def main():
+    syn = synapseclient.Synapse()
+    syn.login()
+    my_entity = syn.get("syn52569429")
+    print(my_entity)
+
+main()
+```
+
+
+
 
 License and Copyright
 ---------------------

setup.cfg

@@ -51,6 +51,9 @@ install_requires =
     keyring>=15,<23.5
     keyrings.alt==3.1; sys_platform == "linux"
     deprecated>=1.2.4,<2.0
+    opentelemetry-api~=1.20.0
+    opentelemetry-sdk~=1.20.0
+    opentelemetry-exporter-otlp-proto-http~=1.20.0
 tests_require =
     pytest>=6.0.0,<7.0
     pytest-mock>=3.0,<4.0

synapseclient/__main__.py

@@ -29,6 +29,14 @@
     SynapseFileNotFoundError,
     SynapseNoCredentialsError,
 )
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
+from opentelemetry.sdk.resources import SERVICE_NAME, Resource
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.trace.sampling import ALWAYS_OFF, ALWAYS_ON, ParentBased
+
+tracer = trace.get_tracer("synapseclient")
 
 
 def _init_console_logging():
@@ -92,6 +100,7 @@ def _getIdsFromQuery(queryString, syn, downloadLocation):
         )
 
 
+@tracer.start_as_current_span("main::get")
 def get(args, syn):
     syn.multi_threaded = args.multiThreaded
     if args.recursive:
@@ -324,7 +333,8 @@ def cat(args, syn):
                 sys.stdout.write(line)
 
 
-def ls(args, syn):
+@tracer.start_as_current_span("main::ls")
+def ls(args, syn: synapseclient.Synapse):
     """List entities in a Project or Folder"""
     syn._list(
         args.id,
@@ -532,6 +542,7 @@ def _generate_new_config(auth_section):
     return new_config_text
 
 
+@tracer.start_as_current_span("main::config")
 def config(args, syn):
     """Create/modify a synapse configuration file"""
     user, secret = _prompt_for_credentials()
@@ -631,6 +642,7 @@ def get_download_list(args, syn):
     syn.logger.info(f"Manifest file: {manifest_path}")
 
 
+@tracer.start_as_current_span("main::login")
 def login(args, syn):
     """Log in to Synapse, optionally caching credentials"""
     login_with_prompt(
@@ -812,6 +824,14 @@ def build_parser():
         help="suppress checking for version upgrade messages and endpoint redirection",
     )
 
+    parser.add_argument(
+        "--otel",
+        type=str,
+        dest="otel",
+        choices=["console", "otlp"],
+        help="enabled the usage of OpenTelemetry for tracing",
+    )
+
     subparsers = parser.add_subparsers(
         title="commands",
         dest="subparser",
@@ -1725,6 +1745,7 @@ def build_parser():
     return parser
 
 
+@tracer.start_as_current_span("main::perform_main")
 def perform_main(args, syn):
     if "func" in args:
         if args.func != login and args.func != config:
@@ -1743,6 +1764,7 @@ def perform_main(args, syn):
         build_parser().print_help()
 
 
+@tracer.start_as_current_span("main::login_with_prompt")
 def login_with_prompt(
     syn, user, password, rememberMe=False, silent=False, forced=False
 ):
@@ -1782,6 +1804,7 @@ def _prompt_for_credentials(user=None):
     return user, passwd
 
 
+@tracer.start_as_current_span("main::_authenticate_login")
 def _authenticate_login(syn, user, secret, **login_kwargs):
     # login using the given secret.
     # we try logging in using the secret as a password, a auth bearer token, an api key in that order.
@@ -1838,13 +1861,32 @@ def main():
     synapseclient.USER_AGENT["User-Agent"] = (
         "synapsecommandlineclient " + synapseclient.USER_AGENT["User-Agent"]
     )
-    syn = synapseclient.Synapse(
-        debug=args.debug,
-        skip_checks=args.skip_checks,
-        configPath=args.configPath,
-        silent=args.silent,
-    )
-    perform_main(args, syn)
+    if args.otel:
+        trace.set_tracer_provider(
+            TracerProvider(
+                sampler=ParentBased(ALWAYS_ON),
+                resource=Resource(attributes={SERVICE_NAME: "synapseclient"}),
+            )
+        )
+        if args.otel == "otlp":
+            trace.get_tracer_provider().add_span_processor(
+                BatchSpanProcessor(OTLPSpanExporter())
+            )
+        elif args.otel == "console":
+            trace.get_tracer_provider().add_span_processor(
+                BatchSpanProcessor(ConsoleSpanExporter())
+            )
+    else:
+        trace.set_tracer_provider(TracerProvider(sampler=ParentBased(ALWAYS_OFF)))
+
+    with tracer.start_as_current_span("main::main"):
+        syn = synapseclient.Synapse(
+            debug=args.debug,
+            skip_checks=args.skip_checks,
+            configPath=args.configPath,
+            silent=args.silent,
+        )
+        perform_main(args, syn)
 
 
 if __name__ == "__main__":