Create and manage traces using OpenTelemetry open standards

Note	This repository contains the guide documentation source. To view the guide in published form, view it on the Open Liberty website.

Distributed tracing helps DevOps teams keep track of requests between microservices. MicroProfile adopts OpenTelemetry Tracing to allow developers to observe requests across their distributed systems.

What you’ll learn

The use of microservices architecture may increase difficultly to see how services depend on or affect other services. Consequently, making it harder to find the source of latency or inaccuracy.

One way to increase observability of an application is by emitting traces. OpenTelemetry is a set of APIs, SDKs, tooling and integrations that are designed for the creation and management of telemetry data such as traces, metrics, and logs. MicroProfile Telemetry 1.0 adopts OpenTelemetry so your MicroProfile applications can benefit from both manual and automatic traces!

Traces represent requests and consist of multiple spans. Spans are representative of single operations in a request and contain a name, time-related data, log messages and metadata to give information about what occurs during a transaction.

Context is an immutable object contained in the span data to identify the unique request that each span is a part of. This data is required for moving trace information across service boundaries, allowing developers to follow a single request through a potentially complex distributed system. Exporters are components that send data to a backend service so you can visualise and monitor the generated spans.

You’ll configure the provided inventory and system services to use Jaeger for distributed tracing with MicroProfile OpenTracing. You’ll run these services in two separate JVMs made of two server instances to demonstrate tracing in a distributed environment. If all the components were run on a single server, then any logging software would be sufficient.

Additional prerequisites

Before you begin, deploy the Jaeger all-in-one executable file to start the Jaeger tracing system. The Jaeger all-in-one executable file is configured for quick local testing. You can find information about the Jaeger server and instructions for starting the all-in-one executable file in the Jaeger documentation.

Before you proceed, make sure that your Jaeger server is up and running. Jaeger can be found at the http://localhost:16686 URL.

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/gitclone.adoc

Try what you’ll build

The finish directory in the root of this guide contains the finished application. Give it a try before you proceed.

Open a command-line session and navigate to the finish/system directory. Run the following Maven goal to build the system service and deploy it to Open Liberty:

mvn liberty:run

Open another command-line session and navigate to the finish/inventory directory. Run the following Maven goal to build the inventory service and deploy it to Open Liberty:

After you see the following message in both command-line sessions, both of your services are ready:

The defaultServer server is ready to run a smarter planet.

Make sure that your Jaeger server is running and point your browser to the http://localhost:9081/inventory/systems/localhost URL.

When you visit this endpoint, you make two GET HTTP requests, one to the system service and one to the inventory service. Both of these requests are configured to be traced, so a new trace is recorded in Jaeger.

To view the traces, go to the http://localhost:16686 URL. You can view the traces for the inventory or system services under the Search tab. Select the services in the Select a service menu and click the Find Traces button at the end of the section.

If you only see the jaeger-query option listed in the dropdown, you might need to wait a little longer and refresh the page to see the application services. View the traces for inventory. You’ll see the following trace:

The trace has five spans, four from inventory and one from system. Click the trace to view its details. Under Service & Operation, you see the spans in this trace. You can inspect each span by clicking it to reveal more detailed information, such as the time at which a request was received and the time at which a response was sent back.

After you’re finished reviewing the application, stop the Open Liberty servers by pressing CTRL+C in the command-line sessions where you ran the system and inventory services. Alternatively, you can run the following goals from the finish directory in another command-line session:

mvn -pl system liberty:stop
mvn -pl inventory liberty:stop

Building the application

You need to start the services to see basic traces appear in Jaeger.

When you run Open Liberty in development mode, known as dev mode, the server listens for file changes and automatically recompiles and deploys your updates whenever you save a new change.

Open a command-line session and navigate to the start/inventory directory. Run the following Maven goal to start the inventory service in dev mode:

mvn liberty:dev

Open a command-line session and navigate to the start/system directory. Run the following Maven goal to start the system service in dev mode:

mvn liberty:dev

After you see the following message, your application server in dev mode is ready:

**************************************************************
*    Liberty is running in dev mode.

Dev mode holds your command-line session to listen for file changes. Open another command-line session to continue, or open the project in your editor.

When the servers start, you can find the system and inventory services at the following URLs:

• http://localhost:9080/system/properties

• http://localhost:9081/inventory/systems

Enabling Telemetry implementation

MicroProfile Telemetry 1.0 allows you to observe traces without modifying source code in your Jakarta RESTful web service (aka JAX-RS) applications. As you can see in the inventory service, you can enable mpTelemetry-1.0 in your server.xml:

<featureManager>
  <feature>mpTelemetry-1.0</feature>
</featureManager>

By default, MicroProfile Telemetry tracing is off. To enable any tracing aspects, specify otel.sdk.disabled=false as a config property or OTEL_SDK_DISABLED=false as an environment property for your server.

For more information on these properties, see OpenTelemetry’s configuration documentation.

Collect traces using automatic instrumentation

To begin, open a new command-line session and navigate to the system directory:

cd system

Start the system service:

mvn liberty:run

Navigate to the system properties endpoint:

http://localhost:9080/system/properties

A span for this request will be automatically created by MicroProfile Telemetry. Check for the span in your chosen exporter’s endpoint. For example:

Manually create traces

Automatic instrumentation only instruments Jakarta RESTful web service applications. To get further spans on other operations, such as database calls, you can add manual instrumentation to the source code.

Enabling OpenTelemetry API and OpenTelemetry Instrumentation Annotations

The OpenTracing API is exposed as a third-party API in Open Liberty. To add the visibility of OpenTracing APIs to the application, add third-party to the types of API packages that this class loader supports.

Third-party APIs must be made visible for your application in the server.xml:

<webApplication location="demo-microprofile-telemetry-inventory.war" contextRoot="/">
    <!-- enable visibility to third party apis -->
    <classloader apiTypeVisibility="+third-party"/>
</webApplication>

server.xml

link:finish/inventory/src/main/liberty/config/server.xml[role=include]

The OpenTelemetry API and OpenTelemetry Instrumentation Annotations must be provided as dependencies to your build path. For example, with Maven, we add the following to the pom.xml file:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.19.0</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-instrumentation-annotions</artifactId>
    <version>1.19.0-alpha</version>
</dependency>

pom.xml

link:finish/inventory/pom.xml[role=include]

Manual instrumentation of application code

Now you are all set to instrument your code to manually create traces.

Use the @Traced annotation to define explicit span creation for specific classes and methods. If you place the annotation on a class, then the annotation is automatically applied to all methods within that class. If you place the annotation on a method, then the annotation overrides the class annotation if one exists.

Start by injecting a Tracer and Span object:

@Inject
Tracer tracer;

@Inject
Span getPropertiesSpan;

InventoryResource.java

link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[role=include]

This can then be used to create spans. For example, a span called GettingProperties starts and an event is added before the system service is called:

getPropertiesSpan = tracer.spanBuilder("GettingProperties").startSpan();
Properties props = manager.get(hostname);
try(Scope scope = getPropertiesSpan.makeCurrent()){
    ...
    getPropertiesSpan.addEvent("Received properties");
}
finally{
    getPropertiesSpan.end();
}

InventoryResource.java

link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[role=include]

You can also create new spans by annotating methods in any Jakarta CDI beans using @WithSpan. The InventoryManager.java file in the demo application creates a span when a new system is added to the inventory. The hostname method parameter is annotated with the @SpanAttribute annotation to indicate that it is part of the trace:

@WithSpan
public void add(@SpanAttribute(value = "hostname") String hostname, Properties systemProps) {
    ...
}

InventoryManager.java

link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java[role=include]

A span created using the @WithSpan annotation can be given a name. For example, this span is given the name list:

@WithSpan("list")
public InventoryList list() {
    ...
}

InventoryManager.java

link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java[role=include]

Now that you created the spans manually, we can build the inventory service and deploy it to Open Liberty:

cd inventory
mvn liberty:run

Navigate to the inventory endpoint:

http://localhost:9081/inventory/systems/localhost

You should see five spans in the exporter’s endpoint: four spans from inventory and one span from system, as shown in the following example:

These spans are all part of one single trace that is emitted from the request to the endpoint. It is therefore easier to identify the source of inaccuracy or latency in a single request by debugging spans individually.

For more information, see OpenTelemetry’s manual instrumentation documentation.

Testing the application

No automated tests are provided to verify the correctness of the traces. Manually verify these traces by viewing them on the Jaeger server.

A few tests are included for you to test the basic functionality of the services. If a test failure occurs, then you might have introduced a bug into the code.

Running the tests

Since you started Open Liberty in dev mode, run the tests for the system and inventory services by pressing the enter/return key in the command-line sessions where you started the services.

When you are done checking out the services, exit dev mode by pressing CTRL+C in the shell sessions where you ran the system and inventory services, or by typing q and then pressing the enter/return key.

Finally, stop the Jaeger service that you started in the Additional prerequisites section.

Great work! You’re done!

You developed an application that creates and manages traces using MicroProfile Telemetry 1.0 and Open Liberty.

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/attribution.adoc