This guide explains how your Quarkus application can utilize OpenTelemetry (OTel) to provide Observability for interactive web applications.

On these page we show the signal independent features of the extension.

This document is part of the Observability in Quarkus reference guide which features this and other observability related components.

Introduction

OpenTelemetry is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. Crucially, OpenTelemetry is vendor- and tool-agnostic.

Quarkus provides manual and automatic instrumentation for tracing and manual instrumentation capabilities for metrics.

This will allow Quarkus based applications to be observable by tools and services supporting OpenTelemetry.

Automatic metrics instrumentation in Quarkus is done by the Quarkus Micrometer extension. We plan to provide, in the future, a bridge for those metrics to be available in OpenTelemetry as well.

Quarkus supports the OpenTelemetry Autoconfiguration. The configurations match what you can see at OpenTelemetry SDK Autoconfigure with the quarkus.* prefix.

This guide provides a crosscutting explanation of the OpenTelemetry extension and how to use it. If you need details about any particular signal (tracing or metrics), please refer to the signal specific guide.

With the introduction of OpenTelemetry Metrics, the original, single page guide had to be split according to signal types, as follows:

OpenTelemetry Tracing Guide

The tracing functionality is supported and on by default.

OpenTelemetry Metrics Guide

Enable Metrics

The metrics functionality is experimental and off by default. You will need to activate it by setting:

quarkus.otel.metrics.enabled=true

At build time on your application.properties file.

OpenTelemetry Logging Guide

Enable Logs

The logging functionality is experimental and off by default. You will need to activate it by setting:

quarkus.otel.logs.enabled=true

At build time on your application.properties file.

Using the extension

If you already have your Quarkus project, you can add the quarkus-opentelemetry extension to it by running the following command in your project base directory:

CLI
quarkus extension add opentelemetry
Maven
./mvnw quarkus:add-extension -Dextensions='opentelemetry'
Gradle
./gradlew addExtension --extensions='opentelemetry'

This will add the following to your build file:

pom.xml
<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-opentelemetry</artifactId>
</dependency>
build.gradle
implementation("io.quarkus:quarkus-opentelemetry")

Create the configuration

By default, the exporters will send out data in batches, using the gRPC protocol and endpoint http://localhost:4317.

If you need to change any of the default property values, here is an example on how to configure the default OTLP gRPC Exporter within the application, using the src/main/resources/application.properties file:

quarkus.application.name=myservice (1)
quarkus.otel.exporter.otlp.endpoint=http://localhost:4317 (2)
quarkus.otel.exporter.otlp.headers=authorization=Bearer my_secret (3)
quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, parentId=%X{parentId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n  (4)

# Alternative to the console log
quarkus.http.access-log.pattern="...traceId=%{X,traceId} spanId=%{X,spanId}" (5)
1 All telemetry created from the application will include an OpenTelemetry Resource attribute indicating the telemetry was created by the myservice application. If not set, it will default to the artifact id.
2 gRPC endpoint to send the telemetry. If not set, it will default to http://localhost:4317.
3 Optional gRPC headers commonly used for authentication
4 Add tracing information into log messages.
5 You can also only put the trace info into the access log. In this case you must omit the info in the console log format.

We provide signal agnostic configurations for the connection related properties, meaning that you can use the same properties for both tracing and metrics when you set:

quarkus.otel.exporter.otlp.endpoint=http://localhost:4317

If you need different configurations for each signal, you can use the specific properties:

quarkus.otel.exporter.otlp.traces.endpoint=http://trace-uri:4317 (1)
quarkus.otel.exporter.otlp.metrics.endpoint=http://metrics-uri:4317 (2)
quarkus.otel.exporter.otlp.logs.endpoint=http://logs-uri:4317 (3)
1 The endpoint for the traces exporter.
2 The endpoint for the metrics exporter.
3 The endpoint for the logs exporter.

If you need that your spans and logs to be exported directly as they finish (e.g. in a serverless environment / application), you can set this property to true. This replaces the default batching of data.

quarkus.otel.simple=true

Disable all or parts of the OpenTelemetry extension

Once you add the dependency, the extension will generate tracing data by default. To enable metrics or disable the OpenTelemetry extension globally or partially these are the properties to use (they are extracted from the config reference below):

Affected Signal Property name Default value Description

All

quarkus.otel.enabled

true

If false, disable the OpenTelemetry usage at build time.

All

quarkus.otel.sdk.disabled

false

Comes from the OpenTelemetry autoconfiguration. If true, will disable the OpenTelemetry SDK usage at runtime.

All output

quarkus.otel.exporter.otlp.enabled

true

Deprecated for removal.

If false will disable the default OTLP exporter at build time.

Traces

quarkus.otel.traces.enabled

true

If false, disable the OpenTelemetry tracing usage at build time.

Traces output

quarkus.otel.traces.exporter

cdi

List of exporters to be used for tracing, separated by commas. Has one of the values from ExporterType: otlp, cdi, none. This is a build time property and setting it to none will disable tracing data output.

Metrics

quarkus.otel.metrics.enabled

false

Metrics are disabled by default at build time because they are experimental.

Metrics output

quarkus.otel.metrics.exporter

cdi

List of exporters to be used for metrics, separated by commas. Has one of the values from ExporterType: otlp, cdi, none. This is a build time property and setting it to none will disable metrics data output.

Logs

quarkus.otel.logs.enabled

false

Logs are disabled by default at build time because they are experimental.

Logs output

quarkus.otel.logs.exporter

cdi

List of exporters to be used for logs, separated by commas. Has one of the values from ExporterType: otlp, cdi, none. This is a build time property and setting it to none will disable logs data output.

Logs output

quarkus.otel.logs.handler.enabled

true

If false, disable the OpenTelemetry logs handler at runtime. This removes the bridge between the Quarkus logging system (JBoss LogManager) and OpenTelemetry logs.

If you need to enable or disable the exporter at runtime, you can use the sampler because it has the ability to filter out all the spans if needed.

Particular instrumentation components can be disabled in tracing, like ignore client requests but keep server ones. For more details, please check the OpenTelemetry Tracing Guide.

Resource

A resource is a representation of the entity that is producing telemetry, it adds attributes to the exported trace or metric to characterize who is producing the telemetry. Quarkus follows the resources auto-configuration specified by the Java OpenTelemetry SDK.

Default

The following attributes are added by default to resources.

Attribute name Content example Origin

service.name

"opentelemetry-quickstart"

Value comes from the artifactId, from the quarkus.application.name property or from quarkus.otel.resource.attributes=service.name=cart property.

host.name

"myHost"

Resolved at startup

service.version

"1.0-SNAPSHOT"

Resolved at build time from the artifact version

telemetry.sdk.language

"java"

Static value

telemetry.sdk.name

"opentelemetry"

Resolved at build time

telemetry.sdk.version

"1.32.0"

Resolved at build time

webengine.name

"Quarkus"

Static value

webengine.version

"999-SNAPSHOT"

Quarkus version resolved at build time

Using configuration

You can add additional attributes by setting the quarkus.otel.resource.attributes config property that is described in the OpenTelemetry Configuration Reference. Since this property can be overridden at runtime, the OpenTelemetry extension will pick up its value following the order of precedence that is described in the Quarkus Configuration Reference.

quarkus.otel.resource.attributes=deployment.environment=dev,service.name=cart,service.namespace=shopping

This will add the attributes for deployment.environment, service.name and service.namespace to the resource and be included in traces and metrics.

Using CDI beans

If by any means you need to use a custom resource or one that is provided by one of the OpenTelemetry SDK Extensions you can create multiple resource producers. The OpenTelemetry extension will detect the Resource CDI beans and will merge them when configuring the OTel SDK.

@ApplicationScoped
public class CustomConfiguration {

    @Produces
    @ApplicationScoped
    public Resource osResource() {
        return OsResource.get();
    }

    @Produces
    @ApplicationScoped
    public Resource ecsResource() {
        return EcsResource.get();
    }
}

Semantic conventions

OpenTelemetry provides a set of semantic conventions to standardize the data collected by the instrumentation.

When creating manual instrumentation, while naming metrics or attributes you should follow those conventions and not create new names to represent existing conventions. This will make data correlation easier to perform across services.

Exporters

The Default

The Quarkus OpenTelemetry extension uses its own signal exporters built on top of Vert.x for optimal performance and maintainability. All Quarkus built in exporters use the OTLP protocol through a couple of data senders, using grpc (the default) and http/protobuf.

The active exporter is automatically wired by CDI, that’s why the quarkus.otel.traces.exporter, quarkus.otel.metrics.exporter and quarkus.otel.logs.exporter properties default value is cdi. This is not because of the protocol being used in the data transfer but because of how the exporters are wired.

CDI (Context Dependency Injection) will manage the exporters to use, according to the selected protocol or when applications implement their own CDI exporter, like in tests.

The quarkus.otel.exporter.otlp.protocol property instructs Quarkus to switch the senders and defaults to grpc but http/protobuf can also be used.

If you change the protocol, you also need to change the port in the endpoint. The default port for grpc is 4317 and for http/protobuf is 4318.

Using CDI to produce a test exporter

Leaving the default as CDI is particularly useful for tests. In the following example a Span exporter class is wired with CDI and then the telemetry can be used in test code.

Creating a custom SpanExporter bean:

    @ApplicationScoped
    static class InMemorySpanExporterProducer {
        @Produces
        @Singleton
        InMemorySpanExporter inMemorySpanExporter() {
            return InMemorySpanExporter.create();
        }
    }

Where InMemorySpanExporter is a class from the OpenTelemetry test utilities dependency:

pom.xml
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk-testing</artifactId>
        <scope>test</scope>
    </dependency>
build.gradle
implementation("io.opentelemetry:opentelemetry-sdk-testing")

The bean of that class can be injected to access the telemetry data. This is an example to obtain the spans:

    @Inject
    InMemorySpanExporter inMemorySpanExporter;

    //...

    List<SpanData> finishedSpanItems = inMemorySpanExporter.getFinishedSpanItems();

If this is used in an integration test, you should access the class from inside the running process and not from the test class. A viable option could be to expose that data through a REST endpoint method:

    @GET
    @Path("/export")
    public List<SpanData> exportTraces() {
        return inMemorySpanExporter.getFinishedSpanItems()
                .stream()
                .filter(sd -> !sd.getName().contains("export")) (1)
                .collect(Collectors.toList());
    }
1 This excludes calls to the export endpoint itself.

For more details please take a look at the ExporterResource in the Quarkus integration tests.

The OpenTelemetry OTLP exporter

This is currently not supported in Quarkus. Configuration example for traces: quarkus.otel.tracing.exporter=otlp.

However, it’s also not needed because Quarkus own default exporters will send data using the OTLP protocol.

On Quarkiverse

Additional exporters will be available in the Quarkiverse quarkus-opentelemetry-exporter project.

Currently, are available the following exporters (may be outdated) for:

  • Legacy Jaeger

  • Microsoft Azure

  • Google Cloud

Also on Quarkiverse, the Quarkus AWS SDK has integration with OpenTelemetry.

Logging exporter (for debugging)

You can output all metrics to the console, for debugging/development purposes.

Don’t use this in production.

You will need to add the following dependency to your project:

pom.xml
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-logging</artifactId>
</dependency>
build.gradle
implementation("io.opentelemetry:opentelemetry-exporter-logging")

Then, setting the exporter to logging in the application.properties file:

quarkus.otel.metrics.exporter=logging (1)
quarkus.otel.metric.export.interval=10000ms (2)
quarkus.otel.traces.exporter=logging (3)
1 Set the metrics exporter to logging. Normally you don’t need to set this. The default is cdi.
2 Set the interval to export the metrics. The default is 1m, which is too long for debugging.
3 Set the traces exporter to logging. Normally you don’t need to set this. The default is cdi.

Visualizing the data

We recommend the Getting Started with Grafana-OTel-LGTM.

This provides a Quarkus Dev service using an "all-in-one" Grafana OTel LGTM.

Grafana is used to visualize data, Loki to store logs, Tempo to store traces and Prometheus to store metrics. Also provides and OTel collector to receive the data.

This provides an easy way to visualize all OpenTelemetry data generated by the application.

You can also use the logging exporter to output all traces and metrics to the console.

OpenTelemetry Configuration Reference

Quarkus supports the OpenTelemetry Autoconfiguration for Traces. The configurations match what you can see at OpenTelemetry SDK Autoconfigure adding the usual quarkus.* prefix.

Quarkus OpenTelemetry configuration properties now have the quarkus.otel.* prefix.