Skip to main content

OpenTelemetry v2 integration

View Markdown

Temporal's OpenTelemetry integration lets you understand the internal state of Temporal applications across Clients, Workflows, Activities, and Nexus Operations by instrumenting them with OpenTelemetry.

OpenTelemetry instruments your applications to give you insight into your deployed environments. Temporal Workflows complicate that picture because a trace can span across different Workers over long stretches of time, which can scatter a trace into disconnected fragments. The OpenTelemetry plugin solves this by propagating OpenTelemetry context across those Temporal boundaries, keeping a trace intact end to end. It can also generate spans and emit metrics for Temporal SDK operations automatically.

All code snippets in this guide are taken from the OpenTelemetry v2 sample. Refer to the sample for complete code.

Prerequisites​

Install​

Add the OpenTelemetry v2 integration to your Go module:

go get go.temporal.io/sdk/contrib/opentelemetry-v2@latest

Also add the OpenTelemetry SDK packages and the exporter or metric reader your backend requires.

Set up the tracer provider​

A Tracer Provider is a factory for Tracers, and it configures the Tracers it creates, including how they generate span IDs. A standard Tracer Provider assigns a new random span ID each time a span is created, but Temporal Workflows replay, re-executing the same code and recreating what should be the same span with a different random ID each time. Temporal's replay-safe Tracer Provider avoids this by generating span IDs from a deterministic source tied to the Workflow, so the same span gets the same ID on every replay. Create it and install it as the OpenTelemetry global before you create the plugin or call Tracer.

opentelemetry-v2/setup.go

// ...
	provider := temporalotel.NewReplaySafeTracerProvider(
		// WithBatcher performs exporter I/O outside the Workflow goroutine.
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		)),
	)
	otel.SetTracerProvider(provider)

Your application owns the Tracer Provider for the life of the process. Shut it down before exit so remaining spans can flush through the trace exporter.

Set up the meter provider​

A Meter Provider is a factory for Meters. OpenTelemetry's default global Meter Provider is a no-op, so if you enable MetricsHandlerOptions, you need to supply a configured one yourself, either by installing it with otel.SetMeterProvider before you create the plugin, or by passing a Meter directly through MetricsHandlerOptions.Meter.

Add the plugin​

Pass the plugin to your Temporal Client when you create it. Workers made from that Client get the plugin automatically.

opentelemetry-v2/workflow-activity-propagation/worker/main.go

plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

c, err := client.Dial(client.Options{Plugins: []client.Plugin{plugin}})
if err != nil {
	return fmt.Errorf("unable to create client: %w", err)
}
defer c.Close()

By default the plugin only performs context propagation so Span Context can cross Temporal boundaries.

Add custom spans​

In Workflows​

A Tracer creates spans that capture information about a given operation. A standard Tracer stamps a span with the current time and emits it as soon as it completes, but Temporal Workflows replay, re-executing the same code and stamping what should be the same span with a new time and emitting a duplicate span. Temporal's replay-safe Tracer avoids this by stamping a span with workflow.Now, Temporal's replay-safe clock, and skipping a span that already completed on a previous successful execution. Use it instead of otel.Tracer in Workflows.

opentelemetry-v2/workflow-activity-propagation/opentelemetry.go

// ...
func Workflow(ctx workflow.Context, name string) (string, error) {
	tracer := temporalotel.Tracer(instrumentationName)
	ctx, span := tracer.Start(ctx, "workflow-operation")
	defer span.End()

	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	})

	var result string
	if err := workflow.ExecuteActivity(ctx, Activity, name).Get(ctx, &result); err != nil {
		return "", err
	}

	return result, nil
}

As in OpenTelemetry Go, Start returns a context that contains the active span. Pass that workflow.Context to downstream Temporal calls so later spans nest under it as children.

Outside Workflows​

In Clients, Activities, and other non-Workflow code, use an ordinary OpenTelemetry Tracer:

opentelemetry-v2/workflow-activity-propagation/opentelemetry.go

// ...
func Activity(ctx context.Context, name string) (string, error) {
	_, span := otel.Tracer(instrumentationName).Start(ctx, "activity-operation")
	defer span.End()

	return fmt.Sprintf("Hello, %s!", name), nil
}

Enable automatic instrumentation​

opentelemetry-v2/automatic-instrumentation/worker/main.go

plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{
	TracerOptions: tracing.TracerOptions{
		AddTemporalSpans: true,
	},
	MetricsHandlerOptions: &temporalotel.MetricsHandlerOptions{
		UseMonotonicCounters: true,
	},
})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

AddTemporalSpans​

Set AddTemporalSpans to true to create spans for Temporal SDK operations across Clients, Workflows, Activities, and Nexus Operations.

MetricsHandlerOptions​

Set MetricsHandlerOptions to a non-nil value to emit Temporal SDK metrics through OpenTelemetry.

Configure context propagation​

Context propagation is how OpenTelemetry moves context across process boundaries, injecting it on the way out and extracting it on the way in. The plugin performs this propagation for you across Temporal boundaries, carrying Span Context, which keeps spans linked into one trace, and baggage, optional key-value data that travels with the context.

Do not put credentials, tokens, or personal data in baggage since the plugin serializes it into Temporal headers that can be persisted in Workflow Event History.

TextMapPropagator​

The plugin injects and extracts both with a TextMapPropagator. By default that propagator supports W3C Trace Context and W3C Baggage. Set PluginOptions.TextMapPropagator to override it.

HeaderKey​

Propagated values are stored in the Temporal header under _tracer-data. Set TracerOptions.HeaderKey to use a different key.

DisableBaggage​

Set DisableBaggage to true to stop propagating baggage.

AllowInvalidParentSpans​

Set AllowInvalidParentSpans to true to ignore errors when extracting Span Context from Temporal headers. Use this when migrating between tracing libraries while Workflows or Activities are still in progress.

Resources​