Distributed Tracing in Go with OpenTelemetry: From Zero to Production

Posted on

You fire up your dashboard at 3 AM. Error rates are climbing. The checkout service is returning 500s. Your architecture has six microservices, a message queue, and a shared database. Where is the actual failure?

Logs alone won’t answer that question — they’re scattered across services with no shared correlation ID. Metrics tell you that something is wrong but not where. What you need is distributed tracing: a timeline that follows a single request as it hops from the API gateway to the auth service, through a Kafka topic, into the inventory service, and back.

OpenTelemetry has become the de facto standard for generating, collecting, and exporting this telemetry. It’s a CNCF graduated project with SDKs for every major language, vendor-neutral instrumentation, and support for traces, metrics, and logs under one umbrella. In this post, we’ll set up distributed tracing in a Go service from scratch — provider configuration, automatic HTTP instrumentation, manual span creation, and context propagation across service boundaries.

The Core Concepts

Before writing code, let’s nail down three terms that trip people up:

  • Trace: The complete path of a single request across all services. Identified by a 128-bit Trace ID.
  • Span: A single unit of work within a trace — one HTTP request, one database query, one function call. Spans have parent-child relationships that form a tree.
  • Context propagation: The mechanism that carries trace context (Trace ID, Span ID, trace flags) across process boundaries via HTTP headers. OpenTelemetry uses W3C Trace Context by default, specifically the traceparent header.

When service A calls service B, A creates a span for the outbound call, injects the trace context into the HTTP headers, and B extracts that context to create a child span. The result is a connected trace — one tree, regardless of how many services are involved.

Setting Up the Tracer Provider

Everything starts with a TracerProvider. This is the central configuration object that owns the exporter (where spans go), the sampler (which requests get traced), and the resource (metadata about your service). Let’s build one:

package main

import (
	"context"
	"log"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.41.0"
)

func initTracer() (*sdktrace.TracerProvider, error) {
	// Create the OTLP HTTP exporter (sends spans to a collector)
	exporter, err := otlptracehttp.New(context.Background(),
		otlptracehttp.WithEndpoint("localhost:4318"),
		otlptracehttp.WithInsecure(),
	)
	if err != nil {
		return nil, err
	}

	// Describe THIS service — appears as attributes on every span
	res, err := resource.New(context.Background(),
		resource.WithAttributes(
			semconv.ServiceNameKey.String("checkout-api"),
			semconv.ServiceVersionKey.String("1.4.0"),
		),
	)
	if err != nil {
		return nil, err
	}

	// Build the provider: batch spans, sample 100% in dev,
	// attach the resource to every span
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
		sdktrace.WithSampler(
			sdktrace.ParentBased(sdktrace.TraceIDRatioBased(1.0)),
		),
	)

	// Register as global so instrumentation libraries pick it up
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.TraceContext{},
			propagation.Baggage{},
		),
	)

	return tp, nil
}

func main() {
	tp, err := initTracer()
	if err != nil {
		log.Fatalf("failed to init tracer: %v", err)
	}
	defer func() {
		_ = tp.Shutdown(context.Background())
	}()

	// ... start your HTTP server here
}

A few things to notice. The otlptracehttp exporter sends spans over HTTP/protobuf to an OpenTelemetry Collector on port 4318 — the standard OTLP HTTP port. The BatchSpanProcessor (selected via WithBatcher) buffers spans and flushes them asynchronously in batches of up to 512, with a 5-second default delay. This is what you want in production: non-blocking, efficient, resilient to temporary collector downtime.

The sampler uses ParentBased with TraceIDRatioBased(1.0) as the root sampler. ParentBased means: if a trace already has a parent (it came from an upstream service that already decided to sample it), honor that decision. If this service is the trace origin, sample 100% of requests. In production, you’d drop that ratio to something like 0.1 to reduce volume and cost — but keep ParentBased so sampled traces stay complete across all hops.

The global propagator is critical. Setting TraceContext ensures that traceparent headers are injected on outbound requests and extracted on inbound ones. Without this line, your traces will be broken at every service boundary — each service starts a brand-new, disconnected trace.

Auto-Instrumenting HTTP Handlers

Manually wrapping every handler is tedious. The opentelemetry-go-contrib package provides otelhttp, which auto-instruments both inbound HTTP servers and outbound HTTP clients. One middleware wrapper handles span creation, status codes, HTTP method/route attributes, and context propagation automatically.

import (
	"log"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/checkout", checkoutHandler)
	mux.HandleFunc("/health", healthHandler)

	// Wrap the entire mux — every request gets a server span
	wrapped := otelhttp.NewHandler(mux, "checkout-api")

	log.Println("listening on :8080")
	http.ListenAndServe(":8080", wrapped)
}

Every incoming request now gets a span with the HTTP method, route, status code, and duration. The middleware automatically extracts the trace context from incoming traceparent headers, so if an upstream service started this trace, the span becomes a child of that trace — not a new root.

Propagating Context to Downstream Calls

When your checkout service calls the payment service, you need to propagate the trace context on the outbound HTTP request. The otelhttp.NewTransport wrapper handles this:

import (
	"context"
	"fmt"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

var paymentClient = &http.Client{
	Transport: otelhttp.NewTransport(http.DefaultTransport),
}

func callPaymentService(ctx context.Context, orderID string) error {
	req, err := http.NewRequestWithContext(ctx, "POST",
		"http://payment-service:8080/charge", nil)
	if err != nil {
		return err
	}
	req.Header.Set("X-Order-ID", orderID)

	resp, err := paymentClient.Do(req)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return fmt.Errorf("payment failed: %d", resp.StatusCode)
	}
	return nil
}

The key here is http.NewRequestWithContext — the ctx parameter carries the current span. otelhttp.NewTransport automatically injects the trace context into the outbound request headers and creates a client span. On the payment service side, otelhttp.NewHandler extracts that context and creates a child server span. The trace stays connected across the network boundary.

Creating Custom Spans for Business Logic

HTTP spans tell you when a request happened and how long it took. But they don’t reveal what happened inside — which database queries ran, which external APIs were called, where time was actually spent. For that, you need custom spans around specific operations:

import (
	"context"
	"fmt"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/trace"
)

var tracer = otel.Tracer("checkout-api/internal")

func processOrder(ctx context.Context, orderID string, items []string) error {
	// Create a span for this business operation
	ctx, span := tracer.Start(ctx, "processOrder",
		trace.WithSpanKind(trace.SpanKindInternal),
		trace.WithAttributes(
			attribute.String("order.id", orderID),
			attribute.Int("order.item_count", len(items)),
		),
	)
	defer span.End()

	// Sub-span: validate inventory
	ctx, validateSpan := tracer.Start(ctx, "validateInventory")
	err := validateItems(ctx, items)
	if err != nil {
		validateSpan.RecordError(err)
		validateSpan.SetStatus(codes.Error, "inventory validation failed")
		validateSpan.End()
		return fmt.Errorf("validate: %w", err)
	}
	validateSpan.End()

	// Sub-span: charge payment (calls downstream service)
	_, paymentSpan := tracer.Start(ctx, "chargePayment")
	if err := callPaymentService(ctx, orderID); err != nil {
		paymentSpan.RecordError(err)
		paymentSpan.SetStatus(codes.Error, err.Error())
		paymentSpan.End()
		return fmt.Errorf("payment: %w", err)
	}
	paymentSpan.SetStatus(codes.Ok, "")
	paymentSpan.End()

	return nil
}

Each tracer.Start call returns a new context containing the span and the span itself. Passing that context to child operations (like validateItems(ctx, items)) ensures nested spans become children of the current span — the tree structure builds itself organically through context propagation.

Two patterns worth highlighting: RecordError adds a structured exception event to the span (capturing the error type and message), and SetStatus(codes.Error, ...) marks the span as failed in a way that trace UIs like Jaeger and Tempo highlight visually. Always pair them — RecordError without SetStatus means the span still shows as successful in dashboards, which hides failures in error-rate queries.

Where the Data Goes: The Collector

The OTLP exporter sends spans to an OpenTelemetry Collector — a vendor-neutral intermediary that receives, processes, and routes telemetry. Running a local collector is straightforward with Docker:

docker run -p 4318:4318 -p 4317:4317 \
  otel/opentelemetry-collector:latest

The collector accepts spans via OTLP (HTTP on 4318, gRPC on 4317), applies processing pipelines (batching, attribute enrichment, tail-based sampling), and exports to any backend — Jaeger, Grafana Tempo, Datadog, Honeycomb, or all of them simultaneously. This decoupling means you can switch backends without touching application code.

Practical Tips

  • Always defer span.End(). Forgetting it means the span never gets exported. Using defer at creation is the safe pattern — deferred calls run even if the function panics.
  • Keep attributes low-cardinality. Use order.id for correlation, but don’t use user.email or request.body — they explode cardinality and leak PII. Use stable identifiers instead.
  • Flush on shutdown. The BatchSpanProcessor buffers spans asynchronously. Call tp.Shutdown(ctx) during graceful shutdown to flush the buffer. Without it, you lose the last few seconds of spans.
  • Sample intelligently. 100% sampling is fine for development. In production, TraceIDRatioBased(0.1) with ParentBased keeps traces complete while controlling volume. For finer control, run tail-based sampling on the collector instead.

Wrapping Up

Distributed tracing is the difference between guessing where latency lives and seeing it on a timeline. With OpenTelemetry, the instrumentation is vendor-neutral: you write the code once, and the same data feeds Jaeger in dev, Datadog in staging, and Tempo in production — no code changes required.

The setup we walked through — a configured TracerProvider, auto-instrumented HTTP handlers, context-propagating clients, and custom business spans — covers the vast majority of what you need for production tracing in Go services. Start with HTTP auto-instrumentation for immediate visibility, then add custom spans around the operations where the business logic actually lives.

The opentelemetry-go SDK and the contrib instrumentation library are the two dependencies you need. Pair them with a local collector during development, and you’ll have end-to-end traces flowing within minutes.

Leave a Reply

Your email address will not be published. Required fields are marked *