When a request crosses five services and the latency budget blows up, a single log line rarely tells you where the time went. You end up correlating timestamps by hand across machines, guessing at gaps, and hoping the clocks are roughly in sync. Distributed tracing solves this by recording the full journey of a request — every hop, every RPC, every queue wait — as a tree of timed operations you can query and visualize. This post walks through wiring up distributed tracing in Go with OpenTelemetry: a TracerProvider with a batching exporter, automatic instrumentation of an HTTP server and client with otelhttp, sampling configuration that survives production traffic, and the context-propagation rules that make everything else work.
Traces, spans, and context
Three concepts carry the whole model. A trace is the end-to-end story of one request. A span is one timed operation inside that story — an HTTP handler, a database call, a message publish. Spans form a tree through parent–child relationships, and each span carries a trace ID shared by every span in its trace. The context is how Go keeps track of the “current” span: the active span travels inside a context.Context, and any code that starts a child span must receive that context explicitly.
This last point matters more in Go than in most languages. Go’s idiom is to thread ctx through every function call, and OpenTelemetry leans on that discipline entirely. If you spawn a goroutine without passing the context, the child span either disappears or attaches to nothing — the trace fragments, and you’re back to correlating timestamps by hand.
Setting up the TracerProvider
The OpenTelemetry Go SDK splits into an API and an SDK. Application code (and libraries) use the API; only the application’s main wires up the SDK. The central object is the TracerProvider, which creates tracers, applies sampling, and hands finished spans to one or more processors. Here’s a setup suitable for a real service:
package main
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.34.0"
)
func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
exporter, err := otlptracehttp.New(ctx,
otlptracehttp.WithEndpointURL("http://localhost:4318/v1/traces"),
otlptracehttp.WithInsecure(),
)
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
sdktrace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceName("checkout-service"),
semconv.ServiceVersion("1.4.2"),
)),
)
otel.SetTracerProvider(tp)
return tp, nil
}A few details in there are worth pausing on. The resource describes who is emitting the telemetry — service name, version, deployment environment — and every span exported by this process is stamped with it. The BatchSpanProcessor (that’s what WithBatcher creates) queues spans in the background and ships them in groups: a queue of 2048 spans, a scheduled delay of 5 seconds, and batches of up to 512 spans by default. That batching is what keeps tracing from sitting on your request path; spans are ended synchronously but exported asynchronously. When the process shuts down, call tp.Shutdown(ctx) with a generous timeout so pending spans in the queue get flushed instead of dropped.
One configuration gotcha: when you point the OTLP HTTP exporter at a collector, be explicit about the full URL including the signal path. Depending on which endpoint option you use, the exporter may or may not append /v1/traces for you — passing the complete URL like http://localhost:4318/v1/traces removes any ambiguity. If the collector sits behind TLS, drop WithInsecure; if spans never arrive, an incorrect OTLP endpoint is the first thing to check, ahead of sampling configuration.
Instrumenting the HTTP server
Manual span management is tedious for the boring parts, and HTTP boundaries are exactly that. The otelhttp package from the OpenTelemetry Go contrib repository wraps an http.Handler so that every incoming request starts a server span, extracts propagation headers, and puts the span into the request context:
handler := http.HandlerFunc(paymentsHandler)
wrapped := otelhttp.NewHandler(handler, "payments",
otelhttp.WithTracerProvider(tp),
)
http.ListenAndServe(":8080", wrapped)Inside your handler, the request’s context now carries the active span. You can enrich it with attributes, record errors, or add events:
func paymentsHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
span := trace.SpanFromContext(ctx)
span.SetAttributes(attribute.String("payment.method", "card"))
err := chargeCard(ctx, r)
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "charge failed")
http.Error(w, "payment failed", http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}otelhttp.NewMiddleware is the equivalent for router-based setups — same behavior, but you compose it into a chi or gorilla middleware chain instead of wrapping a single handler.
Instrumenting the HTTP client
The server half only starts a trace. The client half is what continues it across service boundaries, and it’s the half people forget. otelhttp.NewTransport wraps an http.RoundTripper so that outgoing requests inject the current span’s context into W3C traceparent headers, start a client span, and record the response status:
client := &http.Client{
Transport: otelhttp.NewTransport(http.DefaultTransport),
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
"http://inventory:8080/reserve", body)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()Two things make this work. First, the request must be created with the handler’s context (NewRequestWithContext) — that’s where the active span lives. Second, the transport must actually be attached to the client you use; a stray http.Get call bypasses instrumentation silently, and the downstream service sees a request with no traceparent header. When both sides are wired this way, the downstream service’s otelhttp handler extracts the header, links its server span to your client span as its parent, and the trace grows a new branch with zero manual plumbing.
Sampling: keeping the firehose manageable
Full tracing of production traffic is rarely necessary and occasionally expensive. The SDK’s Sampler decides, per span, whether it gets recorded and exported. The two primitives you’ll combine are TraceIDRatioBased, which keeps a deterministic fraction of traces (sampling is decided once, on the root span, by hashing the trace ID), and ParentBased, which defers to the parent span’s decision when one exists and falls back to a delegate sampler when it doesn’t:
sampler := sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))
tp := sdktrace.NewTracerProvider(
sdktrace.WithSampler(sampler),
sdktrace.WithBatcher(exporter),
)The ParentBased wrapper is what keeps traces whole. Without it, a 10% head sampler running independently in five services would keep or drop each service’s spans separately, and the fragments you do export wouldn’t reassemble into complete traces. With it, the very first service to see the request makes the decision, and every downstream service honors it — you get all of a trace or none of it. For debugging a specific failure, you can also force decisions with AlwaysSample or drop health-check noise with a custom sampler that returns Drop for known-path roots.
Context propagation across goroutines and services
Go’s concurrency model is where tracing setups usually break. A span is not goroutine-safe as an “active” concept: only one span can be active per context, and contexts must be passed explicitly. Starting a child span around concurrent work means passing the parent context into each goroutine, not sharing a mutated one:
func fanOut(ctx context.Context) error {
tracer := otel.Tracer("orchestrator")
var wg sync.WaitGroup
errs := make(chan error, 2)
for _, svc := range []string{"inventory", "fraud"} {
wg.Add(1)
go func(svc string) {
defer wg.Done()
ctx, span := tracer.Start(ctx, "call."+svc)
defer span.End()
if err := call(ctx, svc); err != nil {
errs <- err
}
}(svc)
}
wg.Wait()
close(errs)
for err := range errs {
return err
}
return nil
}Note the pattern: each goroutine derives its own child span from the same parent context, so the resulting trace fans out into parallel branches under a single parent. What you must not do is call tracer.Start with context.Background() inside a goroutine — that orphans the span from the trace entirely. If a goroutine outlives the request, capture the trace.SpanContext (trace ID and flags) rather than the whole context and link the async work to the original trace with a span link instead of a parent relationship.
Across services, propagation is just the W3C traceparent header moving over the wire. otelhttp injects and extracts it for you on both sides, and the OpenTelemetry Go repository ships propagators and instrumentation for gRPC, databases, and message systems under the same contrib umbrella. If you’re integrating with a service in another language, the header format is the same — that’s the point of OpenTelemetry.
Where this pays off
Once two services are instrumented this way, a slow checkout stops being a mystery: you open a trace, see that 900 of the 950 milliseconds sat in an inventory lookup, and fix the actual problem instead of guessing. The setup cost is small — a provider in main, two wrappers around HTTP plumbing, a sampler — and it compounds as you add services, since each new instrumented hop extends existing traces automatically. Start with 100% sampling in development, drop to a parent-based ratio in production, ship spans to a collector over OTLP, and let the trace tree do what grep over logs never could.