For years, adding tracing to a Go service meant wrapping every handler, decorating every client, and threading context through every call by hand. The OpenTelemetry Go SDK automates almost all of that now — but only if you wire it up correctly. The difference between a tracing setup that produces a coherent service map and one that produces orphaned fragments usually comes down to three decisions: how you initialize the tracer provider, how you propagate context across process boundaries, and how you sample.
This post walks through a production-grade setup using the current Go SDK (v1.47.0) and the contrib instrumentation packages. Everything here targets Go 1.25 and 1.26.
The one-time setup: TracerProvider, resource, exporter
A TracerProvider is the root object of the tracing SDK. Create it once at startup, register it globally with otel.SetTracerProvider, and never create a second one. Every Tracer you obtain afterwards inherits its configuration — sampler, resource, exporter.
Three components matter in the initialization. The resource describes who is emitting spans: service name, version, deployment environment. Get this wrong and your spans land in an unusable pile in the backend. The exporter ships spans out of process; the OTLP-over-HTTP exporter is the common default. The sampler decides which traces survive — more on that below.
package main
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.41.0"
)
func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
exporter, err := otlptracehttp.New(ctx,
otlptracehttp.WithEndpoint("localhost:4318"),
otlptracehttp.WithInsecure(),
)
if err != nil {
return nil, err
}
res, err := resource.New(ctx,
resource.WithAttributes(
semconv.ServiceName("checkout-api"),
semconv.ServiceVersion("1.7.2"),
attribute.String("deployment.environment", "production"),
),
)
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(res),
sdktrace.WithSampler(
sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)),
),
)
otel.SetTracerProvider(tp)
return tp, nil
}Note the import alias: the SDK lives in sdk/trace while the API lives in otel/trace, so sdktrace is the conventional alias to keep both importable in the same file. The semconv package is version-pinned — v1.41.0 is the latest released set for the SDK version above. Attribute keys move between semconv versions occasionally, so pin and stay consistent rather than upgrading casually.
The batcher’s hidden buffers
WithBatcher wraps your exporter in a BatchSpanProcessor, and its defaults matter more than most people check. The queue holds 2048 spans; the processor flushes every 5 seconds or when a batch of 512 spans accumulates; export has a 30-second timeout.
Two failure modes fall out of this. First, if your service produces spans faster than the exporter can drain the queue, spans are dropped silently — the head of the queue is evicted, and your trace sampling rate quietly becomes lower than configured. If you see inexplicable trace gaps under load, check queue pressure before blaming the sampler. Second, the 5-second flush interval means spans from a short-lived request may not arrive at the backend until seconds after the response — do not write tests that assert a span exists immediately after a request returns.
For CLI tools, batch jobs, and serverless handlers, the defaults actively hurt you: the process may exit before any flush happens. Use sdktrace.WithSyncer(exporter) instead (synchronous, unbuffered export), and call tp.Shutdown(ctx) before exit. Shutdown flushes and releases resources; skipping it is the most common reason “my traces don’t show up in a lambda.”
Sampling: the ParentBased wrapper is not optional
The naive way to sample 10% of traces is sdktrace.TraceIDRatioBased(0.1). This has a subtle, nasty failure mode in any service that calls other services: sampling decisions are made per span, not per trace. Without the wrapper, your service can start a trace with a recorded span, then drop a child span — or vice versa — producing partial traces in the backend that look like instrumentation bugs.
The fix is the ParentBased wrapper:
tp := sdktrace.NewTracerProvider(
// ...
sdktrace.WithSampler(
sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)),
),
)ParentBased means: if a parent span context exists and was sampled, sample; if the parent was not sampled, don’t; only apply the ratio at the trace root. This makes the sampling decision happen exactly once per trace — at the entry point (usually the edge service) — and every downstream service respects it. The trade-off: head sampling means a whole trace is dropped if the entry point decides against it. If you need “always keep traces that hit an error,” head sampling can’t do that; you’d need a tail-sampling collector like the OpenTelemetry Collector with the tail-sampling processor.
Context propagation: the actual hard part
Traces only knit together if span context crosses process boundaries. Go’s answer is context.Context — the span context rides inside it — plus propagators that serialize it into carrier headers on the wire. The W3C standard carrier is the traceparent header, specified in the Trace Context spec.
Register the propagator once, next to the tracer provider:
otel.SetTextMapPropagator(
propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
),
)The composite order matters less than registering something — the SDK’s default propagator is a no-op, so without this line, nothing crosses service boundaries and every service starts a disjoint root trace. TraceContext carries the trace IDs; Baggage carries user-defined key/values (tenant IDs, feature flags) alongside. Baggage is propagated to every downstream service, so keep it small and never put secrets in it.
HTTP servers: wrap the mux, not each handler
The otelhttp package from contrib provides server instrumentation. The cleanest pattern wraps the entire http.ServeMux once:
mux := http.NewServeMux()
mux.HandleFunc("/orders", handleOrders)
mux.HandleFunc("/healthz", healthz)
handler := otelhttp.NewHandler(mux, "checkout-api",
otelhttp.WithSpanNameFormatter(func(operation string, r *http.Request) string {
return r.Method + " " + r.URL.Path
}),
)
log.Fatal(http.ListenAndServe(":8080", handler))The wrapper does three things per request: extracts an incoming span context from the traceparent header (continuing the caller’s trace), starts a SERVER-kind span, and stores it in the request’s context — which *http.Request.Context() hands to your handler. The default span name is the operation string passed to NewHandler, which would give every route the same name and wreck your backend’s aggregation; the WithSpanNameFormatter option fixes that. Route names are still coarse here — for templated paths like /orders/{id} rather than concrete paths, Go 1.22+ pattern routes plus a formatter over r.Pattern keeps cardinality bounded.
One gotcha: wrapping the mux means /healthz gets traced too. Health checks are often your highest-volume, lowest-value spans. Either exclude them (via a filter handler or the formatter returning a NoOp tracer path) or accept the noise and filter at the collector.
HTTP clients: wrap the transport
Server-side instrumentation extracts context; client-side instrumentation injects it. The symmetric tool is otelhttp.NewTransport, which wraps an http.RoundTripper:
client := &http.Client{
Transport: otelhttp.NewTransport(http.DefaultTransport),
Timeout: 5 * time.Second,
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := client.Do(req)The transport reads the span context from the request’s context and writes the traceparent header onto the outgoing request. Two requirements hide in those three lines. The request must carry the handler’s context (r.Context() passed down, or a child of it) — a request built with context.Background() has no span context to inject. And the client must use the wrapped transport — creating a bare http.Client{} somewhere in a dependency bypasses instrumentation entirely and starts a fresh trace at the next service.
This is where most real-world traces silently break: an HTTP call buried three layers deep in a library that constructs its own client. When a service map shows a missing hop, audit for un-wrapped http.Clients before suspecting the propagator.
Custom spans for the parts instrumentation can’t see
HTTP-level spans tell you a request was slow; they don’t tell you why. For database calls, cache lookups, or any interesting internal operation, start explicit spans:
var tracer = otel.Tracer("checkout-api/orders")
func lookupOrder(ctx context.Context, id string) (Order, error) {
ctx, span := tracer.Start(ctx, "db.lookup_order",
trace.WithSpanKind(trace.SpanKindInternal),
trace.WithAttributes(attribute.String("order.id", id)),
)
defer span.End()
order, err := store.Get(ctx, id)
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "order lookup failed")
return Order{}, err
}
return order, nil
}The conventions that keep backends useful: pass the context you received into tracer.Start so the span nests correctly (and hand the returned context to everything downstream of the operation); always pair with defer span.End(); use RecordError plus SetStatus(codes.Error, ...) on failure — recording the error alone does not mark the span failed, and many backends color their waterfall by status, not by recorded events.
Keep span names low-cardinality: db.lookup_order aggregates; db.lookup_order order-8842 explodes into unqueryable noise. Values that vary per call — order IDs, user IDs — belong in attributes, which most backends index separately.
Putting the pipeline together
A complete minimal service: initialize the provider and propagator at startup, wrap the mux with otelhttp.NewHandler, wrap every outbound client transport, add custom spans for expensive internal operations, and shut down cleanly:
func main() {
ctx := context.Background()
tp, err := initTracer(ctx)
if err != nil {
log.Fatal(err)
}
defer func() {
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := tp.Shutdown(shutdownCtx); err != nil {
log.Printf("tracer shutdown: %v", err)
}
}()
mux := http.NewServeMux()
mux.HandleFunc("/orders", handleOrders)
handler := otelhttp.NewHandler(mux, "checkout-api")
log.Fatal(http.ListenAndServe(":8080", handler))
}To see the result locally, run an OTLP-capable backend such as Jaeger and point the exporter at it — the exporter defaults to localhost:4318 for HTTP, with the endpoint and credentials configurable via OTEL_EXPORTER_OTLP environment variables instead of code if you prefer configuration over compilation.
Wrapping up
- Create one
TracerProviderat startup with a proper resource; register it globally. - Always wrap root samplers in
ParentBasedso the sampling decision happens once per trace. - Register
TraceContext(plus Baggage if needed) as the propagator — the default is a no-op. - Wrap the mux with
otelhttp.NewHandlerand every outbound client withotelhttp.NewTransport; one bare client breaks the chain. - Use
WithSyncer+ explicitShutdownfor short-lived processes; check batch-processor queue pressure when traces vanish under load. - Keep span names and route names low-cardinality; put per-request values in attributes.
Once spans flow reliably, the next step is usually reducing noise: tail sampling in the Collector to keep error traces at 100%, and adding metrics and logs with the same trace IDs so all three signals correlate. But none of that matters until the spans knit together — and now they will.