Microservices promised us smaller, more maintainable codebases. They delivered — along with a debugging nightmare. When a request flows through five services and the response time spikes, where’s the bottleneck? Without proper instrumentation, you’re reduced to reading timestamps in log files and guessing.
OpenTelemetry has become the standard answer. It’s a vendor-neutral specification and toolkit for generating, collecting, and exporting telemetry — traces, metrics, and logs — from your applications. The Go SDK is mature, with traces and metrics at stable status and logging in beta as of the current v1.44.0 release.
This post walks through setting up real, production-grade observability in a Go service: configuring the OTLP exporter, creating traces with proper span attributes, recording metrics that matter, and instrumenting HTTP handlers automatically. No theoretical survey — just the code and configuration you need to make it work.
The Three Signals and What You Actually Need
OpenTelemetry defines three telemetry signals. Traces follow a single request across service boundaries, giving you a call tree with timing data. Metrics are aggregated numerical measurements — request counts, latencies, error rates — collected at regular intervals. Logs are timestamped events, structured and correlated with traces through shared context.
For a Go microservice, the practical setup involves four pieces. First, a Resource that identifies your service (name, version, environment). Second, a TracerProvider configured with a batch span processor and an OTLP exporter. Third, a MeterProvider with the same exporter setup for metrics. Fourth, auto-instrumentation for HTTP, gRPC, or database drivers from the contrib repository to capture telemetry without littering your business logic with boilerplate.
Setting Up the Provider Pipeline
The first step is initializing the SDK and connecting it to a collector. You can use the OTLP HTTP exporter to send telemetry to any OpenTelemetry-compatible backend — Jaeger, Grafana Tempo, Datadog, Honeycomb, or a self-hosted collector.
package main
import (
"context"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.41.0"
)
func initProvider() (func(context.Context) error, error) {
ctx := context.Background()
// Resource identifies your service in the telemetry backend.
res, err := resource.New(ctx,
resource.WithAttributes(
semconv.ServiceName("order-service"),
semconv.ServiceVersion("1.4.2"),
attribute.String("environment", os.Getenv("DEPLOY_ENV")),
),
)
if err != nil {
return nil, fmt.Errorf("resource: %w", err)
}
// Trace exporter — OTLP over HTTP.
traceExporter, err := otlptracehttp.New(ctx,
otlptracehttp.WithEndpoint("otel-collector:4318"),
otlptracehttp.WithInsecure(),
)
if err != nil {
return nil, fmt.Errorf("trace exporter: %w", err)
}
// Batch span processor for efficiency.
bsp := sdktrace.NewBatchSpanProcessor(traceExporter)
tp := sdktrace.NewTracerProvider(
sdktrace.WithResource(res),
sdktrace.WithSpanProcessor(bsp),
sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.5)),
)
otel.SetTracerProvider(tp)
// Metric exporter.
metricExporter, err := otlpmetrichttp.New(ctx,
otlpmetrichttp.WithEndpoint("otel-collector:4318"),
otlpmetrichttp.WithInsecure(),
)
if err != nil {
return nil, fmt.Errorf("metric exporter: %w", err)
}
mp := metric.NewMeterProvider(
metric.WithResource(res),
metric.WithReader(metric.NewPeriodicReader(metricExporter)),
)
otel.SetMeterProvider(mp)
// W3C Trace Context propagation for cross-service tracing.
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
))
// Return a shutdown function for graceful termination.
return func(ctx context.Context) error {
shutdownErr := tp.Shutdown(ctx)
_ = mp.Shutdown(ctx)
return shutdownErr
}, nil
}
func main() {
shutdown, err := initProvider()
if err != nil {
log.Fatalf("failed to init telemetry: %v", err)
}
defer func() {
_ = shutdown(context.Background())
}()
// ... start your HTTP server here ...
// Block until shutdown signal.
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
<-sigCh
}Several things deserve attention here. The sampler is set to TraceIDRatioBased(0.5), which exports half of all traces. In production, you'll tune this based on traffic volume — too high and you overwhelm your backend, too low and you miss rare errors. The BatchSpanProcessor buffers spans and flushes them asynchronously, which is essential for throughput. The composite propagator enables W3C Trace Context, which is what allows spans to be linked across service boundaries when using HTTP clients instrumented with the otelhttp middleware.
Auto-Instrumenting HTTP Handlers
Manually creating spans for every handler is tedious and error-prone. The otelhttp package wraps net/http handlers and clients with automatic span creation, status codes, and error recording.
import (
"net/http"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("GET /orders/{id}", getOrder)
mux.HandleFunc("POST /orders", createOrder)
// Wrap the entire mux — every route gets traced.
handler := otelhttp.NewHandler(mux, "order-service-http")
http.ListenAndServe(":8080", handler)
}
func getOrder(w http.ResponseWriter, r *http.Request) {
// The otelhttp middleware already created a span.
// Retrieve it from context to add custom attributes.
ctx := r.Context()
span := trace.SpanFromContext(ctx)
span.SetAttributes(
attribute.String("order.id", r.PathValue("id")),
)
// ... business logic ...
}The middleware automatically sets standard HTTP semantic conventions — http.method, http.route, http.status_code, and http.url — on each span. When your handler calls another service, wrap the outbound HTTP client the same way to propagate the trace context:
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
// Wrap the client transport.
client := &http.Client{
Transport: otelhttp.NewTransport(http.DefaultTransport),
}
// Requests made with this client will propagate
// the trace context header to downstream services.
resp, err := client.Get("http://inventory-service:8080/stock/" + sku)
That's the entire mechanism for distributed tracing across services. The client transport injects the W3C Trace Context header, the downstream service's otelhttp middleware extracts it, and suddenly your trace tree spans multiple services — all without you manually threading correlation IDs.
Recording Business Metrics
Traces tell you about individual requests. Metrics tell you about aggregate behavior. The Go SDK supports four core instrument types: counters (monotonically increasing), up-down counters (can decrease), histograms (distribution of values), and gauges (point-in-time readings).
import (
"go.opentelemetry.io/otel/metric"
)
var (
orderCounter metric.Int64Counter
orderDuration metric.Float64Histogram
activeOrders metric.Int64UpDownCounter
)
func initMetrics() {
meter := otel.Meter("order-service")
orderCounter, _ = meter.Int64Counter(
"orders.created",
metric.WithUnit("1"),
metric.WithDescription("Number of orders created"),
)
orderDuration, _ = meter.Float64Histogram(
"orders.processing_duration",
metric.WithUnit("ms"),
metric.WithDescription("Time to process an order"),
)
activeOrders, _ = meter.Int64UpDownCounter(
"orders.active",
metric.WithDescription("Orders currently being processed"),
)
}
func createOrder(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
activeOrders.Add(ctx, 1)
defer activeOrders.Add(ctx, -1)
start := time.Now()
// ... process order ...
elapsed := float64(time.Since(start).Milliseconds())
orderCounter.Add(ctx, 1,
metric.WithAttributes(
attribute.String("order.status", "completed"),
),
)
orderDuration.Record(ctx, elapsed)
}One critical lesson: be careful with metric attributes that have high cardinality. Each unique combination of attribute values creates a new time series. A counter tagged with user_id will create one series per user — tens of thousands in a real system, which can overwhelm both your memory and your backend. The SDK now enforces a default cardinality limit of 2000 series per instrument, but you should still design metrics with bounded attribute sets. Tag by status_code, endpoint, and error_type — not by entity ID.
Adding Custom Spans for Internal Operations
Auto-instrumentation covers HTTP boundaries. For internal logic — database calls, external API requests, expensive computations — you'll create child spans manually. The pattern is always the same: start a span, defer its end, and let the context propagate.
var tracer = otel.Tracer("order-service")
func chargeCreditCard(ctx context.Context, cardToken string, amountCents int64) error {
ctx, span := tracer.Start(ctx, "charge_credit_card",
trace.WithAttributes(
attribute.String("payment.method", "credit_card"),
attribute.Int64("payment.amount_cents", amountCents),
),
)
defer span.End()
// Call the payment gateway
resp, err := paymentClient.Charge(ctx, cardToken, amountCents)
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return err
}
span.SetAttributes(attribute.String("payment.gateway_id", resp.TransactionID))
return nil
}Two things make this pattern effective. First, ctx is the first argument — the span is automatically a child of whatever span is active in the context, creating the correct parent-child relationship in the trace tree. Second, span.RecordError(err) plus span.SetStatus(codes.Error, ...) marks the span as failed, which most backends will highlight in red and use for error-rate dashboards.
Zero-Code Instrumentation with eBPF
If you have a large existing codebase and can't refactor every service, the OpenTelemetry Go Automatic Instrumentation project offers an alternative. It uses eBPF to attach instrumentation at the kernel level — no code changes required. You run the auto-instrumentation agent alongside your binary, and it intercepts Go runtime calls to libraries like net/http and database/sql to generate spans automatically.
This is still work in progress and doesn't cover every library, but it's useful for brownfield services where manual instrumentation isn't practical. The trade-off is less control over span attributes and naming — you get what the auto-instrumentation provides.
Production Checklist
Before shipping instrumentation to production, verify these things:
- Sampling rate is appropriate. At high traffic, 100% sampling creates massive data volume. Start at 10-50% and adjust based on backend capacity.
- Shutdown is wired correctly. On SIGTERM, call
tp.Shutdown(ctx)to flush buffered spans. Skipping this loses the last batch of telemetry during deploys. - OTLP exporter has retry enabled. The SDK retries failed exports with exponential backoff by default. Don't disable this unless you have a custom retry layer.
- Resource attributes are consistent. Use the semantic convention constants (
semconv.ServiceName, etc.) rather than raw strings. This ensures your service names match across traces and metrics. - Cardinality is bounded. Never use unbounded attributes like
user_idorrequest_pathon metrics. The SDK's 2000-series default limit will silently drop data beyond it.
Wrapping Up
OpenTelemetry in Go is production-ready for traces and metrics, with logging following close behind in beta. The combination of manual spans for business logic and auto-instrumentation for transport layers gives you end-to-end visibility without drowning in boilerplate. The key architectural decisions are choosing the right sampling rate, wiring graceful shutdown to avoid data loss during deploys, and keeping metric cardinality bounded.
Start with auto-instrumented HTTP handlers and a few custom spans around database and external API calls. That alone will transform how you debug production issues — from guessing at log timestamps to seeing the exact call tree with per-span latency. Add metrics for the core operations your service performs, and you'll have dashboards that catch degradations before users notice.
The Go getting started guide covers the fundamentals, and the SDK repository has examples for each signal. Once your pipeline is running, instrumenting a new handler is a three-line change.