At 02:14 on a Sunday, an alert fires: payment success rate dropped four percent. You run kubectl logs across a few dozen pods and start grepping for “declined”. Every service formats failures differently — one prints plain sentences, another uses a custom prefix, a third logs the reason code but not the order it belongs to. Twenty minutes later you still cannot answer the only question that matters: which customers were affected?
This is the tax on unstructured logging. Free-form text is cheap to write and expensive to read, and the bill always arrives during an incident. Structured logging treats each log line as an event — a message plus a set of typed fields — so that a machine, not a human, does the searching. This post walks through doing it properly in Go: the standard library’s log/slog package, a handler strategy that keeps logging decisions in one place, a bridge into OpenTelemetry for trace correlation, and the operational pitfalls that bite teams after the first month.
The Problem With Text Logs
The standard library’s log package produces exactly what its design implies: a line of text with a timestamp. Anything you want to query later — order IDs, user IDs, latency, error kinds — is buried inside prose.
package main
import (
"log"
"os"
)
func main() {
requestID := "req-7f3a"
userID := 4821
log.Printf("request %s failed for user %d: timeout after 2s", requestID, userID)
logger := log.New(os.Stdout, "payments ", log.LstdFlags)
logger.Printf("payment declined order=ORD-99 amount=49.99 reason=insufficient_funds")
}Both lines describe failures, but neither is queryable. Extracting user 4821 from the first line requires a regular expression that breaks the moment someone rewords the message. The second line’s pseudo key-value pairs are only parseable if every service agrees on the same ad-hoc format — and they never do. Multiply this by every service you operate, and log search becomes archaeology.
slog: Structure in the Standard Library
Go 1.21 added log/slog to the standard library, and it settles the structured-logging question for most services. The core idea: a log call takes a message plus key-value pairs, and a handler decides how those pairs get serialized. The built-in JSONHandler emits one JSON object per event:
package main
import (
"log/slog"
"os"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
}))
slog.SetDefault(logger)
slog.Info("payment declined",
"order_id", "ORD-99",
"amount", 49.99,
"currency", "EUR",
"reason", "insufficient_funds",
)
}The same failure now lands as a single JSON document with a dedicated field per dimension. A log platform can index order_id as a field, and the query “all declines for reason insufficient_funds in the last hour” becomes a filter instead of a regex. Typed alternatives like slog.String("order_id", ...) and slog.Int(...) exist, but the plain key-value form is fine for everyday calls — the handler handles serialization.
One Logger, Many Handlers
The design decision that pays off over time is centralizing handler selection. Give every service a single constructor that decides the format, the level, and the base fields. Deployment-time behavior — “pretty text in development, JSON in staging, different verbosity per environment” — then changes in one file instead of at every call site.
package main
import (
"context"
"log/slog"
"os"
"strings"
)
func newLogger() *slog.Logger {
level := slog.LevelInfo
if strings.EqualFold(os.Getenv("LOG_LEVEL"), "debug") {
level = slog.LevelDebug
}
return slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: level,
}))
}
type requestKey struct{}
func handleRequest(ctx context.Context, base *slog.Logger, orderID string) {
logger := base.With(
slog.String("order_id", orderID),
slog.String("handler", "checkout"),
)
logger.InfoContext(ctx, "checkout started")
// ... do the work ...
logger.InfoContext(ctx, "checkout completed")
}
func main() {
logger := newLogger()
handleRequest(context.Background(), logger, "ORD-2041")
}Two details here carry most of the value. First, With() bakes the order_id into the derived logger, so every event the request path emits carries it automatically — you cannot forget it halfway through the handler. Second, prefer the Context variants (InfoContext, ErrorContext): the context is what later ties log events to the active trace, and retrofitting it across a codebase is tedious. Teams that skip this find, months later, that half their events correlate and half do not, with no visible difference at the call sites.
Writing JSON to stdout is a complete production setup. Whatever log collector your platform already runs — the Loki agent, Fluent Bit, a vendor agent — tails stdout, parses the JSON, and ships it. Some teams stop here and are right to.
Bridging to OpenTelemetry
The stdout approach has one structural gap: the log events are disconnected from your traces and metrics. Correlating them requires copying trace IDs into log fields by hand, and if a service forgets, its logs are orphaned. OpenTelemetry’s logs signal closes this gap — log records flow through the same pipeline as traces, carrying trace context and resource attributes automatically. The otelslog bridge implements slog.Handler, so your call sites do not change:
package main
import (
"context"
"log/slog"
"time"
"go.opentelemetry.io/contrib/bridges/otelslog"
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
"go.opentelemetry.io/otel/log/global"
"go.opentelemetry.io/otel/sdk/log"
"go.opentelemetry.io/otel/sdk/resource"
"go.opentelemetry.io/otel/semconv/v1.41.0"
)
func installLogger(ctx context.Context) (*slog.Logger, func(context.Context) error) {
res := resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceName("checkout-api"),
semconv.ServiceVersion("1.8.2"),
)
exporter, err := otlploghttp.New(ctx)
if err != nil {
panic(err)
}
provider := log.NewLoggerProvider(
log.WithResource(res),
log.WithProcessor(log.NewBatchProcessor(exporter)),
)
global.SetLoggerProvider(provider)
logger := slog.New(otelslog.NewHandler("checkout-api",
otelslog.WithLoggerProvider(provider),
))
shutdown := func(ctx context.Context) error { return provider.Shutdown(ctx) }
return logger, shutdown
}
func main() {
ctx := context.Background()
logger, shutdown := installLogger(ctx)
defer func() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
shutdown(ctx)
}()
logger.InfoContext(ctx, "order placed",
slog.String("order_id", "ORD-2041"),
slog.Float64("amount", 49.99),
)
}Each record now exports over OTLP with the service name and version attached as resource attributes — the fields your backend groups by — and with trace and span IDs filled in whenever the log call runs inside an active span. In a trace-aware UI, clicking a failed span shows the log events emitted during it, no manual correlation required.
Three operational notes that matter in production. The batch processor buffers records and flushes in bulk; a process that exits without Shutdown loses its last batch, so wire the shutdown into your service’s normal termination path — the five-second grace context above is deliberate. Short-lived processes (CLI tools, jobs) should use NewSimpleProcessor instead, which exports each record immediately. And the bridge costs more than printing JSON — conversion and batching are real work — though for nearly all services it is noise. If profiling ever shows logging as a genuine bottleneck, the documented alternative architecture is to keep stdout JSON with the plain JSONHandler and let the OpenTelemetry Collector convert on ingestion, injecting trace fields with a context-aware handler wrapper at the call site.
The Pitfalls: PII, Cardinality, and Cost
Structured fields make bad habits more dangerous, not less. A regex cannot accidentally exfiltrate a thousand customer emails; a copy-pasted slog.String("email", ...) can. Under regulations like GDPR, raw personal data in logs is a liability with a retention clock attached. The standard fix is pseudonymization at the boundary: a wrapping handler that rewrites sensitive attributes before any handler serializes them.
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"log/slog"
"os"
"regexp"
"strings"
)
var emailPattern = regexp.MustCompile(`^[^@\s]+@[^@\s]+\.[^@\s]+$`)
func hashEmail(email string) string {
if !emailPattern.MatchString(email) {
return ""
}
sum := sha256.Sum256([]byte(strings.ToLower(email)))
return hex.EncodeToString(sum[:8])
}
type privacyHandler struct {
slog.Handler
}
func (h privacyHandler) Handle(ctx context.Context, r slog.Record) error {
cleaned := slog.NewRecord(r.Time, r.Level, r.Message, r.PC)
r.Attrs(func(a slog.Attr) bool {
if a.Key == "email" {
a.Value = slog.StringValue(hashEmail(a.Value.String()))
}
cleaned.AddAttrs(a)
return true
})
return h.Handler.Handle(ctx, cleaned)
}
func main() {
base := slog.NewJSONHandler(os.Stdout, nil)
logger := slog.New(privacyHandler{Handler: base})
logger.Info("signup", slog.String("email", "jane@example.com"))
}The same customer identifier still appears in every event — pseudonymized consistently — so you can trace one user’s journey without storing who they are. Note the wrapper builds a fresh record: attributes arrive in the callback by value, so mutating the copy without re-adding it silently does nothing.
The second trap is unbounded cardinality. Every attribute on an event is a potential label in the log store, and fields like user_id, request_id, and order_id have unbounded value counts. On stores that index everything — ClickHouse, Elasticsearch, and most vendor platforms charge similarly — a field you log but never query is pure cost, and one high-cardinality field can dominate storage and query time. The rule that works in practice: log identifiers you need for join-correlation, and put diagnostic detail behind debug level or sample it, so a panic storm cannot also become a log bill. Deciding “what do we alert on, what do we grep for, what do we only want when something is already wrong” per field is ten minutes of design that saves real money later.
Wrapping Up
Structured logging is one of the few upgrades where the effort is front-loaded and the payoff compounds forever: slog’s JSON handler gives every event stable fields, a single constructor per service keeps handler policy in one place, and the otelslog bridge ties logs into the same trace context as the rest of your OpenTelemetry signals. Wrap a handler to scrub personal data before it reaches storage, treat high-cardinality fields as a cost decision rather than a default, and your incident self at 02:14 goes from grepping prose to filtering on reason = "insufficient_funds". If your service already emits traces, the OpenTelemetry logs bridge is a half-day change that makes every future incident cheaper to understand.