Every team starts logging with good intentions and ends up with a bill. Somewhere between the first incident and the third quarter of growth, the pipeline stops being a debugging tool and becomes one of the largest line items in the observability budget — and half of what it contains is either useless (the same “cache miss” line a million times) or dangerous (an access token logged “temporarily” two years ago).
Structured logging was supposed to fix this, and it does — but only if you treat it as a data engineering problem: schema first, correlation built in, volume actively managed, and secrets treated as a first-class threat. Here are those four layers using Go’s log/slog package, with code you can drop into a real service.
Schema discipline: every line is a record, not a sentence
The entire value of structured logging comes from one shift: instead of writing prose that humans grep, you emit records that machines query. A line like user 42 timed out after 3000ms is unqueryable. The same event as {"msg": "upstream timeout", "user_id": "42", "timeout_ms": 3000} can be filtered, grouped, and charted.
But ad-hoc structure is almost as bad as no structure. If one service logs user_id, another logs userId, and a third logs uid, every query needs three branches and someone always forgets one. The fix is a written schema — field names, types, casing, units — enforced in code. A small ReplaceAttr hook in slog’s JSON handler is a good enforcement point, because it touches every record before it leaves the process:
package main
import (
"log/slog"
"os"
"strings"
)
func main() {
handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
ReplaceAttr: func(groups []string, a slog.Attr) slog.Attr {
if a.Key == slog.LevelKey {
if level, ok := a.Value.Any().(slog.Level); ok {
a.Value = slog.StringValue(strings.ToLower(level.String()))
}
}
return a
},
})
logger := slog.New(handler)
logger.Info("order processed", "order_id", "ord-8291", "amount_cents", 4200)
}
Two conventions worth adopting from the start. First, suffix fields with their unit (duration_ms, amount_cents) — a bare duration field will eventually contain a mix of nanoseconds and milliseconds from different authors. Second, keep the set of top-level fields short and stable. Log aggregators index every distinct key, so every new field name across your fleet is a small permanent tax.
Correlation: put the request ID in the logger, not in every call
The question every incident investigation starts with is “what else happened during this request?” Answering it requires that every log line emitted while handling a request carries the same identifier. The failure mode is passing the ID manually to every log call — some call sites will forget, and the trail breaks exactly where the bug is.
The robust pattern in Go is request-scoped loggers: middleware builds one logger with the correlation fields attached and stores it in the request context. Nothing downstream needs to remember the ID, because it is already baked into every record:
package main
import (
"context"
"crypto/rand"
"encoding/hex"
"log/slog"
"net/http"
"os"
"time"
)
type ctxKey int
const loggerKey ctxKey = 0
func WithLogger(ctx context.Context, l *slog.Logger) context.Context {
return context.WithValue(ctx, loggerKey, l)
}
func LoggerFrom(ctx context.Context) *slog.Logger {
if l, ok := ctx.Value(loggerKey).(*slog.Logger); ok {
return l
}
return slog.Default()
}
func newRequestID() string {
b := make([]byte, 8)
_, _ = rand.Read(b)
return hex.EncodeToString(b)
}
type statusWriter struct {
http.ResponseWriter
status int
}
func (w *statusWriter) WriteHeader(code int) {
w.status = code
w.ResponseWriter.WriteHeader(code)
}
func LoggingMiddleware(base *slog.Logger) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
requestID := r.Header.Get("X-Request-Id")
if requestID == "" {
requestID = newRequestID()
}
logger := base.With("request_id", requestID, "method", r.Method, "path", r.URL.Path)
sw := &statusWriter{ResponseWriter: w, status: http.StatusOK}
next.ServeHTTP(sw, r.WithContext(WithLogger(r.Context(), logger)))
logger.Info("request completed",
"status", sw.status,
"duration_ms", time.Since(start).Milliseconds(),
)
})
}
}
func main() {
base := slog.New(slog.NewJSONHandler(os.Stdout, nil))
mux := http.NewServeMux()
mux.HandleFunc("/orders", func(w http.ResponseWriter, r *http.Request) {
logger := LoggerFrom(r.Context())
logger.Info("processing order", "order_id", "ord-8291")
w.WriteHeader(http.StatusOK)
})
_ = http.ListenAndServe(":8080", LoggingMiddleware(base)(mux))
}
When you adopt distributed tracing, the same slot carries the trace ID instead. The OpenTelemetry log data model attaches trace_id and span_id fields to every log record for exactly this purpose, and most log platforms can jump from a log line to the trace that produced it. Emitting those field names today buys you the correlation for free later.
Levels are a cost decision, not a mood
Most teams use levels emotionally: Debug for things they might want, Info for things that happened, Error for things that are embarrassing. In production that translates to gigabytes of Info nobody reads and Debug noise that buries the three Error lines per day that matter.
A workable contract: Error means a human should look (alerts may fire on it), Warn means the system recovered but the cause should be counted, Info means an operation completed — and each Info event must pay its way by being queryable and useful, not merely reassuring. Debug stays off in production and is turned on per-service during an incident, ideally without a redeploy — slog evaluates Enabled per record, so flipping the handler level at runtime is cheap.
The remaining volume problem is high-frequency events: cache misses, pool waits, retried calls. Dropping them entirely loses the signal you need during an incident, so the middle path is sampling — keep the first occurrence of each distinct message and then one in N, with a counter attached so you can still tell “1 event” from “1 of 84,113 events”. That is a small wrapper around slog.Handler:
package main
import (
"context"
"log/slog"
"os"
"sync"
)
type sampledHandler struct {
next slog.Handler
seen map[string]int
mu *sync.Mutex
every int
}
func newSampledHandler(next slog.Handler, every int) *sampledHandler {
return &sampledHandler{
next: next,
seen: make(map[string]int),
mu: &sync.Mutex{},
every: every,
}
}
func (h *sampledHandler) Enabled(ctx context.Context, level slog.Level) bool {
return h.next.Enabled(ctx, level)
}
func (h *sampledHandler) Handle(ctx context.Context, r slog.Record) error {
h.mu.Lock()
h.seen[r.Message]++
count := h.seen[r.Message]
h.mu.Unlock()
if count%h.every == 1 {
rec := r.Clone()
rec.Add(slog.Int("occurrences", count))
return h.next.Handle(ctx, rec)
}
return nil
}
func (h *sampledHandler) WithAttrs(attrs []slog.Attr) slog.Handler {
return &sampledHandler{next: h.next.WithAttrs(attrs), seen: h.seen, mu: h.mu, every: h.every}
}
func (h *sampledHandler) WithGroup(name string) slog.Handler {
return &sampledHandler{next: h.next.WithGroup(name), seen: h.seen, mu: h.mu, every: h.every}
}
func main() {
var handler slog.Handler = slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelDebug})
handler = newSampledHandler(handler, 100)
logger := slog.New(handler)
for i := 0; i < 250; i++ {
logger.Debug("cache miss", "key", "user:42")
}
}
Once the sampled handler is in place, watch one thing: the seen map grows without bound in this sketch. That is fine for a bounded set of message strings; if your messages embed variable data, move the variable part into attributes so the message stays constant — it makes sampling work and makes aggregation cheaper, since log platforms group by message text.
Redaction: logs are a breach surface
The OWASP Logging Cheat Sheet puts it plainly: authentication data, session identifiers, and personal data do not belong in logs. Yet credentials end up in logs constantly — through error messages that embed request bodies, debug lines that dump objects, an authorization header logged “to trace the API key that failed”. Once a token reaches the log store, it outlives the session: log retention routinely keeps records for months, far longer than any access token’s lifetime.
Defense in depth, in order of reliability: never log the request body wholesale (log its identifiers instead); scrub known-dangerous keys at the handler level so a stray "password" attribute becomes "[REDACTED]"; and treat new log statements near authentication code as review flags. The scrubber is the same ReplaceAttr pattern as the level normalizer from earlier — check the key against a deny-list and replace the value — and because it sits in the handler, it catches records from every call site, including the ones that forgot.
One more often-missed rule: log authentication events (login success and failure, permission denied, token refresh) as first-class records with stable names. They are the events you will need for an audit, and retro-fitting them after an incident is a bad position to be in.
The bill: retention, tiers, and labels
The last layer is economic. Log cost is roughly lines × bytes × retention, and you control all three. Retention tiers do most of the work: hot storage for the last week (when real incidents happen), cheap object storage for the tail that only compliance cares about. Sampling attacks lines. Field hygiene attacks bytes — the label cardinality guidance for Loki applies to most aggregators: low-cardinality fields belong in indexed labels, high-cardinality values like user IDs belong in the log body, or index cost and query latency both explode.
Finally, measure the pipeline itself. A log line with no reader in 90 days is a strong candidate for deletion — the cheapest log is the one you decided not to write.
Wrapping up
Structured logging pays off only when it is run like a data platform: a schema everyone shares, correlation fields injected by middleware rather than by memory, levels treated as a volume budget, secrets scrubbed at the handler boundary, and retention reviewed like any other spend. The log/slog building blocks — ReplaceAttr, context-scoped loggers, composable Handler wrappers — cover all four layers in an afternoon. Start with the request-ID middleware; it is the smallest change with the biggest incident-night payoff.