Traces are the most expensive observability signal you collect. A single request through a microservice stack can produce hundreds of spans across a dozen services, and at any real traffic volume, keeping every trace is a bill you pay twice: once in ingestion, again in storage. So everyone samples. And that is where most teams get burned — not by the cost of keeping traces, but by which traces they throw away.
Uniform head sampling — the default in most setups — decides at the top of the trace whether this request gets recorded at all, then propagates that decision downstream. It is cheap and consistent, but it is blind. Sample at 1% and the interesting traces live in the other 99%: the one request that hit a timeout cascade, the rare path that errors only under specific tenants, the slow endpoint nobody has profiled. Uniform sampling keeps 1% of those failures too — statistically faithful in aggregate, practically useless when you actually need to debug tonight’s incident.
Why the Decision Has to Come Later
The information you want to sample on — latency, errors, which downstream service fell over — only exists after the request finishes. Head sampling decides before any of it is known; that is the structural problem. Tail sampling flips it: every trace is kept in memory at the collection layer until it completes, and only then does something decide whether it is worth storing. Errors and slow requests get kept at 100%; the boring majority gets sampled down hard.
This is exactly what the tail sampling processor in the OpenTelemetry Collector does. It buffers spans grouped by trace ID, waits until the trace appears complete (or a timeout expires), evaluates your policies against the finished trace, and either forwards the whole thing or drops it. Sampling decisions at the SDK level can still exist alongside it — you typically run head sampling at 100% at the application and let the collector do the actual cutting.
A policy set that matches how most teams actually think about sampling looks like this:
processors:
tail_sampling:
decision_wait: 15s
num_traces: 50000
expected_new_traces_per_sec: 2000
decision_cache:
sampled_cache_size: 100000
non_sampled_cache_size: 100000
policies:
- name: keep-errors
type: status_code
status_code:
status_codes: [ERROR]
- name: keep-slow
type: latency
latency:
threshold_ms: 1000
- name: drop-health-checks
type: drop
drop:
drop_sub_policy:
- name: health-and-metrics-routes
type: string_attribute
string_attribute:
key: url.path
values: [/health, /metrics]
- name: baseline
type: probabilistic
probabilistic:
sampling_percentage: 5Read it as: keep every trace containing an error span, keep every trace slower than a second, throw away synthetic health-check traffic entirely, and keep 5% of everything else as a baseline. Policy evaluation is a vote — if any policy says keep, the trace is kept — and the drop policy exists specifically to carve out exceptions before other rules see them. Composite policies go further, allocating fixed span-per-second budgets across ordered sub-policies so a burst of slow traces cannot blow up your retention bill.
The Application Side: Give the Policies Something to Key On
Tail policies are only as good as the span data underneath them. Status codes and latency come for free from instrumented frameworks, but the high-value decisions usually key on attributes your code has to set deliberately. In Go, that means marking spans the collector should pay attention to:
package checkout
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
)
var tracer = otel.Tracer("checkout-service")
type Cart struct{ TotalCents int64 }
func HandleCheckout(ctx context.Context, cart Cart, customerTier string) error {
ctx, span := tracer.Start(ctx, "checkout",
trace.WithAttributes(
attribute.String("app.customer_tier", customerTier),
))
defer span.End()
if err := chargePaymentGateway(ctx, cart); err != nil {
// The tail processor's status_code policy sees this.
span.RecordError(err)
span.SetStatus(codes.Error, "payment gateway failed")
return err
}
return nil
}
// chargePaymentGateway is the outbound call to the payment provider.
func chargePaymentGateway(ctx context.Context, cart Cart) error {
return nil
}The call to SetStatus(codes.Error, ...) is what feeds the status_code policy; the tier attribute is what a string_attribute policy would filter on. Attributes set at span creation propagate into every policy evaluation the collector performs later — but only if the spans survive to the collector, which is why running head sampling at 100% in the SDK and cutting in the pipeline is the usual arrangement.
The Costs Nobody Mentions Until Production
Tail sampling moves the memory bill from your storage backend to your collection layer. Every in-flight trace is buffered in the collector until a decision is made, which means the collector’s memory footprint scales with your trace volume times decision_wait. The num_traces setting caps how many in-flight traces are tracked — exceed it and the oldest pending traces get evicted without a decision, silently. Set expected_new_traces_per_sec roughly to your real rate; it sizes internal structures, and being far off costs you memory efficiency or evictions.
decision_wait is the subtlest knob. A trace is considered complete when no new spans arrive for it — but the collector cannot know a span will never come, so it waits out this timer for stragglers. Too short, and long traces (slow requests, background jobs) get decided before their final spans arrive, splitting them across contradictory decisions. Too long, and you hold everything longer while burning memory. Start near your p99 request duration and tune from actual trace-duration data.
Two operational properties deserve respect. First, the processor is stateful: all spans of a trace must reach the same collector instance, or decisions get made on partial traces. Behind a load balancer that means routing by trace ID — the load-balancing exporter exists precisely for this — or running a single collector sized for the load. Second, decisions are cached: late spans that arrive after their trace was decided are handled according to the cached decision, which is what decision_cache sizes above. Skimp on those caches and you get partial traces leaking into the backend with random halves missing.
There is also a failure-mode inversion to plan for: when the collector itself is overloaded or restarts mid-decision, buffered traces are lost. That is acceptable for observability — the same outage is hitting everything else you would be debugging with — but it is worth knowing that the sampling layer is now a component that can silently discard data, and it deserves a memory alert like any other stateful service.
Making It Pay Off
The economics work out as a brutal compression ratio in your favor. A stack doing 5,000 requests per second with an error rate of a fraction of a percent and a p99 latency above a second for a few percent of traffic will keep roughly that few percent of traces — yet that slice contains essentially all of the debugging value. Uniform head sampling at the same retention volume keeps a random smear where failures are invisible. The tail policy set keeps the failures at full fidelity and spends its remaining budget as you choose.
Start simple: errors at 100%, slow traces at 100%, a low probabilistic baseline, a drop rule for synthetic traffic. Watch the collector’s memory for a few days, tune decision_wait against your real trace durations, and add attribute-based policies only when you find yourself searching for a trace you lost. Sampling rules are production code — version them, review changes, and treat a dropped trace class like a deleted log file.
The endgame is a pipeline where cost is a knob you turn deliberately: uniform fidelity for what matters, statistical coverage for what doesn’t, and a collector that makes the call with complete information. That is the difference between a tracing backend you can afford and one you can actually use. If you are evaluating backends, Jaeger pairs naturally with this setup, and the trace data model is defined by the W3C Trace Context standard that every vendor now speaks.