Every Prometheus failure I have debugged that was not a network problem eventually came down to the same thing: too many time series. Not a bad query, not undersized hardware — an explosion of unique label combinations that quietly multiplied until ingestion, memory, and query latency all fell over at once. Cardinality is the load-bearing concept in Prometheus, and it is the one most often discovered only after the outage.
This post covers how cardinality actually compounds, how to find which metrics are consuming your Prometheus, and the instrumentation patterns that keep series counts sane — with enough concrete PromQL and Go code to put the advice into practice the same day.
Cardinality is multiplicative, and that is the trap
Every unique combination of metric name plus label values is a distinct time series. Each one has to be scraped, parsed, held in the in-memory head block, indexed, compacted to disk, and considered by every query that touches the metric name. The number of series is not the sum of your label cardinalities — it is the product.
A quick worked example. One histogram on an HTTP handler:
- 3 HTTP methods that matter
- 8 route patterns
- 5 status-code buckets
- 12 buckets plus sum and count for the histogram itself
That is 3 × 8 × 5 × 12 = 1,440 series — from one metric. The trap is that growth in multiple dimensions compounds: add a fourth method, a ninth route, and a sixth machine, and you nearly double the count while each change looked trivial. Multiply this across every histogram and counter in a mid-size fleet and you are at millions of series before anyone notices. Ten million active series is roughly where a single Prometheus server starts sweating; unwise labels can eat that budget shockingly fast.
The classic offenders, in descending order of how often I have seen them take down a monitoring stack:
- User IDs, session IDs, request IDs as labels. Unbounded by definition. A label whose value set grows with traffic will eventually kill the server.
- Unnormalized URL paths.
/users/12345,/users/12346… each distinct URL is a new label value on your request counter. Always use route templates (/users/:id) instead. - Error messages.
error="connection refused"looks innocent until every unique downstream failure string becomes a series. - Container/pod names in application metrics. Already covered by Prometheus’s target labels — duplicating them per-metric doubles the cost.
- Over-bucketed histograms. Every extra bucket multiplies across every other label dimension.
One misconception worth killing early: moving a high-cardinality value into the metric name instead of a label does not help at all. A unique series is a unique series regardless of which part of its identity carries the explosion — it just makes the queries harder to write.
Finding the offenders before they find you
Prometheus exposes its own internals, so the diagnosis is a few queries against the /api/v1 endpoint or your Grafana instance. The most important metric is prometheus_tsdb_head_series — the number of active series in the head block:
prometheus_tsdb_head_seriesTrack that number’s trend the way you track disk usage. To find which metric names contribute the most series, count per name:
topk(
10,
count by (__name__) ({__name__=~".+"})
)This query is expensive on a large server — it walks the entire index — but running it occasionally (or from a scheduled rule) usually produces a top-10 list where a handful of metrics account for the majority of everything. Then check which label values are exploding on a suspect metric:
topk(
10,
count by (job, instance) ({__name__="http_request_duration_seconds"})
)And to see cardinality per label name across the board:
count by (__name__) (
{__name__=~".+"}
)If you want per-label cardinality analysis, the community standard is mixin dashboards plus the prometheus_tsdb_symbol_table_size and head-stats endpoints — but the per-__name__ count above answers 90% of “why is my Prometheus slow” questions.
Designing labels that scale
The naming and labeling guidance in the official docs is short, and every line of it is cardinality management in disguise. The rules that matter most in practice:
Keep label value sets bounded and small. A useful rule of thumb: be suspicious of any label that can exceed ~10 values, and audit any metric that can exceed ~100 series before it ships. Not a hard limit — but a metric at 10 today is plausibly at 50 next year, and histograms multiply the problem.
Use route templates, never raw paths. Most instrumentation frameworks make this easy once you know to look for it. In Go with the standard client library, measuring per-handler latency with a bounded route label looks like:
package main
import (
"net/http"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
var requestDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request latency in seconds.",
// DefBuckets if omitted: .005, .01, .025, .05, .1,
// .25, .5, 1, 2.5, 5, 10
Buckets: []float64{.01, .05, .1, .25, .5, 1, 2.5},
},
// Cardinality budget: 3 methods x ~15 routes x 7+2 buckets
// = bounded. NEVER add labels like path, user_id, or err.
[]string{"method", "route"},
)
func init() {
prometheus.MustRegister(requestDuration)
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/api/users", func(w http.ResponseWriter, r *http.Request) {
timer := prometheus.NewTimer(requestDuration.WithLabelValues(
r.Method, "/api/users",
))
defer timer.ObserveDuration()
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"ok":true}`))
})
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe(":8080", mux)
}Note what is deliberately absent: no status-code label (drop it unless you actually alert on it per-route), no error string, no query parameters. Every label you skip is a dimension that cannot explode later.
Separate the concern, don’t duplicate it. If you need per-customer visibility, metrics are usually the wrong tool — that is a logs or traces question. The pattern that works: keep the Prometheus histogram coarse (per-route), and export customer-scoped events to a log-based backend where cardinality is not structurally limited. When a per-customer latency question comes in, correlate via timestamps and route.
When you have already exploded
If the head series count is already in the danger zone, triage in this order:
- Fix the instrumentation first. Relabeling at the server is a band-aid; every scrape still pays the cost. Find the exporter or library emitting the offending labels and fix the source.
- Use metric relabeling to drop what you don’t use. A
metric_relabel_configsblock with alabeldroporkeepaction prevents known-bad label combinations from ever entering the TSDB. This is the correct tool for third-party exporters you cannot patch, not for your own services. - Check your recording rules. Aggregating a high-cardinality metric into low-cardinality recording rules lets dashboards read the pre-aggregated series, which shrinks query cost even if raw ingestion stays high. This reduces query pain, not ingestion cost — pair it with source fixes.
- Shard deliberately. Splitting scrape targets across multiple Prometheus servers (by function, team, or shard hash) keeps per-server head series manageable. The federation docs cover the standard pattern for rolling up aggregates upward.
There is no DROP METRIC in Prometheus. Series age out only after the retention window (default 15 days), so a cardinality fix takes effect on new series immediately but never un-sprawls the old ones — plan capacity accordingly and do not be surprised when memory does not drop the minute you redeploy.
A quick word on the rest of the observability stack
None of this means Prometheus is weak — it means metrics are a deliberately low-cardinality, low-latency abstraction, and the design holds up precisely because of that constraint. For high-dimension data, use logs (Loki, Elastic) or traces, which are built for per-event detail; the OpenTelemetry metrics specification leans on the same cardinality discipline when mapping onto Prometheus backends. The teams that run large Prometheus fleets successfully are the ones that treat the label schema like a database schema: reviewed, versioned, and changed on purpose.
Wrapping up
Three habits prevent most cardinality disasters: budget series per metric before shipping it, keep a top-N-by-series dashboard on prometheus_tsdb_head_series and the per-name count, and treat any new label like a new database index — a real cost that must earn its place. Monitoring that falls over during your worst incident is not monitoring; it is a second incident. Cardinality discipline is how you keep it from becoming one.