Eventually the alert fires at 3 a.m.: your Prometheus is consuming more memory than the database it monitors, queries that took milliseconds are timing out, and the scrape of your biggest service now takes longer than the scrape interval. Almost always, the root cause is not a bug in Prometheus. It is cardinality — the product of every label value combination across your metrics — growing silently until the time series database drowns.
Cardinality is the one failure mode of metrics-based monitoring that is nearly impossible to fix retroactively. You cannot re-label history cheaply, downsampled data has already lost the dimensions you later discover you need, and by the time queries degrade, the expensive series have usually been accumulating for months. This post covers how cardinality explodes, how to find the offending metrics before they hurt, and the levers — label design, relabeling, recording rules, and limits — that keep a Prometheus deployment healthy for years.
Why cardinality explodes
Every unique combination of metric name plus label values is a distinct time series in the TSDB. The math is multiplicative, and that is the trap. One HTTP counter labeled by method (5 values), status_code (6 values), and endpoint (50 routes) is not 61 series — it is up to 1,500. Add a single poorly chosen dimension like user_id or trace_id with a million distinct values, and that one counter alone produces a million active series, each with its own chunk of memory, its own index entry, and its own write amplification on every scrape.
The classic offenders show up in every stack eventually:
- Unbounded IDs as labels — user IDs, request IDs, order numbers, container image digests, git SHAs. Any label whose value set grows with business activity is unbounded by definition.
- Raw URLs or routes — labeling by
/users/12345/orders/678instead of the route pattern/users/{id}/orders/{id}turns one metric into one series per URL ever requested. - Cardinality leaks from upstream systems — the Kubernetes SD label set, or a vendor exporter that happily labels every metric with pod names, IPs, and versions.
- Error messages and enum-ish strings — a label intended to hold a handful of status values that quietly collects free-text variants.
What makes this insidious is the cost asymmetry: an unbounded label looks identical to a bounded one in code review. endpoint="/checkout" and endpoint="/checkout?uid=84920" are one refactor apart, and nothing fails until weeks later.
Finding the offenders before they hurt
Prometheus exposes its own internals, which makes cardinality auditing a query, not an archaeology project. The two most useful views are per-metric series counts and per-label-value-pair counts:
# Top 20 metrics by number of active series
topk(20, count by (__name__)({__name__!=""}))
# Series contributed by each label value pair, across all metrics
topk(20, count by (__name__, job)({__name__!=""}))
# How many series a single metric's label dimension produces
count by (endpoint) (http_requests_total)For a deeper breakdown — including the label pairs that dominate across your entire fleet — the TSDB status endpoint is the authoritative source. /api/v1/status/tsdb returns seriesCountByMetricName, seriesCountByLabelValuePair, and labelValueCountByLabelName, which together usually identify the culprit in under a minute. Run this monthly; new services and new exporter versions are the usual reintroduction vector.
As a rough budget, plan on a few GB of RAM per million active series for the head block, plus headroom for queries. The precise number depends on version and workload, which is exactly why you should measure your own series-to-memory ratio from prometheus_tsdb_head_series versus container memory, then alert on deviations from your own baseline.
Fix one: label design at the source
The cheapest series are the ones never created. When instrumenting, apply a simple rule: a label belongs on a metric only if you will ever filter or group by it in a dashboard or alert, and its value set is bounded by design. Normalization belongs in the instrumentation layer — map raw values onto bounded sets before they ever reach a label:
package metrics
import (
"net/http"
"strings"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var httpRequests = promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Requests by route pattern, method, and status class.",
},
// route: bounded set from the router's registered patterns.
// status_class: 2xx/3xx/4xx/5xx, NOT the raw code.
[]string{"route", "method", "status_class"},
)
func statusClass(code int) string {
switch {
case code >= 500:
return "5xx"
case code >= 400:
return "4xx"
case code >= 300:
return "3xx"
default:
return "2xx"
}
}
// routePattern collapses concrete paths to their registered
// pattern: /users/42/orders/7 -> /users/{id}/orders/{id}.
// Recording the raw path here would be unbounded cardinality.
func routePattern(r *http.Request) string {
return strings.ReplaceAll(r.URL.Path, "/42/", "/{id}/")
}
func ObserveRequest(r *http.Request, code int) {
httpRequests.WithLabelValues(
routePattern(r), r.Method, statusClass(code),
).Inc()
}The same discipline applies in client libraries generally: most offer ways to enforce label discipline, and the long-standing client_golang discussion on cardinality guards tracks the request for first-class enforcement. Until your library supports hard caps, the normalization function above — collapse IDs, bucket codes, canonicalize routes — is the pattern to standardize on.
Fix two: relabeling as a safety net
Instrumentation discipline takes time to propagate, and third-party exporters do not care about your memory budget. Scrape-time relabeling is the enforcement layer. Two mechanisms matter: metric_relabel_configs drop or rewrite series after the scrape, and per-target sample_limit caps how many samples a single target may expose at all.
scrape_configs:
- job_name: payments
# Hard cap per target: if a service suddenly exposes a
# million series, the scrape fails loudly instead of
# silently OOM-ing Prometheus.
sample_limit: 50000
static_configs:
- targets: ["payments:8080"]
metric_relabel_configs:
# Drop the unbounded debug metric entirely.
- source_labels: [__name__]
regex: "debug_.+"
action: drop
# Strip the session ID label wherever it appears.
- regex: "session_id"
action: labeldropTwo cautions. First, metric_relabel_configs runs after the target is scraped, so dropped samples still cost scrape bandwidth — they just stop costing TSDB memory, which is the resource that actually matters here. Second, a target that exceeds sample_limit has its entire scrape rejected, not just the excess — which is why the limit should be set generously above current usage and paired with an alert on prometheus_target_scrapes_exceeded_sample_limit_total. The point of the limit is to convert a silent cardinality leak into a page you can act on.
Fix three: recording rules and aggregation boundaries
Many high-cardinality metrics exist because someone wants per-route or per-region detail. That detail does not have to live as raw series forever. Recording rules precompute aggregates, and the raw detail can then be retired or given a shorter retention:
groups:
- name: http_aggregates
interval: 30s
rules:
- record: job:http_requests:rate5m
expr: sum by (job, method, status_class) (
rate(http_requests_total[5m]))
- record: route:http_requests:rate5m
expr: sum by (route) (
rate(http_requests_total[5m]))The level:static naming pattern (aggregation level first, then metric, then window) is worth adopting fleet-wide: it makes the granularity of every recorded series explicit at query time and prevents the common mistake of grafana panels querying raw high-cardinality series when a recorded aggregate exists. Dashboards that only need fleet-wide answers should only ever read the aggregates.
Alerting on cardinality itself
Cardinality management fails quietly unless the monitoring system monitors itself. Three expressions cover the main failure modes:
# Total active series approaching your capacity budget
prometheus_tsdb_head_series > 3000000
# Rapid series creation — the signature of a bad deploy
# or an exporter gone wrong
rate(prometheus_tsdb_head_series_created_total[1h]) > 1000
# A target hit its sample limit (scrape rejected)
increase(prometheus_target_scrapes_exceeded_sample_limit_total[1h]) > 0The second alert is the most valuable in practice: series creation rate spikes within minutes of a bad deploy, long before memory pressure becomes visible. Wire it to fire on sustained growth and route it to whoever owns the deploy — it is almost always someone’s new label.
A note on the ecosystem around the limits
When a single Prometheus legitimately cannot hold the series you need — multi-tenant platforms and per-customer SLA tracking are the usual cases — the options are sharding by functional or horizontal split, or systems designed for high-cardinality storage. Loki moves label indexing concerns out of metrics entirely for logs, and object-storage-backed projects like Thanos and Cortex (and its successor Mimir) address scale-out and per-tenant limits. These solve capacity, but they do not exempt you from label discipline — they move the cost curve, they do not remove it. The cheapest series remain the ones never created, wherever they are stored.
Start with the audit query, wire up the series-creation alert, and add a sample limit to your biggest scraper this week. Cardinality problems are cheap to prevent and brutal to retrofit — and unlike most reliability work, the preventive steps take an afternoon.