Every Prometheus failure I’ve seen in production started the same way: someone added a label. A user ID on a request counter, a pod name that includes a deployment hash, a full URL on an error metric. Each one seemed harmless. Two weeks later, Prometheus is consuming 30 GB of RAM, scrapes are timing out, and queries that used to return in 200 ms are timing out the Grafana tab.
The root cause is almost always cardinality: the number of unique time series your metrics produce. Prometheus keeps every active series in memory (the “head block”), and its cost scales with series count, not sample count. Two labels with 10 values each produce up to 100 series; swap one for a user ID with a million values and you have a database that cannot survive a restart. This post covers how cardinality explodes, how to measure it, and the five controls that keep it bounded.
Why cardinality multiplies instead of adds
The most common mental model error is treating labels as independent costs. They aren’t. Every unique combination of label values is its own series, so cardinality is a Cartesian product. A histogram with 12 buckets and 4 quantile summary series, labeled by endpoint, multiplies the endpoint count by roughly 16, not by 1.
Concretely, compare these two counters:
// Bounded: method has ~6 values, status has ~5.
// Worst case: 30 series. Realistic: fewer, most methods never 501.
httpRequests := prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "http_requests_total"},
[]string{"method", "status"},
)
// Unbounded: path includes IDs, query strings, everything.
// Worst case: one series per distinct URL ever served.
httpRequests := prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "http_requests_total"},
[]string{"method", "status", "path"},
)The second version is the classic production killer. Instrumentation libraries make this easy to do by accident: request loggers and tracing middleware happily attach path or route to a counter without checking whether the value is bounded. REST path parameters (/users/4821/orders), UUIDs, and cache-busting query strings each become a unique label value. An e-commerce site with a few million product pages can turn a single counter into millions of series in an afternoon.
Why is this worse than “more data”? Prometheus is an in-memory time series database by design: the head block holds every active series in RAM, and every query must consider all series matching its selectors. Rapid series churn — restarted pods constantly minting new series — forces the index and query engine to do more work per scrape.
Measuring what you actually have
Before changing anything, find out which metrics dominate. Prometheus exposes its own internals, and three queries answer most of the question.
Total active series, the single number that determines server health:
prometheus_tsdb_head_seriesTop 10 metrics by series count, which usually reveals one or two offenders responsible for most of your cardinality:
topk(10, count by (__name__)({__name__=~".+"}))Series churn rate — how fast new series are being created, which catches the “label contains a timestamp” class of bugs:
rate(prometheus_tsdb_head_series_created_total[5m])The TSDB stats API endpoint (/api/v1/status/tsdb) gives the same information as a table: series count per metric name and the label pairs contributing the most series. The Grafana-built Prometheus web UI also surfaces this under Status → TSDB. Run this before any optimization; the top offender is almost never the metric you expect.
Fix 1: Bound label values at the instrumentation layer
The cheapest fix is the one applied where the metric is defined. Any label value that comes from user input, a request path, or an external system must be normalized or bucketed before it reaches a label. The pattern is a small allowlist or normalizer:
var knownRoutes = []string{
"/users", "/users/{id}", "/orders", "/orders/{id}", "/healthz",
}
func normalizeRoute(path string) string {
for _, route := range knownRoutes {
if routeMatches(route, path) {
return route
}
}
return "unmatched"
}
// Label value is now bounded: at most len(knownRoutes)+1 distinct values.
routeLabel := normalizeRoute(r.URL.Path)Some frameworks hand you the route pattern for free: Go 1.22+’s ServeMux exposes the registered pattern as r.Pattern, so wiring it into a middleware is a few lines. Frameworks without pattern-aware routing need the normalizer above — it’s 15 lines of code and prevents the single most common cardinality incident.
Values that genuinely have many distinct values but don’t need per-value breakdowns (user ID, tenant, email domain) belong in logs or traces, not metric labels. That’s what the three signals are for: metrics answer “how much, how often,” and the identity of which specific user sits in the trace you can find by time and endpoint.
Fix 2: Enforce scrape limits before the explosion
Instrumentation discipline fails eventually — a new service ships with a bad label, and one restart later your server is drowning. Prometheus has per-scrape limits that act as a circuit breaker:
scrape_configs:
- job_name: app
sample_limit: 20000 # max samples per scrape
label_limit: 30 # max labels per series
label_name_length_limit: 128
label_value_length_limit: 512
static_configs:
- targets: ["app:9090"]When a target exceeds sample_limit, Prometheus rejects the entire scrape for that target and increments prometheus_target_scrapes_exceeded_sample_limit_total. This is deliberate: a partially ingested scrape corrupts counter math (rate calculations need complete sequences), so the all-or-nothing behavior is the correct design. The trade-off is that a misbehaving service loses all its metrics — which is still better than the whole server degrading.
Set a global default and tune per job, and alert on the rejection counter so you find out before users do:
increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0Fix 3: Drop and rewrite labels with metric relabeling
When you can’t change the application — a third-party exporter, a legacy service — metric_relabel_configs runs on every sample after the scrape and can drop labels, rewrite them, or discard entire metrics:
metric_relabel_configs:
# Drop a known-harmful label entirely.
- action: labeldrop
regex: "(user_id|session_id|request_id)"
# Collapse Pod template hashes into a stable workload name.
- source_labels: [pod]
regex: "(.+)-[a-z0-9]+-[a-z0-9]{5}"
target_label: workload
replacement: "$1"
# Discard a debug metric family outright.
- source_labels: [__name__]
regex: "debug_.*"
action: dropTwo things matter here. First, ordering: relabel_configs applies to targets before scraping, while metric_relabel_configs applies to samples after — filtering samples in the former does nothing. Second, cost: metric relabeling runs per sample per scrape, so a heavy regex on a job exposing 100k samples burns measurable CPU on the Prometheus server. Keep the regexes simple and specific.
Fix 4: Aggregate with recording rules
Some cardinality is legitimately useful but rarely queried at full resolution. Dashboards almost always aggregate away the detail anyway — so precompute the aggregate, and drop the underlying detail. Recording rules do this inside Prometheus:
groups:
- name: cardinality_reduction
interval: 1m
rules:
- record: job:http_requests:rate5m
expr: |
sum by (job, method, status) (
rate(http_requests_total[5m])
)Now dashboards and alerts query job:http_requests:rate5m (a handful of series) instead of the raw metric (potentially thousands). The catch: recording rules don’t reduce the ingested cardinality by themselves — the raw series still land in the head block and consume RAM. To actually cut series, combine the rule with a relabel drop of the source metric, or configure the retention so detail metrics age out quickly. The rule preserves the queries; the drop reclaims the memory.
Fix 5: Alert on cardinality itself
Cardinality incidents are silent until queries get slow, so make the growth visible. Two alerts cover most cases: an absolute ceiling on total series, and a churn alert that catches runaway series creation before it hits the ceiling.
groups:
- name: cardinality
rules:
- alert: HighTotalCardinality
expr: prometheus_tsdb_head_series > 1000000
for: 15m
labels:
severity: warning
annotations:
summary: "Active series above 1M"
- alert: RapidCardinalityGrowth
expr: rate(prometheus_tsdb_head_series_created_total[1h]) > 500
for: 30m
labels:
severity: warning
annotations:
summary: "Series being created faster than 500/s"Size the threshold to your environment: a single Prometheus handling a mid-size fleet typically lives comfortably under a few million series; beyond roughly 5–10 million, RAM usage and query latency degrade noticeably on commodity hardware. If you legitimately need more, that’s the point to look at sharding (functional partitioning by service or team, with each Prometheus owning a slice) or object-storage-backed long-term stores like Thanos and Grafana Mimir, which exist precisely because the single-node model has a cardinality ceiling.
A checklist that prevents the next incident
- Run
topk(10, count by (__name__)({__name__=~".+"}))monthly and know your top offenders. - Never label by unbounded values: IDs, URLs, emails, raw paths. Normalize to route patterns or buckets.
- Set
sample_limiton every scrape job, and alert on the exceeded-limit counter. - Move per-entity identity questions from metrics to logs and traces.
- Pre-aggregate high-detail metrics with recording rules before they feed dashboards.
The theme across all of these is the same: cardinality is a design decision made at instrumentation time, and every control after that point is damage limitation. A metric schema reviewed like an API contract — bounded values, documented labels, a cost estimate per series — costs an hour and saves the 2 a.m. page when Prometheus OOMs. Your future self, staring at a Grafana tab that finally loads, will appreciate it.