Prometheus Cardinality Explosions: Why Your Monitoring Server Ran Out of Memory

Every Prometheus outages post-mortem tells the same story in a slightly different costume. The server ran fine for months, then memory crept up, queries got slower, and one morning the TSDB head block ate the last of the RAM and the whole monitoring stack went down — usually during an incident, when you needed it most. The culprit is almost never a new version of Prometheus. It’s cardinality: the number of unique time series your server holds in memory, which grows multiplicatively with every label you add.

Cardinality failures are nasty because they’re non-linear. One label with ten values seems harmless. A second label with ten values seems equally harmless. But together they produce up to a hundred series, and a third takes you to a thousand. Teams routinely discover this only after shipping a feature that puts a user ID, request path, or tenant ID into a label. This post walks through what cardinality actually costs, where it typically explodes, and the concrete tools Prometheus gives you to detect and control it before it takes the server down.

What a “series” actually costs you

A time series is defined by a metric name plus every label name-value pair. http_requests_total with a status label of 200 is one series; the same metric with status="500" is another. Each active series holds an entry in the in-memory head block — the index structures alone are typically several kilobytes per series, independent of how many samples you actually scrape.

The samples themselves are cheap. The storage documentation notes that Prometheus stores an average of only 1-2 bytes per sample on disk, and gives a capacity formula: retention time in seconds multiplied by ingested samples per second, multiplied by bytes per sample. That formula tells you about disk. It says nothing about the head block’s index and chunk structures in RAM, which scale with series count, not sample count. This is why a server can handle millions of samples per second yet fall over at a few tens of millions of series.

The multiplication works like this. Take a metric with four labels: method (say 6 values), status (5 values), endpoint (40 values), and region (12 values). In the worst case you don’t have 6 + 5 + 40 + 12 values — you have 6 × 5 × 40 × 12 = 14,400 series per instance the metric is scraped from. Add a fifth high-cardinality label and you’re in the hundreds of thousands. Worse, the cross product rarely stays dense; the index has to accommodate every combination that ever appears, and churn (containers dying, pods rescheduling) keeps creating brand-new series that live on even after their source is gone, at least until retention cleanup catches up.

The usual suspects

Almost every cardinality incident traces back to one of a handful of label design mistakes:

Unbounded IDs as labels. User ID, session ID, order ID, trace ID. Each new request mints a new series. This is the single most common cause — the fix is to bucket or drop the identifier entirely and keep the detail in logs or traces, which are built for high-cardinality event data.
Raw URLs and paths. /users/12345/orders/678 as a label value means every distinct URL is a series. Use the templated route (/users/{id}/orders/{id}) instead.
Email addresses, hashes, error messages. Anything free-form belongs in structured logs. Error message strings as a label are particularly insidious because one new exception text quietly doubles your series count for that metric.
Identity information in instrumentation libraries. Some client libraries will happily attach pod-IP or hostname labels that churn constantly. Strip or normalize them at the source.
Churn, not just size. In Kubernetes, a deployment that restarts pods frequently with instance-specific labels creates new series on every reschedule. The series count grows even if traffic is flat.

A useful rule of thumb: labels should describe categories you’d actually want to aggregate by — region, endpoint pattern, status class, queue name. If you can’t name the set of values in advance, it probably shouldn’t be a label.

Detecting the problem before it pages you

Prometheus exposes its own internals as metrics, so you can watch cardinality the same way you watch anything else. The two queries worth putting on a dashboard are:

# Total active series in the head block
prometheus_tsdb_head_series

# Top 10 metrics by series count
topk(10, count by (__name__)({__name__=~".+"}))

The second query is expensive because it scans everything, so don’t run it every thirty seconds on a large server — run it on demand or at low frequency. What it shows is usually sobering: a handful of metrics are responsible for the vast majority of series. Fixing the top two offenders typically cuts total cardinality by an order of magnitude.

Also watch prometheus_tsdb_head_series_created_total. A steady positive rate when your workload hasn’t changed means something is minting new series continuously — classic churn. And alert on memory pressure: when the head block approaches its limits, query latency degrades long before the process is OOM-killed, so a slow climb is your early warning.

Control knobs: relabeling and limits

Once you know which metrics are the problem, you have three layers of defense, applied in order of preference.

First, fix the instrumentation. Changing a label set at the client library level is the cleanest fix, but it requires a deploy of every exporter and service that emits the metric. Often that’s exactly the right thing to do — the label never belonged there.

Second, relabel at ingestion. metric_relabel_configs runs after scraping and before storage, so it’s the right place to drop or rewrite labels on data you can’t change at the source. A common pattern is dropping a high-cardinality label entirely, or bucketing it:

scrape_configs:
  - job_name: my-app
    static_configs:
      - targets: ["my-app:8080"]
    metric_relabel_configs:
      # Drop the user_id label from request metrics.
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds'
        target_label: user_id
        replacement: ''
        action: replace

Note the ordering gotcha: metric_relabel_configs applies to samples, while the relabel_configs block (without the “metric” prefix) applies to targets before scraping. Dropping a whole metric is done with the drop action on __name__:

    metric_relabel_configs:
      # Stop storing a debug metric nobody charts.
      - source_labels: [__name__]
        regex: 'my_app_debug_internal_state'
        action: drop

Third, enforce hard limits. The scrape configuration supports per-scrape limits that fail the entire scrape when a target exceeds them: sample_limit, label_limit, label_name_length_limit, and label_value_length_limit. This sounds harsh — and it is, deliberately. A failed scrape with a visible up == 0 and an accompanying scrape_samples_scraped spike is far better than silently letting one rogue exporter OOM your server. Setting sample_limit to something like 200,000 for application targets forces every new exporter to either stay bounded or negotiate a higher limit consciously.

scrape_configs:
  - job_name: my-app
    sample_limit: 200000
    label_limit: 30
    static_configs:
      - targets: ["my-app:8080"]

Prometheus also tracks how close targets are to these limits with extra scrape metrics (scrape_sample_limit among them, enabled via extra_scrape_metrics), so you can alert before the limit actually trips rather than discovering it during a deploy.

Designing labels that scale

The naming and labeling best practices are worth internalizing because most cardinality problems are naming problems in disguise. A few habits that keep series counts predictable:

Base units only. Seconds, bytes — and put the unit in the metric name (http_request_duration_seconds, not latency_ms). This doesn’t affect cardinality directly, but consistent naming is what makes your top-k series query readable when you’re hunting for offenders.
One metric, one unit, one quantity. Don’t mix request size and duration in the same metric family with a “type” label; the value sets multiply and the queries get ugly.
Prefer buckets to raw values. A histogram with fixed bucket boundaries contributes a bounded, known number of series per label combination. A “duration” gauge per unique path does not.
Keep labels enumerable. Region, environment, endpoint pattern, status class — all have a value set you can list. If a teammate asks “what values can this label take?” and the answer is “it depends,” you’ve found tomorrow’s incident.

When you genuinely need per-entity analysis — which user is slow, which tenant is heavy — that’s a logs-and-traces question, not a metrics question. Metrics answer “how much, how often, aggregated”; the other signals answer “which one specifically.” Keeping that boundary clean is what lets each system do its job within its cost envelope.

When you outgrow a single server

At some point the honest answer is that one Prometheus box can’t hold your series count. The storage docs are blunt about the local TSDB being single-node: it is not clustered or replicated, and should be managed like any other single-node database. The scaling path is sharding (splitting targets across Prometheus servers by relabeling) plus the remote write API feeding a long-term store such as Thanos, Mimir, or Cortex. These layers change the economics — object storage instead of local disks, query federation across shards — but they don’t forgive unbounded cardinality; they just move the bill. A series that costs you RAM in Prometheus costs you query time and storage in the long-term layer too.

That’s why cardinality discipline comes first and architecture second. Sharding a cardinality problem multiplies the operational surface you have to manage. Trimming it shrinks every layer downstream.

A practical checklist

Dashboard prometheus_tsdb_head_series and alert on sustained growth.
Run a periodic top-k series query and know your top three offenders by name.
Audit every new metric for unbounded label values before it ships.
Set sample_limit per scrape job — pick a number, make violations loud.
Use metric_relabel_configs for exporters you can’t change, and file the fix upstream anyway.
Keep IDs, URLs, and error strings in logs and traces, not labels.

None of this is exotic. It’s the monitoring equivalent of input validation: cheap to do upfront, expensive to skip. The teams that never experience a cardinality outage aren’t luckier — they just treat label design as part of the API review instead of an implementation detail.