Every latency dashboard has the same shape: a line for the average request duration and a line for p99, with the p99 line hovering well above the average and everyone nodding along. The average hides the tail, so percentiles are the right instinct. But there is a subtler problem that most dashboards never acknowledge: the p99 number on the screen is usually an estimate with an unquantified error bar, computed from bucketed counters under an assumption that is almost never stated. Depending on how the histogram was configured, the “real” p99 could be anywhere from right on the mark to double the displayed value.
This post goes through how Prometheus histograms actually store latency data, how histogram_quantile() turns those buckets back into a single number, and the specific ways that pipeline lies to you — plus what to do about each one.
What a classic histogram actually stores
A classic histogram in Prometheus is not a distribution. It is a small set of cumulative counters. Instrumenting a metric named http_request_duration_seconds exposes one counter per bucket plus two totals, per the metric type documentation:
http_request_duration_seconds_bucket{le="0.1"}— cumulative count of observations ≤ 0.1s..._bucket{le="0.5"}— cumulative count of observations ≤ 0.5s (including everything in smaller buckets)..._bucket{le="+Inf"}— every observation..._sum— the sum of all observed values..._count— the total number of observations (identical to the +Inf bucket)
The two properties that matter: buckets are cumulative (each includes all smaller buckets), and the individual observations are gone. Once an instrumented process reports le="0.5", there is no way to know whether those requests took 0.11s or 0.49s. Every percentile you will ever compute is reconstructed from these coarse counts.
The bucket boundaries are a configuration decision, and the default one is worth knowing. The Go client’s client library ships these buckets for histograms, designed for network service latencies in seconds:
.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10If your service answers in single-digit milliseconds, the defaults put your entire distribution into the first bucket and p99 becomes meaningless. If it answers in tens of seconds, everything lands in the top bucket. Neither problem raises an error — the dashboard just goes quiet and useless. Buckets must bracket the range where your latency actually lives, with fine resolution near the thresholds you alert on.
How the estimate is produced
The standard p99 query for a classic histogram is:
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[10m]))Three things happen inside that call, per the function documentation:
rate()converts the cumulative bucket counters into per-second observation rates over the window, which is why forgetting it silently produces garbage.- The function finds the bucket where the 99th percentile falls, using the cumulative counts.
- It linearly interpolates inside that bucket, assuming observations are uniformly distributed between the bucket boundaries.
The uniform-distribution assumption is where the error comes from. Real latency distributions are not uniform — they are typically heavy-tailed, with a dense mass near the fast end and a long sparse tail. If the p99 lands inside a bucket spanning 1s to 2.5s, the true value could be anywhere in that range; the interpolation just picks a linear midpoint-ish answer. The wider the bucket, the bigger the lie.
There are documented edge cases worth knowing. If the quantile falls into the highest bucket (the +Inf one), the function cannot interpolate against infinity, so it returns the upper bound of the second-highest bucket — a strictly understated p99. If the histogram has fewer than two buckets or no observations, it returns NaN. And note the interaction between window size and buckets: a five-minute window on a low-traffic endpoint can leave buckets with fractional counts so small that the percentile jitters wildly between refreshes.
The four ways dashboards lie
1. Averaging percentiles across instances
Percentiles do not average. The mean of ten instances’ p99s is not the fleet’s p99 — it is a number with no meaning at all. Yet this is exactly what happens when a dashboard aggregates with something like avg by (job) (histogram_quantile(...)) applied per-instance first. The correct pattern aggregates the bucket rates first and computes the quantile once, keeping the le label:
histogram_quantile(0.99,
sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))This works because bucket counts, unlike percentiles, are additive. Sum the counters, then take the quantile — never the reverse.
2. Wrong or missing rate()
Passing raw _bucket series without rate() feeds histogram_quantile cumulative-since-process-start counters, which produces a percentile of everything the process has ever observed — a number that moves glacially and tells you nothing about the last hour. Worse, a subtle variant: using rate() but with a window that doesn’t span multiple scrape intervals (e.g. [1m] with a 30s scrape interval gives only two data points), making the percentile noisy or erratic.
3. Coarse buckets at the tail
If your buckets jump from 1s straight to 5s, and your true p99 is 1.2s, the estimate interpolates within that 4-second span. If your true p99 is above 5s, you get the capped boundary value and the dashboard understates the problem — the worst failure mode, because it hides incidents. The fix is to place bucket boundaries densely around your SLO threshold. If you page at p99 > 300ms, you want several boundaries between 100ms and 500ms.
4. Treating sum/count as the median
The expression rate(_sum[5m]) / rate(_count[5m]) computes the mean latency, which is exact and — unlike any percentile — freely aggregatable across instances and dimensions. It is not the median, and for heavy-tailed distributions it can sit far above p50. Still, it deserves a place next to the percentile: it is the only exact aggregate you have, and the gap between mean and p99 is a direct read on tail severity. When the mean jumps but p99 doesn’t, something changed in the middle of the distribution; when p99 jumps but the mean doesn’t, it’s a small-but-slow subset.
Summaries: exact, but not aggregatable
Prometheus offers a second quantitative type, the summary, which exposes pre-computed quantiles directly as {quantile="0.99"} series, calculated client-side over a sliding window. Those numbers are exact — no bucket interpolation — which sounds strictly better until you try to aggregate. A summary’s quantile is computed inside one process; there is no way to combine ten instances’ summaries into a fleet-wide percentile, because Prometheus never sees the underlying distribution. Summaries also require you to fix the quantiles and window at instrumentation time.
The practical rule: histograms for anything that will ever be aggregated by job, route, region, or any other label — which in server-side systems is nearly everything. Summaries only for single-process scenarios where an exact client-side quantile genuinely matters, such as batch job runtimes.
Native histograms and where things are heading
Newer Prometheus versions support native histograms, where each sample is a composite structure carrying a dynamic set of buckets rather than a fixed grid of counters. They are substantially more efficient to store and query, adapt their resolution automatically, and remove the bucket-configuration problem entirely — no more choosing boundaries by hand. Client library support is still maturing and the format’s specification continues to evolve, so check your version’s docs before migrating. The histogram_quantile() call for native histograms drops the _bucket suffix and the le handling:
histogram_quantile(0.99, rate(http_request_duration_seconds[10m]))Even there, the quantile is still an interpolation within buckets — native histograms shrink the error bars dramatically but do not eliminate them. An exact percentile over all observations remains computationally out of reach for a metric system; the goal is buckets fine enough that nobody can tell the difference.
Making it survive in practice
Instrumentation is the place where bucket decisions get locked in, so it is worth doing deliberately. In Go:
package metrics
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var requestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request duration in seconds.",
// Dense below the SLO threshold (300ms), coarse above it.
Buckets: []float64{0.025, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3,
0.5, 1, 2.5, 5, 10},
}, []string{"method", "route"})
func ObserveRequest(method, route string, seconds float64) {
requestDuration.WithLabelValues(method, route).Observe(seconds)
}Beyond bucket design, three habits keep the numbers honest. Precompute expensive quantiles with recording rules so dashboards and alert rules read a cheap series instead of re-aggregating bucket rates on every render. Prefer alerting on the fraction of requests within your threshold — histogram_fraction(0, 0.3, rate(...[5m])) returns exactly that share, and it is what an SLO actually promises — rather than on a percentile estimate that sits on the wrong side of a bucket boundary. And periodically sanity-check a histogram’s buckets against observed traffic: a boundary set tuned for last year’s latency does no favors when the service got 10x faster or slower.
Percentiles from histograms are estimates, but they are good estimates when the buckets are designed for the distribution they measure. The dashboard lies only when nobody made that decision deliberately.