Understanding Cardinality in Observability
Cardinality is a measure of how many distinct values, or combinations of values, exist for a given dimension in your telemetry data. In a metrics context, if a metric named http_requests_total has labels for method, status_code, and endpoint, the cardinality is the number of unique combinations of those label values that actually occur in production. A metric with three labels that each have ten possible values could produce up to a thousand unique time series. Add a label like user_id or request_id, each with millions of possible values, and the metric’s cardinality can explode into the millions or billions of series.
Why Cardinality Matters
Time series databases like Prometheus, Mimir, and Thanos store each unique combination of a metric name and its label set as a separate time series, each with its own chunk of samples over time. Every new unique label combination allocates new memory, disk, and index space. Because of this, cardinality is the single biggest driver of cost and performance problems in metrics-based observability. A single accidental high-cardinality label, such as attaching a raw user ID or a full URL path with query parameters to a metric, can multiply the number of active series by orders of magnitude and cause a Prometheus instance to run out of memory or become unqueryable.
A Concrete Example
Imagine an API gateway that emits a metric http_request_duration_seconds with a label named path. If the label captures the raw request path including path parameters, such as /users/8213/orders/55891, every unique user and order ID creates a brand new time series. What should have been one logical metric describing a single endpoint’s latency turns into millions of unrelated series. Queries that used to return instantly now scan enormous amounts of data, dashboards time out, and the underlying TSDB’s ingestion rate degrades. The fix is to normalize the label to a route template, such as /users/:id/orders/:id, which keeps cardinality bounded regardless of how many users or orders exist.
Where High Cardinality Belongs
- High-cardinality data like user IDs, request IDs, and session IDs belongs in logs and trace attributes, not in metric labels, because logging and tracing backends like Loki and Jaeger are designed to index and query high-cardinality fields efficiently.
- Exemplars, a feature in Prometheus and OpenTelemetry, let you attach a single high-cardinality trace ID to a metric data point without multiplying the number of series, giving you a bridge from an aggregate metric to an individual trace.
Detecting and Controlling Cardinality
- Use tools like Prometheus’s TSDB status page, or Grafana Mimir’s cardinality analysis API, to identify which metrics and labels contribute the most active series.
- Apply relabeling rules in the Prometheus scrape configuration or the OpenTelemetry Collector to drop or normalize high-cardinality labels before they are ingested.
- Set per-tenant or per-metric series limits in your metrics backend to catch cardinality explosions before they take down the whole system.
- Review new instrumentation in code review specifically for label choices, since cardinality problems are usually introduced by a well-meaning engineer adding a label for debugging convenience.
Trade-offs
Some cardinality is necessary and valuable: labels for environment, region, service name, and status code let you slice and dice dashboards meaningfully. The goal is not zero cardinality but bounded, intentional cardinality. Teams that are too conservative end up with metrics that cannot answer basic questions, while teams that are too permissive end up with unusable, expensive systems. Getting this balance right is one of the most common and costly mistakes in production observability deployments.
Frequently Asked Questions
What is Cardinality (in Observability)?
Cardinality in observability refers to the number of unique combinations of label or tag values attached to a metric, log line, or trace. High cardinality data, such as per-user or per-request identifiers on a metric, drives up storage cost and can overwhelm time series databases.
How does Cardinality (in Observability) work?
Cardinality (in Observability) works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Cardinality (in Observability) matter?
Teams adopt Cardinality (in Observability) to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Cardinality (in Observability)?
Use Cardinality (in Observability) when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
