Choose metric labels deliberately
A metric label should be a dimension you aggregate or filter by, with a bounded set of values. This milestone gives you the rules for choosing metric labels and shows where cardinality mistakes usually enter a stack.
Use bounded labels
The single most important rule: label values must be bounded. A label like env has a handful of values forever. A label like user_id grows without limit, and every new value creates another active series that’s billed and indexed.
Never use these as metric label values:
- User, customer, or session IDs
- Request or trace IDs
- IP addresses
- Email addresses
- Timestamps
Each of these creates a new series per unique value, and the combinations multiply against every other label on the metric.
Questions to ask yourself before you add a label
For every label, ask:
- Will I aggregate or filter by it? If nobody queries by this dimension, it’s pure cost.
- Is the value set bounded? Think tens of values, not thousands.
- How does it multiply? Check the label against the labels already on the metric. Three bounded labels can still combine into thousands of series.
Where the mistakes happen
Cardinality problems usually enter a stack at ingestion setup, not in application code reviews. Enabling an integration with an app or service with default settings, such as the Kubernetes Monitoring Helm chart across all your clusters, can turn on thousands of metrics and labels at once. That visibility is valuable, but only if your budget expects it. When you set up a new integration:
- Review what metrics the defaults send before pushing them to production.
- Use allow lists to keep only the metrics and labels you’ll use.
- Check the Cardinality tab after enabling anything new, and watch for New badges.
Filter at the source
You can keep unnecessary series from ever leaving your infrastructure with relabeling rules in Prometheus, Alloy, or your collector:
- Allowlisting: keep an explicit set of important metrics and labels and drop everything else. For example, for Prometheus metrics, use
keepandlabelkeepactions. - Denylisting: drop an explicit set of high-cardinality metrics and labels and keep everything else. For example, for Prometheus metrics, use
dropandlabeldropactions.
For the full relabeling reference and examples, refer to Reduce metrics costs by filtering collected and forwarded metrics.
Enforce your strategy at a gateway
Relabeling rules only help if you control the collectors that apply them. In larger organizations, many teams send telemetry, and the team that pays the bill often can’t change every client. In that case, route telemetry through a tier of gateway collectors that your platform or admin team owns, and apply your labeling and filtering policies at that layer. Whatever upstream teams send, your rules run at the gateway before data reaches Grafana Cloud.
Fleet Management helps you run this pattern at scale by giving you centralized control over your collector configurations, so you can roll out and update gateway policies from one place instead of editing collectors one by one.
You now have the prevention rules. Bounded values only, add labels only when they will be used in queries, audit defaults at ingestion setup, and enforce policy at a layer you control. Next, you’ll fix cardinality you already have.
More to explore (optional)
At this point in your journey, you can explore the following topics: