Understanding an Alerting Rule
An alerting rule is a defined, automatically evaluated condition that determines when a system should notify a human that something needs attention. In the Prometheus ecosystem, an alerting rule consists of a PromQL expression that is evaluated on a regular interval, a duration the condition must remain true before firing (to avoid triggering on brief, self-resolving blips), and metadata like severity and a description that gets attached to the resulting notification. When the expression evaluates to true continuously for the configured duration, Prometheus considers the alert firing and sends it to Alertmanager, which handles deduplication, grouping, silencing, and routing the notification to the right destination, such as PagerDuty, Slack, or email.
Symptom-Based vs. Cause-Based Alerting
A key design decision in writing alerting rules is choosing what to alert on. Symptom-based alerting fires on user-facing impact, such as elevated error rate or latency breaching an SLO’s error budget burn rate, and is generally preferred because it directly reflects whether users are actually affected. Cause-based alerting fires on internal conditions, such as high CPU usage or a specific pod restarting, which may or may not correspond to actual user impact. Teams that alert primarily on causes tend to accumulate large numbers of low-value alerts, since many internal fluctuations are harmless, while teams that alert primarily on symptoms keep the alert volume focused on things that genuinely matter to users, deferring cause investigation to the debugging phase after an alert fires.
A Concrete Example
A common symptom-based alerting rule uses error budget burn rate: rather than a flat “alert if error rate exceeds 1 percent,” a burn-rate alert calculates how quickly a service’s SLO error budget is being consumed and fires with different urgency depending on the burn speed. A fast burn, for example one that would exhaust a full month’s error budget in a few hours, triggers a high-severity page immediately, while a slow burn that would only exhaust the budget over several days triggers a lower-severity, non-paging notification for the team to investigate during business hours. This pattern, popularized by Google’s SRE practice, avoids paging engineers for minor, short-lived blips while still catching genuinely serious degradations quickly.
How Alerting Rules Are Structured
- Expression: the query, such as a PromQL expression, that defines the condition being evaluated.
- For duration: how long the condition must remain true before the alert actually fires, filtering out transient noise.
- Labels: metadata like severity and team, used by Alertmanager to route and group the alert appropriately.
- Annotations: human-readable context, such as a summary and a link to a runbook, included in the notification to speed up the response.
Trade-offs and Common Pitfalls
- Alerts with no clear owner or unclear next steps contribute directly to alert fatigue, where engineers begin ignoring or muting notifications because too many are noisy or non-actionable.
- Setting the “for” duration too short causes flapping alerts that fire and resolve repeatedly on normal variance; setting it too long delays detection of genuine, fast-moving incidents.
- Alerting purely on infrastructure metrics without regard to actual user impact tends to produce a high volume of alerts that don’t correspond to anything customers notice.
Best Practices
- Prefer symptom-based alerting tied to SLOs and error budgets over alerting on every internal metric fluctuation.
- Attach a runbook link and clear description to every alert so the responder knows what to do immediately, without needing to reconstruct context under pressure.
- Route alerts by severity and team ownership through Alertmanager so the right person is paged, and low-urgency issues don’t wake someone up unnecessarily.
- Regularly review fired alerts, especially ones that were acknowledged but required no action, and tune or remove rules that aren’t providing real signal.
Frequently Asked Questions
What is Alerting Rule?
An alerting rule is a defined condition, typically a metrics query evaluated on a schedule, that triggers a notification when a system enters an undesired state. Rules are commonly written in PromQL, evaluated by Prometheus, and routed through Alertmanager.
How does Alerting Rule work?
Alerting Rule works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Alerting Rule matter?
Teams adopt Alerting Rule to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Alerting Rule?
Use Alerting Rule when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
