Understanding Alert Fatigue
Alert fatigue happens when the volume or noise of monitoring alerts exceeds what a human can meaningfully triage, so responders start treating every page with less urgency, regardless of whether it’s a real emergency. The pattern is well documented outside software too, most notably in hospital settings where nurses become desensitized to constant equipment alarms, but it maps directly onto SRE and on-call work. A pager that goes off twenty times a night for issues that self-resolve within minutes trains the human on the other end to snooze first and investigate later, which is exactly the wrong reflex during the one alert in twenty that represents a genuine outage.
What Causes It
- Alerting on symptoms instead of causes, such as firing a page for every instance of high CPU rather than for actual user-facing latency or error-rate degradation.
- Static thresholds that don’t account for normal variance, for example alerting whenever CPU crosses 80 percent even though that’s expected during a known nightly batch job.
- Duplicate alerts from multiple layers, where a single root cause, like a downstream database outage, triggers alerts from ten dependent services simultaneously.
- Alerts with no clear action, sometimes called “FYI” alerts, that don’t require anyone to actually do anything right now but still page a human at 3 a.m.
- Flapping conditions, where a metric oscillates around a threshold and repeatedly fires and resolves the same alert within a short window.
A Concrete Scenario
An on-call engineer at a mid-size SaaS company gets paged 40 times over a single on-call week, most of them disk-space warnings on ephemeral worker nodes that get replaced automatically by the autoscaler within minutes anyway. By Thursday night, the engineer has started acknowledging pages on their phone and going back to sleep without opening a laptop, assuming it’s another disk warning. That night, a page fires for the same alert type, but this time it’s a genuine primary database volume approaching capacity, and the engineer’s learned habit of dismissing that alert type delays the response by three hours, during which the database briefly goes read-only during peak traffic. The postmortem traces the root cause not to the disk issue itself but to the alert fatigue that trained the engineer to distrust that alert category.
Why It Matters for Reliability
Alert fatigue directly undermines mean time to detect (MTTD) and mean time to respond, because the entire on-call system depends on humans trusting that a page means something worth waking up for. Google’s SRE book explicitly recommends that every page be actionable, urgent, and something only a human can resolve; anything else belongs in a dashboard or a lower-priority notification channel, not a page. Persistent alert fatigue also drives on-call burnout and attrition, since engineers who dread their on-call week because of noise, not genuine incident volume, are more likely to leave the rotation or the team entirely.
How Teams Reduce Alert Fatigue
- Audit paging alerts regularly and delete or downgrade any alert that hasn’t required action in the last quarter, or that consistently resolves itself without intervention.
- Alert on symptoms tied to user impact (the RED method: rate, errors, duration, or the four golden signals) rather than every internal cause.
- Use alert grouping and deduplication in tools like Alertmanager or PagerDuty so a single root cause produces one page, not ten.
- Set appropriate severity so only truly urgent, actionable conditions page a human; route informational signals to Slack, email, or a dashboard instead.
- Tune thresholds using historical data and percentile-based alerting rather than fixed static numbers that don’t reflect normal traffic patterns.
- Track a “pages per on-call shift” metric over time and treat a rising trend as a reliability problem worth fixing, not a fact of life.
Trade-offs
Reducing alert volume always carries the risk of cutting a signal that turns out to matter, so teams need to be deliberate rather than simply muting alerts wholesale. The safer path is investing in better alert quality, correlation, and routing rather than blanket suppression, and reviewing every alert change against the error budget and SLOs it was meant to protect.
Frequently Asked Questions
What is Alert Fatigue?
Alert fatigue is the desensitization that on-call engineers experience from too many low-value, noisy, or non-actionable alerts, causing them to ignore or slow down on notifications that might signal a genuine emergency.
How does Alert Fatigue work?
Alert Fatigue works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Alert Fatigue matter?
Teams adopt Alert Fatigue to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Alert Fatigue?
Use Alert Fatigue when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
