Understanding Anomaly Detection
Anomaly detection is the practice of automatically identifying data points or patterns in telemetry, most commonly metrics, that deviate significantly from what is considered normal behavior for that system. It exists as an alternative and complement to static threshold alerting, which requires a human to decide in advance that, for example, error rate above 5 percent is bad. Static thresholds work well for signals with a clear, stable acceptable range, but they struggle with metrics that have strong daily or weekly seasonality, gradual growth trends, or behavior that varies significantly across services, none of which anomaly detection assumes in advance.
How Anomaly Detection Works
Most anomaly detection approaches fall into a few categories. Statistical methods, such as calculating a rolling mean and standard deviation and flagging values beyond a certain number of standard deviations (a z-score approach), are simple and interpretable but can produce false positives on data with irregular seasonality. Seasonality-aware methods, such as those used in Facebook’s Prophet library or built into commercial observability platforms, model expected daily and weekly patterns and flag deviations from that expected curve rather than from a flat baseline. More advanced machine learning approaches train models on historical telemetry to learn normal multivariate behavior across many metrics simultaneously, flagging combinations of signals that look unusual together even if no single metric crosses an obvious threshold.
A Concrete Example
Consider an e-commerce checkout service where traffic naturally follows a strong daily pattern, high during business hours, low overnight, with an even bigger surge every Friday evening. A static threshold alert set for “error rate above 2 percent” might be too sensitive during the Friday evening traffic surge, where a slightly elevated error rate is actually normal load-related noise, and too insensitive at 3 a.m., where even a small absolute increase in errors represents a proportionally large and meaningful problem. An anomaly detection model that has learned the service’s normal seasonal pattern can flag the 3 a.m. deviation as significant while correctly treating the Friday evening numbers as expected, catching a real regression that a static threshold would have missed entirely.
Where Anomaly Detection Fits in an Observability Stack
- Commercial platforms like Datadog, New Relic, and Dynatrace include built-in anomaly detection features that run against ingested metrics automatically.
- In open source stacks built on Prometheus and Grafana, teams implement anomaly detection using PromQL functions like holt_winters or predict_linear for simple forecasting-based alerting, or by exporting metrics to a dedicated analysis pipeline running a more sophisticated model.
- Anomaly detection is also applied to log volume and trace latency distributions, not just metrics, to catch unusual spikes in error log rates or shifts in the shape of a latency histogram.
Trade-offs and Limitations
- False positives remain a real challenge; anomaly detection models can flag legitimate but unusual events, such as a marketing-driven traffic spike, as anomalies, requiring tuning or human feedback loops to improve over time.
- Models need sufficient historical data to learn a system’s normal patterns, so anomaly detection is less effective for new services with little telemetry history.
- Unlike a static threshold, an anomaly score alone doesn’t always tell an engineer what “normal” would have looked like or why the deviation matters, so anomaly-based alerts often need to be paired with clear context to be actionable.
Best Practices
- Use anomaly detection to complement, not replace, threshold-based alerting on critical signals like error budgets and SLOs, where a clear, agreed-upon bar for acceptable behavior already exists.
- Start with anomaly detection on high-value, well-understood signals, such as overall request rate or checkout conversion, rather than applying it indiscriminately across every metric in the system.
- Feed detected anomalies into the same alert routing and on-call workflow as other alerts, through tools like Alertmanager, so they don’t become a separate, ignored notification channel.
- Continuously tune and review false positive rates, since an anomaly detection system that cries wolf too often will train engineers to ignore it, contributing to alert fatigue.
Frequently Asked Questions
What is Anomaly Detection?
Anomaly detection uses statistical or machine learning methods to automatically flag metrics, logs, or traces that deviate from expected patterns, without a human-defined static threshold. It catches issues that fixed thresholds miss, such as gradual drift or seasonal pattern breaks.
How does Anomaly Detection work?
Anomaly Detection works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Anomaly Detection matter?
Teams adopt Anomaly Detection to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Anomaly Detection?
Use Anomaly Detection when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
