Understanding Observability as Code
Observability as code applies the same discipline that infrastructure as code brought to servers and networks, defining resources declaratively in version-controlled files instead of manually configuring them through a UI, to the observability layer itself: dashboards, alerting rules, SLO definitions, and even OpenTelemetry Collector pipelines. Instead of an engineer clicking through Grafana’s UI to build a dashboard or manually creating a Prometheus alerting rule on a server, these definitions live as YAML, JSON, or Jsonnet files in a Git repository, get reviewed through pull requests, and are deployed automatically through the same CI/CD pipeline used for application code and infrastructure.
Why Teams Adopt This Approach
Manually configured dashboards and alerts tend to drift over time: someone tweaks a threshold during an incident and forgets to document why, a dashboard panel references a metric that no longer exists after a refactor, or two similar services end up with subtly different alerting thresholds for no principled reason. Defining observability configuration as code brings the same benefits infrastructure as code brought to servers: changes are reviewable before they take effect, the full history of who changed what and why is preserved in Git, configurations can be reliably recreated after a disaster, and the same dashboard or alerting template can be reused consistently across dozens of services instead of manually rebuilt each time.
How It Works in Practice
- Dashboards are commonly defined using Grafonnet, a Jsonnet library for generating Grafana dashboard JSON programmatically, or through Terraform’s Grafana provider, which can create and update dashboards, data sources, and folders declaratively.
- Alerting rules for Prometheus are defined as YAML rule files, which are themselves just files in a Git repository, loaded by Prometheus and validated using tools like promtool as part of a CI pipeline before being deployed.
- SLOs are increasingly defined using dedicated tools and formats, such as OpenSLO or Sloth, which take a declarative SLO definition and generate the underlying Prometheus recording and alerting rules automatically.
- Collector pipelines, such as OpenTelemetry Collector configuration defining how telemetry is received, processed, and exported, are themselves YAML files that fit naturally into this same version-controlled workflow.
A Concrete Example
A platform team maintains a Git repository containing Jsonnet templates for a standard service dashboard, along with Terraform configuration that provisions this dashboard automatically for every new service registered in their internal developer platform. When a new microservice is created through the team’s self-service platform, a CI pipeline runs terraform apply against the Grafana provider, instantiating a consistent dashboard with golden-signal panels already wired to that service’s metrics, without any engineer manually building it. When the platform team later decides every dashboard should also show a panel for in-flight database connections, they update the shared template once, open a pull request, and the change rolls out to every service’s dashboard automatically once merged and deployed, rather than requiring someone to manually edit dozens of individual dashboards.
Trade-offs and Limitations
- There is a learning curve and initial setup cost to templating tools like Jsonnet or Terraform providers, which is harder to justify for a very small number of services.
- Rapid, exploratory changes during an active incident are sometimes faster to make directly in a UI, and teams need a clear policy for reconciling emergency UI changes back into version control afterward so configuration doesn’t silently drift.
- Not every observability platform has equally mature APIs or Terraform providers, so the practicality of full observability as code varies by which tools an organization has standardized on.
Best Practices
- Start with the highest-leverage, most duplicated artifacts, typically standard service dashboards and common alerting rule templates, rather than trying to convert every existing manual configuration at once.
- Validate configuration in CI before deployment, for example using promtool to lint Prometheus rules or a dry-run Terraform plan for dashboard changes, catching mistakes before they reach production.
- Establish a clear process for reconciling any emergency manual changes made during an incident back into the version-controlled source of truth afterward.
- Treat observability configuration with the same review rigor as application code, since a broken alerting rule or a dashboard querying the wrong metric can be just as costly as a bug in production code.
Frequently Asked Questions
What is Observability as Code?
Observability as code defines dashboards, alerting rules, and SLOs as version-controlled configuration rather than clicking through a UI, then deploys them through CI/CD. It applies infrastructure-as-code principles to tools like Grafana, Prometheus, and Datadog.
How does Observability as Code work?
Observability as Code works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Observability as Code matter?
Teams adopt Observability as Code to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Observability as Code?
Use Observability as Code when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
