Monitoring as Code: Why Your Monitors Should Live in Git

Your infrastructure is defined in Terraform. Your application deploys through CI/CD. Your database schema migrates through version-controlled files. But your monitors — the thing that tells you everything else is working — live in a vendor's web UI, maintained by whoever clicked the buttons last, with no history, no review process, and no way to recreate them if the vendor loses the configuration.

Monitoring as code is the practice of defining monitors, alert channels, notification policies, escalation policies, and status pages in version-controlled configuration files that deploy through the same pipeline as your application. When a developer changes the checkout flow, the monitor that watches the checkout flow is updated in the same pull request — not three days later when someone remembers to update the dashboard.

Why monitors rot in a web UI

Every team that maintains monitors in a vendor dashboard eventually hits the same failure modes:

Configuration drift. Two engineers edit the same monitor at the same time. One changes the threshold, the other changes the URL. The vendor keeps the last write. Nobody knows what the "correct" state is.

Orphaned monitors. A service is decommissioned. Its monitors keep running, alerting on expected failures, training the team to ignore the alert channel. Six months later a real failure in that channel goes unnoticed because everyone learned to tune it out.

Undocumented changes. Someone increased the alert threshold from 500ms to 5000ms during an incident "temporarily." There is no record of who changed it, when, or why. The threshold never goes back.

Unreviewed configuration. A new engineer creates a monitor that alerts the entire team on every 404 response. No peer review caught it because there is no review process for monitor changes in a web UI.

Disaster recovery failure. The vendor has an outage. Or you switch vendors. Your monitoring configuration — dozens of monitors, alert channels, notification policies — cannot be recreated because it was never captured in a reproducible format.

What monitoring as code looks like in practice

Monitoring as code means your repository contains declarative files that define your monitoring setup. The format varies by tooling — YAML, HCL, JSON, TypeScript — but the principle is the same: the repository is the source of truth, and the running state is derived from it.

# monitors/checkout-api.yaml
name: Checkout API Health
type: http
url: https://api.example.com/v1/checkout/health
method: GET
frequency: 30s
regions:
  - us-east
  - eu-west
  - ap-southeast
assertions:
  - type: status_code
    value: 200
  - type: response_time
    threshold: 1000ms
  - type: json_body
    path: $.status
    value: "healthy"
alerts:
  - channel: pagerduty-checkout-team
    severity: critical
  - channel: slack-engineering
    severity: warning
    condition: degraded

# monitors.tf — Terraform approach
resource "devhelm_monitor" "checkout_api" {
  name      = "Checkout API Health"
  type      = "http"
  url       = "https://api.example.com/v1/checkout/health"
  frequency = 30

  regions = ["us-east", "eu-west", "ap-southeast"]

  assertions {
    status_code    = 200
    response_time  = 1000
    json_path      = "$.status"
    expected_value = "healthy"
  }

  alert_channels = [
    devhelm_alert_channel.pagerduty_checkout.id,
    devhelm_alert_channel.slack_engineering.id,
  ]
}

Both formats express the same intent. The HCL version integrates with Terraform's plan/apply workflow. The YAML version integrates with a CLI (devhelm apply -f monitors/). Either way, the monitor definition lives in Git.

The payoff: review, history, and reproducibility

Once monitors are code, you get everything version control provides:

Pull request review for monitor changes. When someone changes an alert threshold, the diff shows up in a PR. A teammate can ask "why did you change the checkout timeout from 1s to 5s?" before it ships. Bad configurations get caught before they reach production — the same way bad code does.

Git blame for debugging. "Who changed this monitor, and when?" is a git log query, not a support ticket to the vendor.

Atomic deploys with the application. When a developer renames an endpoint from /v1/users to /v2/users, the monitor definition updates in the same commit. The monitor never points at a dead endpoint.

Environment parity. The same monitor definitions deploy to staging and production with environment-specific variables. Staging monitors catch issues before production monitors do.

Disaster recovery. If you need to recreate your entire monitoring setup — new vendor, new account, after an outage — you run one command: devhelm apply -f monitors/ or terraform apply. Everything recreates from the source of truth.

Audit trail. For SOC 2 and ISO 27001, you need evidence that monitoring changes are reviewed and authorized. Git history provides this automatically.

The workflow

A mature monitoring-as-code workflow looks like:

1. Developer changes application code
2. Developer updates monitor definition in the same branch
3. PR review covers both code and monitoring changes
4. CI validates monitor syntax (lint, dry-run)
5. Merge to main triggers deploy pipeline
6. Application deploys first
7. Monitors deploy/update second (same pipeline)
8. Monitors validate the deploy succeeded

Step 7 is the key: monitors deploy through CI/CD, not through a web UI. The running state always matches what's in the repository.

What should be code — and what should not

Not everything needs to be version-controlled configuration:

Should be code:

• Monitor definitions (URL, frequency, assertions, regions).
• Alert channel configuration (which Slack channel, which PagerDuty service).
• Notification policy rules (which severity routes where).
• Status page component mappings (which monitors feed which status page components).
• Escalation policies.

Can stay in a UI:

• One-time ad-hoc debugging checks (temporary monitors during incident investigation).
• Dashboard visualizations (the view of data, not the data source).
• Historical alert data and incident timelines (these are records, not configuration).

The rule: if losing it would require manual recreation, it should be code. If it's ephemeral or a view of data, a UI is fine.

Common objections — and responses

"Our team isn't technical enough for config files." If your team manages monitors, they're technical enough to edit YAML. A config file with clear field names is less complex than navigating a 15-field form in a web UI. And unlike the form, the config file has documentation, examples, and review.

"We'd need to deploy every time we change a monitor." Yes — that's the point. An unreviewed, un-deployed change to a production monitor is exactly as risky as an unreviewed change to production code. The deploy pipeline is the safety net.

"What about quick changes during an incident?" Some tools support both: config-as-code for steady-state, with a UI for temporary overrides that sync back to the repo. If your tool doesn't, keep a documented escape hatch ("during incidents, mute via UI; file a follow-up to update the config post-incident").

Getting started

1. Export your current monitors. Most tools have an export or API endpoint. Capture your current state into files.
2. Pick a format. YAML for simplicity, HCL for Terraform users, TypeScript/JSON for programmatic generation.
3. Version control them. Commit to your application repository (same repo = atomic changes) or a dedicated infrastructure repo (if you manage many services).
4. Add a deploy step. Wire your CI/CD to apply monitor definitions on merge to main. Start with a dry-run step that validates syntax.
5. Enforce the workflow. Once monitors deploy from code, disable (or restrict) direct UI edits so drift cannot accumulate.

The first time a PR catches a bad monitor change before it reaches production, the investment pays for itself. The second time — when you recreate your entire monitoring setup in a new environment in 30 seconds — it pays for itself again.

Set up monitors as code with a CLI (devhelm apply), a Terraform provider, or the dashboard — and a status page that updates from the same monitor data — at app.devhelm.io. Your first monitor is live in 60 seconds, no credit card.

---

Originally published on DevHelm.