Introduction
When you run a product, there is a familiar scene: a Datadog error alert lands in Slack, and someone starts a first-response investigation in the thread — "what is this error?". You read the logs, follow the trace, open the code, and form a hypothesis about the cause. The first 30 minutes are mostly routine work.
This article is about an AI agent that automates those 30 minutes with Cloudflare Workers + GitHub Actions + Claude Code. It covers the architecture, why I chose this stack, and the measured cost of about a month in production.
What I built: the architecture
There are only four components.
| Component | Role |
|---|---|
| Datadog | An error monitor fires and posts a notification to a Slack alert channel |
| Cloudflare Workers (Hono + D1) | A thin receiver that detects the alert from Slack events and launches GitHub Actions |
| GitHub Actions | Started via repository_dispatch; the environment where Claude Code (headless) runs the investigation |
| Slack | The investigation result comes back to the alert thread |
The happy path is just this:
The key decision is that the Worker stays a thin receiver and the heavy investigation runs on GitHub Actions. Slack's Events API expects a response within 3 seconds, so the Worker only verifies the signature, classifies the message as an alert, and immediately fires repository_dispatch. The investigation itself runs Claude Code CLI headless on Actions (claude -p --model sonnet), working from a shallow checkout of the product repository plus the Datadog MCP server (read-only tools only).
Datadog webhooks could call the Worker directly, but the flow deliberately goes through Slack. Keeping the alert, the investigation result, and the follow-up discussion in one thread turns the Slack alert channel into the archive of error handling. With the Slack post as the trigger, a threaded reply to that message is all it takes.
When the investigation concludes that a code change is needed, the reply carries a Create Issue button. If we decide to act on it, the button creates a GitHub Issue, and a local Claude Code session picks up that Issue to open a fix pull request. The agent's job ends at the first-response investigation and the Issue draft; fixes go through a flow where human review applies.
The agent never posts to Slack directly. It only writes the investigation result to a Markdown file, and a subsequent trusted step does the posting. Alert text can contain user input, so it is an attack surface: the payload is always handled as data, the skill directory is made read-only, and the post step refuses to post if the output contains anything that looks like a secret — a set of prompt-injection countermeasures.
Why this stack
1. The investigation can reach into your own codebase
Datadog's built-in AI features stay within telemetry — logs, metrics, traces. What you actually want from a first response is a hypothesis down to "which file and line, and roughly how to fix it". With your own agent, you can check out the repository and have it pinpoint file:line and sketch the fix. That is the biggest difference from the off-the-shelf option.
2. Fully usage-based, with visible per-run cost
Every part of this agent bills only when it runs. Idle cost is zero; if no alert fires, you pay nothing. SaaS AI features tend to be tied to seats or a higher-tier SKU, and you cannot see what one alert's investigation cost. With your own build, separating API keys per feature is enough to get the measured per-run cost shown below.
3. Cloudflare makes the receiver cheap and simple to run
Workers run on V8 isolates, which start fast and use little memory, so they fit the requirement of "handle a Slack event within 3 seconds and hand off to async work". For state, D1 (SQLite) is directly available from the Worker, and one dedup table was all it needed. Written with Hono, the Worker is just a small web app, and the free tier covers this workload entirely.
The investigation itself lives on GitHub Actions instead of Workers because it needs to check out the whole repository and run Claude Code. A filesystem-heavy workload that takes minutes fits an Actions job started by repository_dispatch better than Workers with their CPU-time and filesystem constraints. Actions sits right next to the code, secret management is built in, and being event-driven it has no standby cost.
The cost trick: never investigate the same error twice
This mechanism turned out to matter most in production.
The reality of error alerts is that the same error fires again and again. The errors that stay unfixed the longest are the ones with small real-world impact — they keep firing while the fix stays on the backlog. Those repeated alerts cause notification fatigue, and if an AI reruns the same investigation each time, the waste converts directly into API cost. There is no point running a full investigation on "that known error fired again".
So the Worker keeps an investigated-issues table in D1, fingerprinted by the issue.id that Datadog Error Tracking assigns.
CREATE TABLE incident_issues (
issue_id TEXT PRIMARY KEY, -- Datadog Error Tracking issue.id
channel TEXT NOT NULL,
thread_ts TEXT NOT NULL, -- first investigation thread
triaged_at TEXT NOT NULL DEFAULT (datetime('now'))
);
If an alert with the same issue.id arrives within 14 days, Actions is not launched (= zero tokens), and the new alert's thread only gets a link: "this error is already being handled — see the first investigation thread".
The check is a D1 INSERT ... ON CONFLICT DO NOTHING, so even concurrent alerts for the same issue cannot start two investigations.
A side effect of this mechanism is that the investigation and all follow-up discussion accumulate in the first thread. Duplicate threads for the same error — "where were we discussing this again?" — structurally cannot happen. After 14 days, a recurrence is treated as "still not fixed, or the situation changed" and gets re-investigated.
One deliberate design choice: when the dedup check fails (a D1 outage, or an alert without an issue.id), the system fails toward running the investigation. An occasional duplicate investigation costs less than a missed one.
A few more guards keep cost and blast radius bounded:
- The model is pinned to Sonnet, and the job has
timeout-minutes: 20 - The agent's allowed tools are read-only Datadog MCP tools and local file operations only (no Bash)
- API keys are separated per feature, so cost is measurable per feature
Measured cost
These are measured values from about a month in production, 2026/06/21–07/19.
For context, the investigated codebase is a TypeScript-centric monorepo: roughly 4,600 source files, about 1,800 directories, over one million lines. It is a business application with users on the order of a thousand — production code at real scale, not a toy sample, running at the cost below.
Alert volume vs. investigations run
First, the denominators. Over this period, the product's error monitors fired (Triggered) 265 times in production alone. Actual investigations (GitHub Actions runs) numbered 121. Filtering out Recovered / Re-notified posts plus the dedup mechanism cut executions to less than half of the alert volume. Every skipped run costs exactly zero.
AI (Anthropic API)
The API key dedicated to this feature consumed $93.90 (including test runs during initial setup). Divided by 121 investigations, that is about $0.78 per investigation. Against a human spending 30 minutes on the same first response, under $1 per run is easily worth it.
Is it actually cheaper than Datadog's built-in option?
Datadog offers autonomous investigation via Bits Investigation, billed through AI Credits. At the time of writing: $500 per 500 credits per month (annual; on-demand is $1.30/credit), and one autonomous investigation consumes about 6.5 credits — roughly $6.5–8.5 per investigation.
At $0.78 per run, the self-built agent costs 1/8 to 1/10 of that; 121 runs a month through Bits would come to roughly $790–1,000. The credit commitment also starts at $500/month, while this stack has zero fixed cost. Bits is managed and needs no setup, so it is not a strict win-lose comparison — but for the specific job of alert-driven first-response investigation, self-built runs an order of magnitude cheaper.
GitHub Actions
The ubuntu-latest jobs averaged 3 minutes each; 121 runs totaled about 360 minutes for the month. That fits inside the Actions free allowance included in GitHub's paid plans (3,000 min/month on Team, 50,000 on Enterprise), so the added cost was zero. Even if the allowance were exhausted and every minute were billed, Linux runners cost $0.008/min — under $3/month. With timeout-minutes: 20 on each job, the worst case is 2,420 minutes ≈ $19.
Cloudflare
The Worker only receives alert events, so request volume never approaches the free tier (100k requests/day). D1 reads and writes a few rows per alert — a rounding error against its free tier. Effectively $0 (or just the $5/month base fee if you are on Workers Paid anyway).
Summary
| Item | Measured (~1 month) |
|---|---|
| Anthropic API (the investigations, 121 runs) | $93.90 (≈ $0.78/run) |
| GitHub Actions | $0 (within free allowance) |
| Cloudflare Workers + D1 | $0 |
| Total | ≈ $94 (reference: ≈ $790–1,000 in Bits Investigation terms) |
For $94 a month, every alert gets an automatic first-response investigation, and the results accumulate in threads. Against the cost of a human doing that first response every time, it pays for itself. And since nearly the entire bill is Anthropic API usage — pure investigation cost — the stack carries no infrastructure fixed cost at all.
Closing
The first-response agent for Datadog alerts is composed of Cloudflare Workers (receiver) + GitHub Actions (execution) + Claude Code (the investigation). The essence of the stack is that it is built only from event-driven parts, so fixed cost structurally cannot occur.
In operation, the mechanism that mattered more than any investigative capability was the plain one: never investigate the same error twice. Error-alert operations live with the reality that low-impact, unfixed errors keep firing, and naively wiring an AI agent to alerts converts that reality directly into API spend. Dedup turned out to be a cost control, a notification-fatigue control, and an operational improvement that concentrates discussion into one thread, all at once.
As a next step, I want to extend past the first response: first, having the Actions-side agent go as far as cutting a fix branch and opening a draft PR (merge decisions stay with humans). Beyond that, since Cloudflare keeps expanding agent-oriented runtimes such as Sandboxes and Dynamic Workers, I want to try moving the investigation and fix execution from GitHub Actions onto Cloudflare, completing everything from the receiver to the fix PR at the edge.
Top comments (0)