BurnSwitch: Stopping a Runaway AWS Bill With Switch, Two AI Agents and One Human Tap

A Switch room, bridged to Telegram, where an AI watchdog spots a Lambda/SQS retry storm, Claude Code finds and fixes the bug, and nothing changes in AWS until the on-call human taps Allow once on their phone.

The problem

One bad deploy can quietly burn money all night, and nobody notices until the bill arrives.

Here's the classic case. A new client starts sending orders in a slightly different shape. The Lambda function that processes them throws an error. There is no dead-letter queue and no partial batch reporting, so SQS retries the whole batch, again and again. The good orders in that batch fail along with the bad one, and they get retried too. Invocations climb, errors climb, and the bill climbs with them.

Fixing it takes four different jobs: spot the burn, find the cause, pause the worker without losing orders, then ship a patch and recover in the right order. Today that usually means one tired human at 2 a.m. copying alerts between a dashboard, a chat app and a terminal.

The obvious shortcut is to hand the whole thing to an autonomous AI agent. But then nobody signs off on the commands that change production. That felt like the wrong trade.

In simple words

Imagine a kitchen tap left running in an empty house. The water bill grows every minute.

• A smoke alarm for water (our watchdog) notices the meter spinning too fast and calls the plumber.
• The plumber (Claude Code) finds the broken pipe and explains exactly what they want to do: "turn off the main valve", "replace this pipe", "turn the water back on".
• The homeowner (the on-call human) says yes or no to each step, from their phone. The plumber can't touch the valve without that yes.

BurnSwitch is that, for AWS. The "yes" is the switch.

A few words explained

• Lambda: a small piece of code AWS runs for you whenever there's work. You pay per run and per second of running.
• SQS: a queue. Orders wait in it until the Lambda picks them up.
• Retry storm: when failing messages are retried endlessly, so you pay over and over for work that never succeeds.
• Dead-letter queue (DLQ): a side queue where messages go after failing too many times, so they stop being retried.
• Switch: a platform where humans and AI agents share a "room", like a group chat, with approvals built in. Our room is bridged to a Telegram group.

What BurnSwitch does

Everything happens in one Switch room, with four participants:

Participant	What it does
@burn-watchdog	Our custom agent. Prices the Lambda's spend every 15 s, raises one alert, and keeps the ledger
@claude-code	Reproduces the bug, writes the patch and tests, and proposes each command
The human	The on-call engineer. Approves or declines every live change from Telegram
@finops-auditor	Joins at the end and writes the post-mortem from the ledger

The switch is the human's Allow once tap. Every live command (pause, deploy, reconfigure, resume, move messages) gets its own approval card. Nothing reaches AWS without one.

How it works, step by step

1. Watch the money. Every 15 seconds the watchdog reads Lambda invocations, errors, duration and queue stats, and turns them into dollars per hour using prices from a config file (never hardcoded).
2. Open one incident. When spend is at least 1.5× the healthy baseline, errors are at least 5% and messages have been received 3+ times, for two samples in a row, the watchdog opens an incident.
3. Hand off once. It sends exactly one alert, addressed to @claude-code. After that the agents only talk to the human, so they never ping-pong.

1. Diagnose. Claude Code reproduces the failure, names the cause and the commit that introduced it, writes a patch with tests, and proposes the first command: pause the worker.
2. Flip the switch. The human types @claude-code run 3a in Telegram. Claude Code asks for approval, Switch shows an Allow / Deny card, the human taps Allow once, and only then does the command run.

1. Recover in a fixed order. The same loop repeats for each step: deploy the patch, turn on partial batch failures, attach the dead-letter queue, resume, and finally move the good orders back from the DLQ.
2. Trust, but verify. The watchdog doesn't take anyone's word for it. It compares the infrastructure on every sample, traces each change to the command that made it, pairs it with the human's request, and counts every order: sent = processed + still queued + in the DLQ.
3. Write it down. The ledger (cost-ledger.md, a room document) records every change. The auditor joins last, reads it, and writes the post-mortem.

That's the whole idea: the agents do the work, the human holds the switch, and the watchdog keeps everyone honest.

The recovery order

Order matters here, and getting it wrong loses orders:

Step	Change	Why
3a	Reserved concurrency 0	Stops the burn at once; every message stays on the queue
4b	Deploy the patch	Validates input and reports only the bad orders as failed
4c	Turn on ReportBatchItemFailures	A bad order fails alone; the rest of its batch succeeds
4d	Redrive policy: orders-dlq, max 3 receives	Failing messages stop after 3 tries
4e	Run a safety check, then resume	Resuming is refused until 4c and 4d are both in place
4f	Move the DLQ back to the main queue	Brings back the good orders that resuming dead-lettered

Containment is just "pause". Nothing in the project ever purges a queue or deletes a message.

How we use Switch

• Rooms with a Telegram bridge: the whole incident lives in one room, so the human can handle it from a phone.
• A custom agent: @burn-watchdog speaks the Switch agent protocol (WebSocket events plus HTTP operations), reconnects with backoff, and follows Switch when the local server moves ports.
• Targeted messages: exactly one alert per incident, sent only to @claude-code.
• Console agents: Claude Code runs in the scenario/ folder with permission rules that make every live command ask, every time.
• Approval requests: each of those prompts becomes an Allow / Deny card in Telegram. This is the switch.
• Room documents: the ledger and the runbook live in the room, where every agent can read them.
• Room instructions and commands: shared rules for all agents ("never purge", "one command per approval"), plus !reset and !invite-agent for clean runs and a late-joining auditor.

The switch is enforced in four layers, not just one: the room instructions, Claude Code's own permission rules (raw aws, purge and delete are denied outright), Switch's approval cards, and our helper scripts, which refuse unsafe commands even if asked.

Building on Switch: what I wish I'd known on day one

If you're starting with Switch, these are the things that cost me the most time. Hopefully they save you some.

1. Run a local server first

Install Switch Console and Docker, then choose Run a server on this computer. The first start downloads several GB, so do it before you need it. A local server comes with its own messaging app, so you can build and test without any company workspace.

2. Connecting Telegram, in the right order

The order matters here, and getting it wrong produces confusing errors:

1. Create a bot with @botfather and save the token somewhere safe. Never paste it into a chat or a room.
2. In BotFather, turn Group Privacy off before adding the bot to any group. Otherwise the bot only sees messages that mention it.
3. Connect Telegram in Switch Console with the token and the bot's username.
4. Create the Telegram group yourself (bots can't), add the bot, and send a message in the group.
5. Only then link your Telegram account in Console, by searching your own @username.

My mistake: I tried to link my account straight after connecting and got "Bridge is not running". Switch can only find people who have already posted where the bot can see, so posting first, then linking, fixed it. A quick sanity check for your token: curl -s "https://api.telegram.org/bot$TOKEN/getMe" should return "ok":true, and "can_read_all_group_messages":true confirms Group Privacy is off.

3. Registering a custom agent

A custom agent needs a registration key, created on the Gateway's API Keys page. Two things to know:

• Create the key while signed in as yourself. The person who creates the key owns the agent. If you use the deployment's bootstrap key instead, the agent belongs to a separate account, and it won't be allowed to address your other agents.
• The agent's API key is shown once. Save it straight into .env.

To let one of your agents hand work to another, set the receiving agent's addressing to owner_and_owner_agents. By default, an agent answers only its owner.

4. The agent protocol in practice

• Events come over a WebSocket; actions are HTTP calls. Older SSE clients are deprecated, so build on the WebSocket.
• You receive every message in your rooms. Act only when payload.addressed is true, or your agent will reply to everything.
• The bridge URL isn't the Gateway URL. Check you have the right one with GET /health.
• Plan for disconnects. Answer pings, reconnect with backoff, and keep outgoing messages in an outbox until Switch accepts them.

Here's how @burn-watchdog connects and handles events:

And here's the single targeted alert it sends to @claude-code:

5. Making approvals actually block

The approval card comes from Claude Code's own permission rules. In scenario/.claude/settings.json, every mutating command is an ask rule, and dangerous ones are deny rules. Switch turns each ask prompt into an Allow / Deny card in the room. Two tips:

• Approve with Allow once, not "Allow for this session", for anything that touches production.
• Back the rules up with helper scripts that refuse unsafe commands themselves. That way, safety doesn't depend on a single layer.

6. Late-joining agents

When an agent joins a room, Switch gives it the room's instructions and the list of room documents. The message history isn't included automatically: the agent reads it with read_context. That's why our auditor can arrive at the very end, read the ledger and the thread, and write an accurate post-mortem without anyone re-explaining anything.

How did it go?

These numbers are from one recorded run on LocalStack:

• Alert in about 1 minute. The burn peaked at 12.4× the healthy baseline.
• Contained in 3 min 06 s, measured from the alert to the pause command. Resolved in 6 min 30 s.
• 6 live changes, 6 approved, each one traced to its command and to the human's request.
• No orders lost. 1,198 sent = 955 processed + 0 queued + 243 in the DLQ, and every one of those 243 is a malformed order.
• Cost avoided (estimate): $0.84. That's (peak − baseline) × 8 hours, assuming the storm would otherwise run until someone checked the bill in the morning.

The dollar figure is small because the demo traffic is small. It's an estimate built on a stated assumption, not a measurement, and the ledger says so.

What fought back

Agents can't see who tapped Allow. Switch stores who answered each approval card, but as far as we could find, no agent operation returns it. So the watchdog pairs each change with two things it can see: the human's "run …" message (up to 2 minutes before the command) and Claude Code's "Approved by …" line. The auditor is told to say this openly rather than claim more than we know.

Resuming sent the good orders to the DLQ. This one surprised us. Once a redrive policy is attached (step 4d), SQS moves any message whose receive count is already above the limit to the DLQ on its next receive, whatever the code does with it. The good orders stuck in the storm had receive counts of 10–120, so the moment we resumed (step 4e), they all went straight to the DLQ. The fix was a sixth step: move the DLQ back with start-message-move-task, which gives every message a fresh receive count. Good orders then succeed, and the malformed ones fail three more times and return.

Changes showed up before their own log line. The helper script writes its change log after the AWS command returns, so the watchdog sometimes saw a change before it knew who made it, and labelled it "source unknown". We added a retry: an unknown change is checked again for up to 2 minutes.

The alert went to the wrong room. We accidentally created a second Telegram group with a similar name, and a substring match picked the old room. The reset script now prefers an exact name match.

LocalStack only reports half the metrics. It emits Invocations and Errors but no duration, which we need to price GB-seconds. The watchdog reads Lambda's REPORT log lines instead. On real AWS, a CloudWatch adapter does the same job.

Limitations

• One Lambda function and one incident at a time.
• Our recorded runs used LocalStack. The real-AWS path is built and guarded (it needs a typed account ID and has a teardown script), but we haven't recorded it.
• The approval pairing above is inferred from messages, not read from Switch.
• Telegram can't open room documents, so the full ledger is read with npm run ledger:show.

Who this is for

Anyone who runs event-driven workloads on AWS and wants AI help during incidents without giving up control. You don't need to trust the agents blindly: every change asks first, and every change is checked afterwards.

What we learned

• Put the human at the switch, not in the loop. The agents do the typing; the human makes one decision per change, from a phone.
• Watch cost, not just errors. A dollars-per-hour number makes "is this bad?" obvious to everyone in the room.
• Verify, don't trust. Diffing the real infrastructure caught things neither agent mentioned.
• The recovery order is the hard part. Pausing is easy. Coming back without losing orders took six careful steps.
• Say what you can't measure. The ledger and the post-mortem both state their own assumptions.

What's next

• A safer deploy script. Our auditor agent suggested this one itself: refuse to deploy unless the queue has its redrive policy and the event source mapping has ReportBatchItemFailures. With both in place, the original bug would have failed only the malformed orders, with no retry storm.
• A guard so a patch can't ship without its tests passing.
• A recorded run on real AWS.
• More than one function and more failure types, such as timeouts and throttling.

Try it

• Code: github.com/Jeevesh2605/BurnSwitch
• Demo video: youtu.be/dIdHmdBiKkk

Built for the Flip the Switch hackathon (Track 01). #HowISwitch