DEV Community: Atomsized

Does Your AI Agent Have a Kill Switch?

Boyan Iliev — Mon, 03 Aug 2026 06:00:00 +0000

The agent believed it was working inside a simulation. The systems it reached were real. Anthropic's report shows why a prompt cannot carry the whole safety boundary for an autonomous task.

An engineer should know what the agent is doing, what it can reach and how to stop it.

On 30 July 2026, Anthropic published a review of 141,006 cybersecurity evaluation runs. It found three incidents, across six runs, where Claude reached the internet and gained unauthorised access to real organisations.

The models were running capture-the-flag exercises. Their prompts said the environment was simulated and had no internet access. A misunderstanding with the evaluation partner left live internet access available, so the models often treated real systems as part of the exercise.

The models followed their task with a false picture of the environment. The runtime allowed actions that the prompt said were impossible, and the activity was found later through transcript review.

An engineer should understand the task before it begins, see its progress and be able to stop it immediately. The platform must enforce those controls even when the model misunderstands the situation.

The agent had the wrong picture of its environment.

Anthropic's evaluation prompt told Claude that it had no internet access. The machines used for the exercise could reach the internet because of a misconfiguration. The agent trusted the description in the prompt and treated reachable systems as part of the simulated challenge.

The incidents had real effects. One reached production infrastructure and a database. Another published a malicious Python package to PyPI. It was available for roughly one hour and ran on 15 real systems. A third scanned thousands of targets before compromising an internet-facing application.

The models reacted differently to signs that the targets were real. An older model continued. Another convinced itself that it was still in a simulation. Anthropic's latest research model stopped after recognising the real environment.

Anthropic describes this as closer to a harness and operational failure than a model alignment failure. Normal production safeguards were absent because the evaluations measured the underlying models. Containment and monitoring therefore carried more responsibility.

The prompt described the intended world. The infrastructure exposed the real one.

Make the environment prove what the agent can reach.

A prompt can name the approved environment, resources and goal. It cannot close a network route, remove an IAM permission or prevent a tool from calling an unapproved endpoint.

Build the smallest working environment for the task. Give the agent a separate identity, short-lived credentials, an allowlist of tools and resources, a network policy and a time limit. Check those controls from inside the runtime it will use.

Name the account, cluster, repository, namespace, service and environment. Describe which systems are real, simulated or outside scope. If the runtime disagrees with that description, stop before the first action.

Review third-party execution environments as well. The team needs to know who owns the network, identity, logs, emergency stop and incident response. An assumption between two organisations can become a live access path.

Verify outbound network access from inside the agent runtime.
Issue one short-lived identity for one task and one environment.
Allow only the tools, resources and actions required for the approved goal.
Record the expected account, cluster, repository and service before execution.
Stop when the observed environment differs from the approved context.
Confirm who owns monitoring and incident response across every vendor boundary.

Controls should still hold when the agent's understanding is wrong.

Give the engineer a live view of the task.

Before the run, the responsible engineer should see the goal, plan, environment, identity, tools, access boundary, time limit and stop conditions. A broad instruction such as investigate the cluster leaves too much room for interpretation.

During the run, show the current step, tool call, target resource, network destination, command, proposed change and evidence. Highlight denied requests, repeated attempts, new destinations and actions outside the plan.

Keep the important state above the raw transcript: the goal, current action, completed changes, remaining access and failed policy checks. Preserve the full trace underneath for audit.

Alert the named owner when the agent approaches a boundary. The owner can follow normal reads through the activity view and receive a clear signal when approval is required or the task changes shape.

Keep the plan, live work and stop path together

The engineer should be able to understand and control the task from one place.

Approved goal: Name the owner, environment, result and time limit.
Bounded identity: Issue short-lived access for the approved tools and resources.
Live activity: Show actions, targets, evidence, retries and denied requests.
Policy checks: Compare every action with the approved scope before the tool runs it.
Engineer checkpoint: Pause before a production change, new destination or high-risk action.
External stop: End the task and remove its access through a control the agent cannot change.
Preserved evidence: Keep the plan, tool calls, logs, diffs and reason for stopping.
Verified recovery: Check the desired state and let the engineer decide the next step.

Awareness means seeing the plan, the current action and the remaining authority.

A kill switch must stop access as well as reasoning.

Industries that use machinery design an emergency stop into the system. An infrastructure agent can cross networks, use credentials and change real resources quickly. Design its stop before it starts.

Ending the model response leaves other work running. Tool calls, CI jobs, cloud operations, retries and scheduled tasks may continue after the visible conversation ends.

Place the kill switch in the orchestrator or a separate control service. The agent cannot disable it, edit its policy or issue itself a new identity. Engineers and automatic policy controls should both be able to trigger it.

Stopping also needs a recovery choice. Preserve the evidence, compare the running system with the approved state and let the engineer choose whether to reconcile, revert or investigate further.

Stop the active run: End new calls, then terminate active sessions and jobs where possible.
Remove authority: Revoke the task identity, expire tokens and block its network and tool access.
Cancel future work: Clear queued jobs, retries, schedules and child tasks created by the run.
Preserve the record: Store the plan, prompts, actions, destinations, responses, diffs, logs and stop reason.
Check the real state: Compare Git, GitOps, Kubernetes and cloud state with the last approved result.
Return ownership: Give the responsible engineer the evidence and recovery choices before any restart.

The stop control must remain available when the agent, model or runtime behaves unexpectedly.

Keep engineers close to actions that change the system.

Engineers should see the plan, the live activity and every point where approval is needed. Routine reads can continue inside the approved boundary.

Read-only evidence gathering can publish a report for review. Infrastructure changes should arrive as a diff or pull request. Git records the proposal, CI checks it and an engineer approves it before GitOps applies the accepted state.

Identity, networking, data, deletion and security changes deserve explicit approval. Small reversible actions may gain more automation after the team tests the limits, recovery and stop path with real runs.

For Atomsized workflows, AI reads and prepares. Engineers decide and approve. Git records the accepted change. GitOps applies it. AWS and monitoring report the result.

The engineer owns the decision, the boundary and the recovery.

Begin with one task and rehearse the stop.

Choose one repeated task with a known owner and clear result. Read-only investigation lets the team compare the agent's work with an engineer's process before giving it change access.

Write down the scope and pause conditions. Build the live view and kill switch, then replay earlier incidents where the correct outcome was to ask for help or stop.

Run beside an engineer. Review the actions, access, weak context and alerts. Expand only after the team trusts the monitoring and has rehearsed recovery.

1. Choose the task: Name one owner, one environment, one expected result and one maximum runtime.
2. Map every action: List the tools, resources, destinations, permissions and data the task may use.
3. Build the boundary: Create the task identity, allowlists, network rules and automatic stop conditions.
4. Show the live state: Put the plan, current action, evidence, changes and policy decisions in one view.
5. Test the kill switch: Stop an active tool call, revoke access, cancel queued work and confirm the evidence remains available.
6. Run in shadow mode: Use real work with no change authority, compare results and fix gaps before adding another action.

A team should prove that it can stop the agent before asking the agent to do more.

Test what happens after the button is pressed.

A kill switch can report success while external work continues. Confirm that credentials fail, network paths close, active jobs end where possible and queued work disappears.

Inspect the running state next. GitOps may restore an approved configuration, while data operations and external API calls can need a separate recovery. Show what completed, stopped halfway and still needs attention.

Keep these tests in the evaluation set. Include wrong context, unexpected internet access, new destinations, repeated denied actions, lost monitoring and a failed stop attempt.

The manual stop remains available outside the agent runtime.
Automatic policy checks can stop the task without model cooperation.
Credentials, network paths, active jobs, queues and retries are checked separately.
The full activity record remains available after access is removed.
The running system is compared with the last approved state.
An engineer confirms recovery before the agent can start again.

Pressing stop begins the recovery. Verified state finishes it.

Design the stop before the agent starts.

Anthropic's report shows how reasonable assumptions can line up badly. The agent believed the prompt, the environment exposed more than intended and monitoring found the activity after real systems had been reached.

Give every task a named owner, accurate context, limited access, live monitoring, checkpoints and a stop control the agent cannot reach.

Start with one read-only workflow. Make its work visible, rehearse the kill switch and verify recovery. Add authority from evidence gathered during real runs.

An engineer should always know what the agent is trying to do, what it can touch and how to stop it.

How Can You Trust AI Agents With Your Infrastructure? — The context, operating experience, permission boundary and recovery path an infrastructure agent needs before it prepares a change.
How Should AI Monitor Your Infrastructure? — How an AI monitoring agent can gather evidence, report useful findings and keep the responsible engineer in control.

Give the agent a visible task and an external stop.

We can map one AI-assisted workflow, limit its access, show engineers what it is doing and build the stop and recovery path before it reaches production.

Schedule a platform call

How Should AI Monitor Your Infrastructure?

Boyan Iliev — Thu, 30 Jul 2026 06:00:00 +0000

An alert tells you that something changed. It rarely says why, who is affected or what to do next. The on-call engineer still opens several tools and pieces the story together. AI can help when it understands the platform around the alert.

The agent collects the evidence and reports the finding. Engineers decide what to do.

Imagine an API latency alert. It identifies the graph that moved. It does not explain whether users are blocked, what changed ten minutes earlier or which dependency is slow.

We have worked through incidents where a deployment looked healthy while a dependency was failing. We have also seen the largest movement on a dashboard lead away from the real user problem. The answer came from connecting the signals to the platform around them.

An agent that opens dashboards faster saves a few clicks. A useful one follows the incident like an experienced engineer. It knows how the service should behave, how the parts connect, what changed and who owns the response.

Start with one incident your team already understands. Write down how the investigation works, connect the information it needs and let the agent follow that path in read-only mode.

The agent should join the investigation early.

Monitoring rules tell the team when a known signal crosses a limit. Someone still works out whether users are affected, which services are involved, what changed and what can be done safely.

The agent can begin when the alert arrives. It can update the incident notes, collect evidence and prepare a clear summary for the responsible engineer.

The first guess will often be incomplete. One service may recover while another starts to fail. A recent deployment may look related and turn out to be harmless. The agent should keep checking its explanation as the situation changes.

What we want is simple: a shorter path from the first signal to a useful understanding of the problem. The engineer should see who is affected, what changed, which dependencies matter, what evidence is missing and what to check next.

The agent keeps the incident picture updated. The engineer decides what to do.

Give the agent a clear map of the platform.

A large collection of metrics and logs is only the starting point. The agent needs to know how those signals relate to the system the team runs.

Start with what the user is trying to do. Follow that path through the applications, databases, queues, networks, cloud resources and external services involved. Connect each part to its owner, expected state, deployment source and recovery steps.

Then add what changed. Git commits, CI runs, Argo CD syncs, Terraform changes, Kubernetes events, AWS activity, feature releases, maintenance and traffic shifts can all change how an alert should be read.

Add the team's experience as well. Runbooks show the usual response. Incident reviews show what happened here before. Maintenance notes explain temporary conditions. Confirmed results show which clues helped and which guesses were wrong.

Connect the user problem to the recovery

Each part answers a different question during the incident.

What healthy looks like: Describe what users and business processes should be able to complete.
How the system connects: Map the applications, data, networks, cloud resources and external services involved.
What is happening: Collect metrics, logs, traces, events and checks with clear names and timestamps.
What changed: Connect deployments, configuration, infrastructure changes and planned work.
What the team knows: Add owners, runbooks, maintenance, known exceptions and earlier incidents.
Agent investigation: Build the timeline, test possible causes, show gaps and prepare the next check.
Engineer decision: The responsible engineer reviews the evidence and chooses the response.
Check the recovery: Watch the result, record what happened and improve the next investigation.

The map should show how a user problem connects to the systems underneath it.

The agent can only use the context you prepare.

Most of the information probably exists already. The problem is that each tool describes the platform differently. A repository may use one service name, Kubernetes another and the cloud account a third. Timestamps may use different zones. The team may know the owner even though the tools do not.

Connect those sources while keeping a link to the original data. The agent should be able to move from a user-facing check to the service, current deployment, related infrastructure, recent change and owner without guessing what names mean.

Access also needs care. The agent may read from several systems, but each source still needs its own permissions and audit record. Secrets and unrelated customer data should stay outside the investigation.

What healthy looks like: Record the user result, expected service behaviour, safe limits and checks that confirm recovery.
One name for each thing: Match services, environments, clusters, accounts, resources and deployments across the tools that use them.
Service connections: Keep the upstream and downstream links across applications, data, network, cloud and external services.
Monitoring data: Collect user checks, metrics, logs, traces and events with clear timestamps and source labels.
History of changes: Put code, configuration, infrastructure, feature, maintenance and traffic changes on one timeline.
Team knowledge: Connect owners, escalation paths, runbooks, known exceptions and incident results confirmed by engineers.
Safe access: Use a separate machine identity, permissions for each source and a record of what the agent reads or proposes.

A stronger model cannot fix missing names, broken timestamps or an old service map.

Let the agent write the first investigation draft.

This is the approach behind the Atomsized Observability Agent. It reads the approved environment context, then checks Kubernetes workloads, pod status, events, logs, Prometheus alerts, selected AWS resources and spend signals on a schedule.

Most checks should stay quiet. Normal deployment churn, short pod restarts and spot node rotation only need a report when the behaviour persists or affects a real workload.

When it finds something worth investigating, the agent writes a short Markdown report in the observability repository and posts it to Slack for review. The report states what happened, why it may matter, which evidence supports it and where the engineer should check next.

That level of automation removes repeated collection work and keeps the team informed. The responsible engineer reviews the finding, decides whether action is needed and owns the response through recovery.

Name the environment, service, affected user result and current owner.
Describe the finding and show what normal looked like before it.
Attach the useful evidence with clear sources and timestamps.
Connect recent deployments, configuration and infrastructure changes.
Show possible causes, weak links and missing information.
Suggest the next useful check for the responsible engineer.
Update the report when the evidence or affected service path changes.
Record whether the response restored the expected result.

Automate evidence collection and reporting. Keep production decisions with the responsible engineer.

Keep the response inside the usual delivery and recovery process.

The agent may need to read from monitoring, delivery and cloud systems. Changing those systems needs a much smaller set of permissions.

For Atomsized workflows, AI reads and prepares. Engineers decide and approve. Git records the accepted change. GitOps applies and reconciles it. AWS reports what happened in the running environment.

The incident summary should name the service owner, incident owner, proposed action, expected result, stop condition and recovery check. If the evidence changes before approval, the proposal should be updated.

Later, the team may allow one small and reversible response to run automatically. The identity, action limits, retries, time limit and stop conditions should be enforced outside the model. Changes to data, identity and networking should keep a stronger review.

The agent prepares the evidence and response. The responsible engineer owns the change and recovery.

Start with one incident your team already understands.

Trying to monitor the whole platform with AI creates too much scope for the first version. Choose one service path and one type of incident that experienced engineers already know how to investigate.

A good starting case has clear user impact, several useful data sources, a known owner and enough past incidents to test against. It should be safe to run with read-only access.

Write down the investigation before connecting the agent. Follow an engineer from the first alert to the confirmed recovery. Record the tools they open, the questions they ask and the points where missing information slows them down.

That path becomes the first monitoring skill. In read-only mode, the agent can run the checks on a schedule, prepare the report and show when the evidence points somewhere else.

1. Choose the incident: Choose the service, type of incident, affected user result, owner and evidence that confirms recovery.
2. Map the steps: List the tools, data sources, questions and decisions used during the investigation.
3. Connect the data: Give services stable names, align timestamps and provide read-only access to the evidence the agent needs.
4. Replay past incidents: Test the workflow with real incidents and near misses, including cases where the right answer is to wait or ask another owner.
5. Run reports beside the engineer: Let the agent publish live findings for the on-call engineer without changing production or paging more people.
6. Expand from results: Fix missing context and add another type of incident only after the first investigation path is dependable.

One person who owns the monitoring workflow and one owner for the service.
A written description of what healthy looks like for users, the service and its dependencies.
A service map connected to current ownership and escalation.
One timeline for monitoring data, application changes and infrastructure changes.
A read-only agent identity with limited access and an audit trail.
A set of real incidents with results confirmed by engineers.
One incident-summary format that shows evidence, missing information and other possible causes.
A review after the shadow period with a clear choice to improve, expand or stop.

One complete incident path teaches the team more than a broad monitoring demo.

Check whether the agent makes the investigation easier.

A clear explanation can still send the engineer in the wrong direction. Judge the agent by the help it provides during the investigation and by the evidence attached to its conclusions.

During replays and live shadow mode, compare its incident picture with the one built by the engineer. Record where it found useful information sooner, where it followed a weak link and where missing ownership or an old service map blocked it.

After each incident, the engineer confirms the cause, useful evidence, chosen response and recovery. Add those confirmed facts to the runbook, incident history and tests used by the monitoring skill.

Time from the first alert to a useful incident summary.
Conclusions that include a source, timestamp and affected service path.
Missing information found before the agent recommends a response.
Weak links or confident guesses the engineer needs to correct.
Checks and responses accepted, changed or rejected by the owner.
Recovery confirmed and recorded after the response.

The agent should learn from what the team confirmed after the incident.

Build useful context first.

AI monitoring becomes useful when the agent can see the platform around the alert. It needs to know what healthy looks like, how services connect, what changed, who owns the response and what happened in similar incidents.

It should keep one clear incident picture from the first signal to recovery. Engineers can see what is happening, which possible causes fit the evidence, what remains unknown and what to check next.

Start with one repeated investigation and run the agent beside the on-call engineer. Let it automate evidence gathering and reports while the team fixes the missing context that real incidents expose. Add another incident type when the first one is dependable.

Engineers remain responsible for decisions and production changes. The agent helps them reach those decisions with better information and less searching.

How Can You Trust AI Agents With Your Infrastructure? — The context, operating experience, permission boundary and recovery path an infrastructure agent needs before it prepares a change.
Why GitOps is so hard to explain and automate. — A practical operating model for proposals, engineer approval, reconciliation and evidence across AWS and Kubernetes.

Start the Observability Agent with one useful report.

We can map what healthy looks like, connect the evidence and automate a report your engineers can review before deciding what to do.

Schedule a platform call

From Shared Staging to Reviewable Environments

Boyan Iliev — Mon, 27 Jul 2026 06:00:00 +0000

One team deploys a feature for QA. Another needs to test a migration. Product wants a stable demo. Each request makes sense. A single shared environment forces them into the same queue.

Shared staging becomes a queue when several teams need it at the same time.

We have seen one deployment overwrite another, testers wait for a stable slot, demos block development work and teams lose track of which revision is running.

The name varies between teams: staging, stage, dev, test or UAT. We use shared staging to mean any environment where several changes have to take turns.

When several decisions wait for the same environment, they have to happen one after another. Engineering, QA and product repeat tests, screenshots and approvals, coordinate deployment windows and wait longer to move a change forward.

Extra permanent environments rarely clear that queue. A reviewable environment has one job: help someone make a specific decision about a known version of a change.

Shared staging slows down when reviews run in parallel.

Waiting and unclear targets cause the delay. A team loses confidence in a review when the target changes underneath it.

Changes collide: A newer deployment replaces the version somebody else is still testing.
Review waits: QA, product and engineering negotiate for a stable window instead of reviewing the work.
Results become unclear: A test result or screenshot may no longer match what is currently deployed.

Measure the wait as part of delivery time.

A staging queue often starts before the deployment metrics begin. We look at how long a change waits for its first useful review, how much work is repeated and how often a newer deployment makes earlier results out of date.

Review delay: Track the median time from ready for review to the first useful validation.
Time spent waiting: Record engineering, QA and product hours spent waiting, coordinating or repeating work.
Repeated review work: Count tests, demos and approvals that need to be repeated after a newer deployment.

Measure one application or service for two weeks. Compare its results over time instead of combining data from teams that work in different ways.

Build the smallest environment the review needs.

A reviewable environment has a clear purpose, a known source revision, an owner, clear rules for test data, visible health and a cleanup rule. It only needs the parts required for that decision.

A preview environment gives developers, testers, product teams and stakeholders their own reviewable version of a change. They can develop, test and demonstrate it without competing for shared staging.

When previews reuse existing Kubernetes capacity and shared platform services, the extra running cost can stay low. A short-lived namespace uses the cluster and services that are already running.

That changes when a preview needs dedicated nodes, databases, load balancers, large data sets or a long lifetime. Clear resource limits and automatic cleanup after a set time help keep costs under control.

Keep staging production-like. Preview environments should use the same built application versions, configuration patterns and deployment path, while scaling down safely and keeping production data out.

Preview, UAT and shared staging answer different questions.

Each stage supports a different kind of review.

Preview environment: Review one meaningful change quickly and in isolation.
Coordinated UAT: Test a fixed combination of services, repositories or business flows.
Shared staging: Keep it for final release checks and tests that need the full system in a common state.

Record each step from creation to deletion.

A pull request records the change. CI builds a fixed application version. Git records how the environment should be set up. Argo CD applies that setup and keeps it in sync. The URL, version, health and test results return to the pull request.

When the change is merged, closed or reaches its time limit, the environment definition is removed and the environment is cleaned up. Git history and review results remain available.

Start with the workflow that waits for shared staging most often.
Build one complete create, update, review and delete lifecycle.
Measure waiting time, failed environments and old environments that were not removed before adding more teams or services.

Fund one review path, then decide from the results.

Choose one staging bottleneck where work regularly waits or needs to be repeated. Run a 30-day pilot, then decide whether the team should spend more time and money on it.

Baseline: Measure the current wait, repeated work and environment cost for one review path.
Pilot: Build one complete create, update, review and delete lifecycle using the existing delivery controls.
Decide: After 30 days, extend, adjust or stop based on review delay, repeated work, stability and AWS spend.

After 30 days, ask whether reviews moved sooner and whether the workflow kept stability and AWS spend under control.

Start with a review that regularly waits.

Find the change that repeatedly waits for staging, give it one controlled review path and measure whether the queue gets shorter.

Shared staging keeps a clear job: final release validation and shared checks. Preview and UAT paths handle review work that needs isolation.

Preview environments for a monorepo team of five or more. - A practical GitOps design for isolated, disposable application states when one repository contains the complete source state.
Preview environments for a microservice architecture with 10+ repositories. - A GitOps operating model for single-service and coordinated previews across multiple repositories.

Turn one blocked review path into a controlled workflow.

We can map the smallest useful preview or UAT workflow around your repositories, GitOps controls and current AWS platform.

Schedule a platform call

DEV Community: Atomsized

Does Your AI Agent Have a Kill Switch?

The agent had the wrong picture of its environment.

Make the environment prove what the agent can reach.

Give the engineer a live view of the task.

Keep the plan, live work and stop path together

A kill switch must stop access as well as reasoning.

Keep engineers close to actions that change the system.

Begin with one task and rehearse the stop.

Test what happens after the button is pressed.

Design the stop before the agent starts.

Related articles

How Should AI Monitor Your Infrastructure?

The agent should join the investigation early.

Give the agent a clear map of the platform.

Connect the user problem to the recovery

The agent can only use the context you prepare.

Let the agent write the first investigation draft.

Keep the response inside the usual delivery and recovery process.

Start with one incident your team already understands.

Check whether the agent makes the investigation easier.

Build useful context first.

Related articles

From Shared Staging to Reviewable Environments

Shared staging slows down when reviews run in parallel.

Measure the wait as part of delivery time.

Build the smallest environment the review needs.

Preview, UAT and shared staging answer different questions.

Record each step from creation to deletion.

Fund one review path, then decide from the results.

Start with a review that regularly waits.

Related articles