An alert tells you that something changed. It rarely says why, who is affected or what to do next. The on-call engineer still opens several tools and pieces the story together. AI can help when it understands the platform around the alert.
The agent collects the evidence and reports the finding. Engineers decide what to do.
Imagine an API latency alert. It identifies the graph that moved. It does not explain whether users are blocked, what changed ten minutes earlier or which dependency is slow.
We have worked through incidents where a deployment looked healthy while a dependency was failing. We have also seen the largest movement on a dashboard lead away from the real user problem. The answer came from connecting the signals to the platform around them.
An agent that opens dashboards faster saves a few clicks. A useful one follows the incident like an experienced engineer. It knows how the service should behave, how the parts connect, what changed and who owns the response.
Start with one incident your team already understands. Write down how the investigation works, connect the information it needs and let the agent follow that path in read-only mode.
The agent should join the investigation early.
Monitoring rules tell the team when a known signal crosses a limit. Someone still works out whether users are affected, which services are involved, what changed and what can be done safely.
The agent can begin when the alert arrives. It can update the incident notes, collect evidence and prepare a clear summary for the responsible engineer.
The first guess will often be incomplete. One service may recover while another starts to fail. A recent deployment may look related and turn out to be harmless. The agent should keep checking its explanation as the situation changes.
What we want is simple: a shorter path from the first signal to a useful understanding of the problem. The engineer should see who is affected, what changed, which dependencies matter, what evidence is missing and what to check next.
The agent keeps the incident picture updated. The engineer decides what to do.
Give the agent a clear map of the platform.
A large collection of metrics and logs is only the starting point. The agent needs to know how those signals relate to the system the team runs.
Start with what the user is trying to do. Follow that path through the applications, databases, queues, networks, cloud resources and external services involved. Connect each part to its owner, expected state, deployment source and recovery steps.
Then add what changed. Git commits, CI runs, Argo CD syncs, Terraform changes, Kubernetes events, AWS activity, feature releases, maintenance and traffic shifts can all change how an alert should be read.
Add the team's experience as well. Runbooks show the usual response. Incident reviews show what happened here before. Maintenance notes explain temporary conditions. Confirmed results show which clues helped and which guesses were wrong.
Connect the user problem to the recovery
Each part answers a different question during the incident.
- What healthy looks like: Describe what users and business processes should be able to complete.
- How the system connects: Map the applications, data, networks, cloud resources and external services involved.
- What is happening: Collect metrics, logs, traces, events and checks with clear names and timestamps.
- What changed: Connect deployments, configuration, infrastructure changes and planned work.
- What the team knows: Add owners, runbooks, maintenance, known exceptions and earlier incidents.
- Agent investigation: Build the timeline, test possible causes, show gaps and prepare the next check.
- Engineer decision: The responsible engineer reviews the evidence and chooses the response.
- Check the recovery: Watch the result, record what happened and improve the next investigation.
The map should show how a user problem connects to the systems underneath it.
The agent can only use the context you prepare.
Most of the information probably exists already. The problem is that each tool describes the platform differently. A repository may use one service name, Kubernetes another and the cloud account a third. Timestamps may use different zones. The team may know the owner even though the tools do not.
Connect those sources while keeping a link to the original data. The agent should be able to move from a user-facing check to the service, current deployment, related infrastructure, recent change and owner without guessing what names mean.
Access also needs care. The agent may read from several systems, but each source still needs its own permissions and audit record. Secrets and unrelated customer data should stay outside the investigation.
- What healthy looks like: Record the user result, expected service behaviour, safe limits and checks that confirm recovery.
- One name for each thing: Match services, environments, clusters, accounts, resources and deployments across the tools that use them.
- Service connections: Keep the upstream and downstream links across applications, data, network, cloud and external services.
- Monitoring data: Collect user checks, metrics, logs, traces and events with clear timestamps and source labels.
- History of changes: Put code, configuration, infrastructure, feature, maintenance and traffic changes on one timeline.
- Team knowledge: Connect owners, escalation paths, runbooks, known exceptions and incident results confirmed by engineers.
- Safe access: Use a separate machine identity, permissions for each source and a record of what the agent reads or proposes.
A stronger model cannot fix missing names, broken timestamps or an old service map.
Let the agent write the first investigation draft.
This is the approach behind the Atomsized Observability Agent. It reads the approved environment context, then checks Kubernetes workloads, pod status, events, logs, Prometheus alerts, selected AWS resources and spend signals on a schedule.
Most checks should stay quiet. Normal deployment churn, short pod restarts and spot node rotation only need a report when the behaviour persists or affects a real workload.
When it finds something worth investigating, the agent writes a short Markdown report in the observability repository and posts it to Slack for review. The report states what happened, why it may matter, which evidence supports it and where the engineer should check next.
That level of automation removes repeated collection work and keeps the team informed. The responsible engineer reviews the finding, decides whether action is needed and owns the response through recovery.
- Name the environment, service, affected user result and current owner.
- Describe the finding and show what normal looked like before it.
- Attach the useful evidence with clear sources and timestamps.
- Connect recent deployments, configuration and infrastructure changes.
- Show possible causes, weak links and missing information.
- Suggest the next useful check for the responsible engineer.
- Update the report when the evidence or affected service path changes.
- Record whether the response restored the expected result.
Automate evidence collection and reporting. Keep production decisions with the responsible engineer.
Keep the response inside the usual delivery and recovery process.
The agent may need to read from monitoring, delivery and cloud systems. Changing those systems needs a much smaller set of permissions.
For Atomsized workflows, AI reads and prepares. Engineers decide and approve. Git records the accepted change. GitOps applies and reconciles it. AWS reports what happened in the running environment.
The incident summary should name the service owner, incident owner, proposed action, expected result, stop condition and recovery check. If the evidence changes before approval, the proposal should be updated.
Later, the team may allow one small and reversible response to run automatically. The identity, action limits, retries, time limit and stop conditions should be enforced outside the model. Changes to data, identity and networking should keep a stronger review.
The agent prepares the evidence and response. The responsible engineer owns the change and recovery.
Start with one incident your team already understands.
Trying to monitor the whole platform with AI creates too much scope for the first version. Choose one service path and one type of incident that experienced engineers already know how to investigate.
A good starting case has clear user impact, several useful data sources, a known owner and enough past incidents to test against. It should be safe to run with read-only access.
Write down the investigation before connecting the agent. Follow an engineer from the first alert to the confirmed recovery. Record the tools they open, the questions they ask and the points where missing information slows them down.
That path becomes the first monitoring skill. In read-only mode, the agent can run the checks on a schedule, prepare the report and show when the evidence points somewhere else.
- 1. Choose the incident: Choose the service, type of incident, affected user result, owner and evidence that confirms recovery.
- 2. Map the steps: List the tools, data sources, questions and decisions used during the investigation.
- 3. Connect the data: Give services stable names, align timestamps and provide read-only access to the evidence the agent needs.
- 4. Replay past incidents: Test the workflow with real incidents and near misses, including cases where the right answer is to wait or ask another owner.
- 5. Run reports beside the engineer: Let the agent publish live findings for the on-call engineer without changing production or paging more people.
- 6. Expand from results: Fix missing context and add another type of incident only after the first investigation path is dependable.
- One person who owns the monitoring workflow and one owner for the service.
- A written description of what healthy looks like for users, the service and its dependencies.
- A service map connected to current ownership and escalation.
- One timeline for monitoring data, application changes and infrastructure changes.
- A read-only agent identity with limited access and an audit trail.
- A set of real incidents with results confirmed by engineers.
- One incident-summary format that shows evidence, missing information and other possible causes.
- A review after the shadow period with a clear choice to improve, expand or stop.
One complete incident path teaches the team more than a broad monitoring demo.
Check whether the agent makes the investigation easier.
A clear explanation can still send the engineer in the wrong direction. Judge the agent by the help it provides during the investigation and by the evidence attached to its conclusions.
During replays and live shadow mode, compare its incident picture with the one built by the engineer. Record where it found useful information sooner, where it followed a weak link and where missing ownership or an old service map blocked it.
After each incident, the engineer confirms the cause, useful evidence, chosen response and recovery. Add those confirmed facts to the runbook, incident history and tests used by the monitoring skill.
- Time from the first alert to a useful incident summary.
- Conclusions that include a source, timestamp and affected service path.
- Missing information found before the agent recommends a response.
- Weak links or confident guesses the engineer needs to correct.
- Checks and responses accepted, changed or rejected by the owner.
- Recovery confirmed and recorded after the response.
The agent should learn from what the team confirmed after the incident.
Build useful context first.
AI monitoring becomes useful when the agent can see the platform around the alert. It needs to know what healthy looks like, how services connect, what changed, who owns the response and what happened in similar incidents.
It should keep one clear incident picture from the first signal to recovery. Engineers can see what is happening, which possible causes fit the evidence, what remains unknown and what to check next.
Start with one repeated investigation and run the agent beside the on-call engineer. Let it automate evidence gathering and reports while the team fixes the missing context that real incidents expose. Add another incident type when the first one is dependable.
Engineers remain responsible for decisions and production changes. The agent helps them reach those decisions with better information and less searching.
Related articles
- How Can You Trust AI Agents With Your Infrastructure? — The context, operating experience, permission boundary and recovery path an infrastructure agent needs before it prepares a change.
- Why GitOps is so hard to explain and automate. — A practical operating model for proposals, engineer approval, reconciliation and evidence across AWS and Kubernetes.
Start the Observability Agent with one useful report.
We can map what healthy looks like, connect the evidence and automate a report your engineers can review before deciding what to do.
Top comments (0)