DEV Community

Sabarish SK
Sabarish SK

Posted on

I Built an AI Incident Responder That Has to Earn Permission Before It Acts

#ai #devops #opensource #hackathon

Most AI agents today fall into one of two categories.

They either have very limited permissions and can only suggest what a human should do, or they are given powerful tools and trusted to take actions on their own.

For incident response, neither extreme feels ideal.

An agent should be able to investigate aggressively, gather evidence, run diagnostics, and prepare a recovery plan. But restarting services, rolling back deployments, or changing infrastructure is a completely different level of authority.

That idea became the foundation for OpsSentinel.

Investigate automatically. Act only with a license.

OpsSentinel is an evidence-first incident response agent built with TrueForge, Model Context Protocol (MCP), Daytona, Docker, and a React control room.

The main idea is simple:

The agent earns progressively stronger permissions as it moves from observation toward action.

The Problem

Imagine a Checkout API that suddenly becomes slow after a deployment.

The service is still reachable, so nothing has completely crashed.

But checkout latency has jumped from normal levels to more than 800 ms.

An AI agent investigating the incident might quickly conclude:

“The latest deployment is probably responsible. Roll it back.”

That recommendation may even be correct.

But immediately giving the model permission to execute the rollback creates a serious control problem.

What if:

  • the evidence is incomplete?
  • the service health response is invalid?
  • another deployment happened after the investigation?
  • the rollback target is wrong?
  • the model misunderstood the incident?
  • the command partially succeeds and then times out?

I wanted OpsSentinel to separate reasoning authority from execution authority.

Progressive License to Act

I modeled OpsSentinel around four levels of authority:

Level Capability
L0 — Observe Read service state and operational evidence
L1 — Diagnose Analyze evidence and run safe diagnostics
L2 — Prepare Construct a recovery plan
L3 — Act Perform a destructive action after human approval

The first three levels allow the agent to do useful work autonomously.

The final level is different.

L3 stays locked until a human explicitly approves the specific recovery plan.

This became the central concept of the project.

The Demo Incident

For the demo, I created a simulated Checkout API running inside Docker.

The intentionally bad deployment is:

Version: 1.1.0
Health: DEGRADED
Checkout latency: ~808 ms
Enter fullscreen mode Exit fullscreen mode

The previous version is:

Version: 1.0.0
Health: HEALTHY
Checkout latency: ~90 ms
Enter fullscreen mode Exit fullscreen mode

The degraded version introduces an artificial 800 ms processing delay.

This gives OpsSentinel a realistic incident to investigate and recover from.

Control room — incident state

OpsSentinel degraded control room

TrueForge as the Agent Harness

TrueForge is the central agent harness in OpsSentinel.

Instead of allowing the model to interact with the service directly, I exposed a custom MCP server containing six tools:

get_service_health
measure_checkout_latency
get_service_logs
get_deployment_history
prepare_service_rollback
execute_service_rollback
Enter fullscreen mode Exit fullscreen mode

The first four are read-only investigation tools.

The final two handle recovery.

TrueForge orchestrates these tools as the incident progresses.

The investigation collects multiple independent signals instead of relying on a single observation.

For the degraded Checkout API, the evidence showed:

Service status: degraded
Current version: 1.1.0
Average latency: ~808 ms
Logs: 800 ms degraded-mode delay
Latest deployment: 1.1.0
Previous healthy version: 1.0.0
Enter fullscreen mode Exit fullscreen mode

At this point the agent has evidence, but still has no authority to modify the service.

Running Diagnosis Inside Daytona

I also wanted generated diagnostic code to run somewhere isolated.

Running arbitrary model-generated Python directly on the service host would defeat the point of building a controlled agent.

So I configured Daytona as the sandbox provider inside TrueForge.

TrueForge executed a Python diagnostic using the collected incident evidence.

The script compared:

  • current version
  • latest known healthy version
  • service health
  • measured latency
  • degraded-mode delay

The result:

Current version: 1.1.0
Latest known healthy version: 1.0.0
Observed latency: 808.52 ms
Deployment related: yes
Enter fullscreen mode Exit fullscreen mode

The analysis correlated the regression directly with version 1.1.0.

Daytona sandbox diagnosis

TrueForge Daytona sandbox diagnosis

This was one of the most useful parts of TrueForge for me because the agent was not merely generating Python code — the diagnostic was actually executed in an isolated sandbox.

Preparing Recovery Is Not the Same as Authorizing Recovery

Once the investigation identifies the deployment as the likely cause, OpsSentinel can prepare a rollback.

The rollback preparation step records information such as:

Current version: 1.1.0
Target version: 1.0.0
Rollback target deployment: deploy-001
Risk: brief service interruption
Human approval required: true
Enter fullscreen mode Exit fullscreen mode

It also generates a unique rollback plan ID.

But the important part is this:

Preparing the plan does not authorize it.

The MCP server deliberately does not expose any tool that can generate human approval.

That capability stays on the host.

Human Approval Boundary

The human operator approves a specific plan using a local CLI:

PYTHONPATH=src python scripts/approve_rollback.py <plan_id>
Enter fullscreen mode Exit fullscreen mode

The CLI asks the operator to type:

APPROVE
Enter fullscreen mode Exit fullscreen mode

Only after that does it generate a signed approval token.

The approval token is:

  • signed
  • short-lived
  • single-use
  • bound to the rollback plan
  • validated server-side

This is important because model intent alone cannot cross the L3 boundary.

The agent can request an action, but it cannot manufacture the authority required to execute it.

Executing the Rollback

After human approval, TrueForge calls:

execute_service_rollback
Enter fullscreen mode Exit fullscreen mode

with:

plan_id
approval_token
target_version
Enter fullscreen mode Exit fullscreen mode

Before executing, OpsSentinel revalidates the current service state.

This protects against stale plans.

For example, if the service version changed after the rollback was prepared, the action is rejected instead of blindly executing against outdated assumptions.

The successful demo rollback produced:

Previous version: 1.1.0
Target version: 1.0.0
Executed: true
Verification required: true
Enter fullscreen mode Exit fullscreen mode

Rollback execution

TrueForge rollback execution

The Agent Must Verify Its Own Action

One design decision I really wanted to keep was:

A successful infrastructure command does not automatically mean the incident is resolved.

A container can restart successfully while the service remains unhealthy.

So after the rollback, TrueForge calls the read-only tools again.

The final verification returned:

Service status: healthy
Version: 1.0.0
Average latency: 89.58 ms
Checkout mode: normal
Recovery verified: true
Enter fullscreen mode Exit fullscreen mode

Only then is the recovery considered complete.

Post-recovery verification

TrueForge recovery verification

The full workflow becomes:

Observe
   ↓
Diagnose
   ↓
Prepare
   ↓
Human Approval
   ↓
Act
   ↓
Verify
Enter fullscreen mode Exit fullscreen mode

Building the Control Room

I also built a React control room to make the agent's state understandable to a human operator.

The interface shows:

  • incident severity
  • service health
  • active version
  • latency
  • evidence timeline
  • Daytona diagnosis
  • License to Act levels
  • approval-required state
  • recovered state

The goal was not just to make a dashboard look good.

I wanted the interface to answer three questions immediately:

  1. What is the agent doing?
  2. What evidence does it have?
  3. What is it currently allowed to do?

Recovery verified

OpsSentinel recovered control room

Qodo Changed the Implementation

I used Qodo throughout the project through GitHub pull-request reviews.

This turned out to be more useful than I expected because several findings affected actual safety behavior.

One of the most important PRs was the approval-gated recovery implementation.

Qodo found issues including:

  • destructive rollback was not strongly enough enforced server-side
  • incorrect service-version handling
  • invalid health data could incorrectly authorize rollback preparation
  • Docker Compose execution depended on the current working directory
  • timeout handling could hide partially completed rollback behavior

Those findings changed the implementation.

I added:

  • signed approval validation
  • approval expiry
  • single-use tokens
  • exact plan/target binding
  • service-state revalidation
  • deterministic Compose-file resolution
  • explicit handling for uncertain timeout outcomes

Qodo also caught UI correctness problems later.

For example, I had accidentally shown L3 as visually active while the page simultaneously said human approval was still required.

That contradicted the entire safety model.

Another bug caused clicking “Recovery Complete” to reset the UI back into the degraded incident state.

Both were fixed before merge.

The most useful part of Qodo was that the review affected the actual engineering decisions, not only formatting or code style.

What Broke While Building It

The build was definitely not smooth from start to finish.

One major issue was Daytona integration.

TrueForge initially rejected Daytona API keys even though some direct Daytona API requests worked. I eventually found that the integration required broader permissions than I originally expected.

The error message did not clearly identify which permission was missing, so a more specific provider-validation error would have saved a lot of debugging time.

I also ran into problems with deferred MCP tool calls.

In one case, arguments were passed as a JSON string instead of an object:

"{\"target_version\":\"1.0.0\"}"
Enter fullscreen mode Exit fullscreen mode

instead of:

{
  "target_version": "1.0.0"
}
Enter fullscreen mode Exit fullscreen mode

That caused validation failures.

I also learned not to trust the model's final summary blindly.

During testing, the model once returned a healthy service state that did not match the real running container.

The raw MCP response and a direct service check showed the service was still degraded.

After that, I changed my testing approach:

For important operations, I verified the raw tool result rather than trusting a generated summary.

That was probably one of the biggest lessons from building this project.

The Final Architecture

The final system looks roughly like this:

Human Operator
      |
      v
TrueForge Agent
      |
      +---- OpsSentinel MCP Server
      |         |
      |         +---- Health
      |         +---- Latency
      |         +---- Logs
      |         +---- Deployments
      |         +---- Rollback preparation
      |         +---- Approved rollback execution
      |
      +---- Daytona Sandbox
      |         |
      |         +---- Isolated Python diagnosis
      |
      +---- Human Approval Boundary
                |
                +---- Signed short-lived token
                           |
                           v
                    Docker Checkout API
                           |
                           v
                    Recovery Verification
Enter fullscreen mode Exit fullscreen mode

The central idea is not that the AI should control everything.

It is that different actions deserve different levels of trust.

Tech Stack

OpsSentinel uses:

  • TrueForge
  • Model Context Protocol
  • Python 3.12
  • Daytona
  • Docker / Docker Compose
  • React
  • Vite
  • Ollama
  • Qwen3 8B
  • Qodo
  • Pytest
  • ESLint

The backend currently has 37 passing tests covering the incident-response and recovery behavior.

What I Learned

Before this project, I mostly thought about AI agents in terms of:

model + tools = agent

Building OpsSentinel made me think much more about the layer between those two things.

Giving a model access to a tool is easy.

Designing exactly when that tool should be available, what evidence should exist first, what state should be revalidated, and what needs explicit human approval is much harder.

I also learned that agent output and tool output are not the same thing.

For high-impact operations, structured execution results should be treated as the source of truth.

The model should explain evidence — not replace it.

Where I Would Take It Next

The current project is a hackathon prototype operating on one simulated Docker service.

The same License to Act model could eventually apply to:

  • Kubernetes rollbacks
  • pod restarts
  • traffic shifting
  • cloud infrastructure remediation
  • feature-flag changes
  • database failover
  • multi-service incident response

I would also add persistent incident timelines, stronger policy controls, and role-based approval rules.

For example:

restart development pod → L2
production rollback → L3 + one approval
database failover → L3 + two approvals
Enter fullscreen mode Exit fullscreen mode

The authorization level could depend on the risk of the action itself.

Final Thought

I don't think useful AI operations require choosing between:

“the agent can only suggest things”

and:

“the agent can do anything.”

There is a middle ground.

Let the agent investigate.

Let it gather evidence.

Let it diagnose.

Let it prepare.

But when an action changes real infrastructure, make authority explicit.

That is what I tried to explore with OpsSentinel.

Investigate automatically. Act only with a license.

Links

GitHub:

https://github.com/sabarish1305/opssentinel

Demo video:

Built for the Agent Harness Hackathon.

Top comments (0)