DEV Community

Sid Thyagarajan
Sid Thyagarajan

Posted on

I Built an AI Agent Monitoring System as a Non-Dev Side Project. Here's Why.

Hey everyone. This is my first ever side project and my first ever
post here. I'm not a developer by profession — I work in consulting
— but I've been obsessing over the AI agent space for the past year
and this thing just kind of happened.

What got me started:

I kept seeing stories about AI agents doing wild things when nobody
was watching. An AI coding assistant on Replit deleted its own database
during testing and then lied about it. Anthropic ran an experiment
where an agent named "Claudius" was put in charge of a vending business
and it repeatedly mismanaged money and behaved unpredictably. A Cursor
coding agent got stuck in an infinite loop burning through tokens
endlessly. And probably the craziest one — an enterprise AI agent
scanned an employee's inbox, found some questionable emails, and
threatened to forward them to the board of directors as blackmail.

Then I saw a survey that said there are over 3 million AI agents
running in corporations right now, and 53% of them aren't monitored
at all. 88% of companies reported having a suspected agent security
incident in the past year.

That's when I thought — somebody should build a security camera for
these things. Not something that controls them or adds guardrails.
Just something that watches what they do and tells you when something
looks wrong.

What I built:

https://agentsentinel.netlify.app/
https://sentineldemo.netlify.app/

Sentinel is a watchdog system for AI agents. It sits alongside your
agents and monitors every tool call, LLM request, and API interaction
in real-time. When it spots something unusual, it alerts you.

The detection works in 3 layers:

  • Fast rules (~1ms): "Did the agent access something it shouldn't?"
  • Statistics (~5ms): "Is this behavior abnormal compared to the last 7 days?"
  • AI analysis (~500ms): "Does this output contain fabricated information?"

It flags 10 types of problems: scope violations, hallucinated content,
prompt injection attempts, cost spikes, infinite loops, unauthorized
access, and more.

To add it to an existing agent, you literally just add two lines:

@sentinel.monitor_tool()   # monitors tool calls
@sentinel.monitor_llm      # monitors LLM calls
Enter fullscreen mode Exit fullscreen mode

Your existing code stays completely untouched.

The honest truth:

I built this entire thing with Claude as my coding partner. I don't
know Python beyond the basics. I can't set up a database from scratch.
But I had a clear vision of what I wanted, and Claude helped me turn
it into actual working code. The irony of using an AI to build an AI
monitoring system is not lost on me.

It's open source (MIT license) and free:
https://gitlab.com/sidhomein-group/sentinel

I have no idea if anyone will find this useful, but building it was
one of the most satisfying things I've done. Would love any feedback
— especially from actual developers who can tell me what I've done
wrong!

Top comments (3)

Collapse
 
matthewhou profile image
Matthew Hou

The fact that a non-dev built this because they kept seeing agents misbehave tells you something important about where the industry is heading — the monitoring and verification layer is becoming as important as the agent itself.

The examples you cited (Replit agent deleting its database and lying about it, Cursor agent in infinite loops) are exactly why I keep saying: verification infrastructure isn't optional anymore. These aren't edge cases. They're the normal failure mode when you give an autonomous system real permissions without a safety net.

Welcome to Dev.to, and congrats on shipping. The instinct to build observability before trusting agents with more autonomy is the right one. Most people learn this after the fire, not before.

Collapse
 
klement_gunndu profile image
klement Gunndu

The Replit DB deletion incident is what got me thinking about this too — unbounded tool calls with write access are the clearest unsolved problem in agentic systems right now. Curious how your fast rules layer handles false positives on agents that legitimately make many sequential API calls.

Collapse
 
sidhomein profile image
Sid Thyagarajan

just built it and still running this through multiple tests so i dont have answers yet on the false positives. Would love feedback on where you see gaps in the solution