Ouroboros AI: We Made a Fake Agent Break Itself, Then Built Something That Fixes It

By Sagar Maurya & Disha Sonowal — WeMakeDevs SigNoz Hackathon

If you're just landing on this series for the first time, here's the short version of where we've been.

In our first post, we installed a tool called SigNoz for the very first time, so we could watch what a piece of software is doing while it runs — which parts are slow, which parts fail, and why. We built a tiny fake AI agent, and just by looking at a picture of its actions laid out in order, we could instantly see that one single step was eating up 85% of the total time. That was the moment this whole thing clicked for us.

In our second post, we went further. We built a slightly smarter fake agent and asked SigNoz a real question: "does this get slower when it has more information to look through?" We didn't calculate the answer ourselves. We just told SigNoz what to group and average, and it handed us a table proving the pattern, clean as anything.

Both of those were practice. Small, safe experiments before the real hackathon clock started ticking. This post is about what we actually built once it did — and honestly, it's the post we're proudest of, because it's the one where we stopped just watching our fake agent and started building something that does something about it.

We're calling the project Ouroboros AI. Repo's here if you want to poke around: github.com/mauryasagar/ouroboros-ai

---

The problem that was quietly bugging us

Right after we published that second post, something about it kept nagging at us.

We'd asked SigNoz a good question and gotten a good answer: yes, more documents means a slower response, and it climbs in an almost perfectly straight line. We felt clever about it for about a day. Then we looked at what we'd actually built and realized... nothing happened next. We saw the pattern, said "yep, makes sense," and closed the browser tab. If this had been a real product used by real people, someone would still have had to notice that pattern themselves, write it up, wait for an engineer to have time, and fix it — probably days later, and only if they happened to be looking at the right dashboard at the right moment.

That bothered us more the more we thought about it. Watching something break in slow motion isn't the same as fixing it. So a simple, slightly stubborn question became the whole idea for this project:

What if the system could notice its own problem and fix itself, without a person in the middle at all?

Not "send someone a faster alert." Actually close the loop.

A few words before the story, so nothing feels confusing

We're going to use a handful of words a lot in this post. If you already know them, skip ahead. If you don't, here they are in plain terms:

• Trace — a record of everything that happened during one single request, start to finish.
• Span — one step inside that trace. Four things happened? Four spans.
• Alert — a rule you set up that says "if X happens, tell someone."
• Webhook — instead of "tell someone" meaning a text message or an email, it can mean "call this web address automatically." That's the piece that let a computer react instead of a human.
• MCP (Model Context Protocol) — a standard way for an AI model to reach out and use real tools — in our case, letting an AI ask SigNoz real questions instead of us clicking around the dashboard ourselves.

That's genuinely all the vocabulary you need. Everything else is just those five ideas, wired together.

What we actually decided to build

We sat down together and sketched this out on a shared doc before writing a single line of code. Five steps, in order:

1. Chaos — a fake AI agent runs constantly, and every so often, on purpose, it messes up.
2. Traced — every single step it takes gets recorded and sent to SigNoz, live.
3. Alert — SigNoz watches all these traces coming in, notices the mess-up pattern, and automatically fires off a webhook — no human clicks anything.
4. Heal — a separate little program receives that webhook and retries the failed thing with better settings. Still no human.
5. Explain — an AI "Sidekick" you can literally chat with, which reads the real SigNoz data through MCP and tells you, in plain English, what just happened.

Steps 1 and 2 were a more serious version of what we'd already built in post two. Steps 3, 4, and 5 didn't exist anywhere yet. That's where basically the entire week went.

Step one: giving our fake agent something real to fail at

Our agent from post two never actually failed — it just got randomly slower or faster each time. That's fine for a screenshot, but it's useless for an alert. You can't tell a computer "notice when things feel kind of slow sometimes." You need something crisp: a clear line that gets crossed.

So we gave our agent one specific way to break: context overload. In plain terms — our agent normally looks through somewhere between 1 and 8 "documents" to answer a question. But 20% of the time, on purpose, we let it get buried under 15 documents at once, which is way more than it can handle cleanly.

def retrieve_context(query, is_chaos_mode=False):
    with tracer.start_as_current_span("retrieve_context") as span:
        # CHAOS MODE: 15 docs, NORMAL: 1-8 docs
        context_docs = 15 if is_chaos_mode else random.randint(1, 8)
        span.set_attribute("llm.context_docs", context_docs)
        time.sleep(random.uniform(0.1, 0.4))
        return f"Mock context data ({context_docs} docs)"

We picked this specific failure on purpose, not randomly. Real AI agents that search through documents actually run into this exact problem — feed them too much information at once, and they get slower, cost more, and often don't even answer better for it. We wanted our fake failure to be one that real systems actually have, not something we made up just to have a demo.

When the overload happens, we don't just log it as "a bit slow." We mark it as an actual error, with a clear message SigNoz can watch for:

if context_docs >= 15:
    logging.error(f"CONTEXT_OVERLOAD: Agent failed. Used {tokens} tokens, cost ${round(cost, 6)}, took {delay:.2f}s.")
    span.set_status(Status(StatusCode.ERROR, "Context Overload"))

One small thing from post two turned out to matter a lot here: we'd mentioned back then that just printing something to your terminal doesn't automatically send it to SigNoz — you have to specifically wire up your logs to go there too, separately from your traces. This time, we made sure that was working properly from day one, because our whole alert depended on SigNoz actually being able to see that error message, not just a slightly-slower-than-usual span.

We let this fake agent run forever in the background, quietly sending a new trace every 200 milliseconds, so there was always something fresh happening while we built the rest.

Step three and four: teaching the system to fix itself

This is the part that didn't exist in either of our earlier posts, and honestly, the part we're most excited about.

The good news is, we didn't have to build anything fancy to make SigNoz watch for the failure — it already does that with alert rules, the exact same feature we used in post one to watch for slow calls. The only difference is where the alert points. Instead of sending a Slack message or an email, we pointed it at a webhook — a plain web address that SigNoz calls automatically the moment it sees the problem.

On the other end of that web address, we built a tiny program whose only job is to sit there, wait, and fix things when it gets called:

@app.route('/webhook', methods=['POST'])
def webhook():
    data = request.json or {}
    alert_name = data.get("alertname", "Unknown Alert")
    print(f"🚨 ALERT RECEIVED FROM SIGNOZ: {alert_name}")

    result = optimized_llm_call("Auto-healed request")

    span.add_event("AUTO-HEAL SUCCESS", attributes={
        "message": "Request auto-healed with optimized parameters",
        "status": "healed"
    })
    return jsonify({"status": "healed", "result": result}), 200

The actual fix is simple on purpose — it retries the request with a smaller, safer amount of information instead of the overload amount. The part we spent real time thinking about wasn't the fix itself, though. It was this question: should the fix be invisible, or should it leave a trace of its own?

We almost let it be invisible — quietly patch things up in the background and move on. But that felt like exactly the thing we were trying to get away from in the first place: a fix that only lives in someone's head, not in the system. So we made the healer trace its own retry too, tag it clearly as an auto-heal, and record an event that says, plainly, "this got fixed automatically." Now the healing itself shows up in SigNoz right next to the failure that caused it. Nobody has to remember it happened. The system remembers for you.

Step five: teaching an AI to explain all of it in plain English

Here's where post two's biggest lesson came back around. Back then, we discovered that if you tag your data with the right detail, SigNoz can answer real questions for you — no manual math required. The bigger question that kept nagging at us afterward was: what if you didn't even need to know which button to click, or which filter to type? What if you could just ask, like you're talking to a person?

That question is what led us into MCP — the protocol that lets an AI model reach out and actually use tools, instead of just talking. Neither of us had touched MCP before this hackathon, so before writing any code, we spent a real evening just reading about how it works. That reading turned out to save us a lot of pain later, even though it didn't save us from all of it.

What we ended up building is a small chat app we call the Sidekick. You type a question, and behind the scenes:

You type a question
   → Groq (the AI model) decides which tool it needs
   → it asks SigNoz for the real data through MCP
   → the real numbers come back
   → the AI reads them and answers you in plain English

SigNoz's MCP connection offers over 40 different tools it can use — way more than we needed — so we limited our Sidekick to five: list services, get traces, get logs, get metrics, get one specific span. Simple, focused, and cheap to run.

Getting this actually working took two separate evenings of head-scratching, and both problems were the sneaky kind — nothing crashed, things just quietly didn't work.

The first problem: our AI would just never use any tools. It would answer vaguely instead of actually checking real data, no matter how directly we asked it something like "what services are running right now." We spent a while assuming the AI itself was being stubborn, tweaking our instructions to it, before we finally printed out exactly what we were sending it — and found out the tool descriptions we were passing along were completely empty. Turns out SigNoz describes its tools using a label called inputSchema, while the AI model we were using expects a label called parameters. We were only checking for one of the two. The fix was a single line of code, but finding it meant working backwards through several layers of "this should be working" first:

schema = tool.get("inputSchema") or tool.get("parameters") or {"type": "object", "properties": {}}

The second problem was stranger. The very first request would work perfectly, and then the second one would get flatly rejected — every time. It took a while to notice that the server hands you an invisible little "session ID" the first time you connect, and expects you to send it back on every request after that, like a wristband at an event. Miss that, and it treats you like a stranger who just walked in:

sid = response.headers.get("Mcp-Session-Id")
if sid:
    self.session_id = sid

On top of that, some answers came back in a slightly unusual format made for streaming data, instead of a plain, simple response — so our little connector had to learn to handle both.

None of this felt exciting while it was happening. It was two nights of adding print statements, staring at raw technical details, and slowly ruling things out one at a time. But the moment both fixes were in, the payoff was immediate. Post two's big win was clicking through a filter and a group-by to get a table. This time, the same kind of insight just... showed up, after typing a normal sentence:

"Which services are currently being monitored in SigNoz?"

{
  "answer": "The services currently being monitored are:\n- auto-healer-service — 1 trace in the last hour (avg ~494ms)\n- deep-agent-service — 688 traces in the last hour (avg ~1.63s)",
  "tools_used": ["signoz_list_services"]
}

We kept the AI's instructions short and strict on purpose — answer in a few sentences, use the real numbers you found, don't waste time asking for logins it already has. It's meant to feel like a fast, useful teammate answering a quick question, not a general chatbot rambling.

The moment it actually came together

Here's the part that felt like more than just code working.

We finally let all three pieces run at the same time and just watched. In one window, the fake agent quietly sending traces every 200 milliseconds, like it had been doing for days. In another, the Sidekick sitting there, waiting for a question. And then, in the third window — completely on its own, with nobody typing a single command — the healer suddenly printed:

🚨 ALERT RECEIVED FROM SIGNOZ

We hadn't triggered that. SigNoz had caught the overload pattern by itself, in the live stream of data, and called the webhook on its own. That's the moment this stopped being a demo we were operating and started being a system that was actually running itself.

About thirty seconds later, we asked the Sidekick what had just happened, half-expecting to have to nudge it toward the right answer. We didn't need to. It described the overload, the automatic retry, and the corrected numbers, pulled straight from the same live data we'd watched land in SigNoz seconds earlier. That was the entire five-step loop, closing itself, start to finish, with nobody standing in the middle of it.

What we're taking away from all three posts

Post one taught us that just seeing where the time goes changes how you think about a problem. Post two taught us that tagging the right detail turns a dashboard into something you can actually have a conversation with. This project is where those two lessons stopped being separate little discoveries and turned into one real habit: instrument things not just to log what's easy right now, but to answer the question you'll actually want answered later — because that same detail is what lets an alert fire cleanly, and what lets an AI explain the whole story back to you without you writing a single extra line of code for it.

If someone asked us what we'd tell ourselves before starting this hackathon, it's that one sentence. Everything else — the failures, the debugging nights, the late realization that our first agent needed to actually break instead of just wobble — all came from following that one idea seriously enough to build around it.

---

Full code, the landing page, and everything else is in the repo: github.com/mauryasagar/ouroboros-ai

By Sagar Maurya & Disha Sonowal — WeMakeDevs SigNoz Hackathon