DEV Community

Cover image for My AI Agent Rejected 90% of Its Own Ideas, and That's Why It Won πŸ†
Sanika Marathe
Sanika Marathe

Posted on Edited on

My AI Agent Rejected 90% of Its Own Ideas, and That's Why It Won πŸ†

This is a submission for the MLH x DEV Writing Challenge

What I Built

Everyone wants AI that improves itself. Almost nobody talks about the scary part: an agent that changes itself can just as easily make itself worse, and confidently tell you it got better.

At TAMU Datathon 2026, I built an agent around one rule:

No change is kept until it proves itself on data it has never seen.

It finished on the podium πŸ†. Here's how it works.

Layer 1: an agent that writes its own rulebook

Instead of hiding everything inside a black box, the agent learns plain-English if-then rules from examples. Then it puts every rule on trial:

  • Keep: the rule improves results on unseen examples.
  • Revise: the rule hurts as a main rule, so it gets demoted to a backup that only speaks up when nothing stronger applies.
  • Reject: everything else.

It runs this in rounds, and here's my favorite part: it decides when to stop. Once a round no longer earns its keep, the agent freezes itself. No babysitting.

The result is a rulebook a human can actually open, read and argue with.

Layer 2: an agent that writes its own AI "skills"

Some decisions need real reading comprehension, not pattern-matching. So the agent looks at where its own rules are unsure and keep failing, and hands those mistakes to a teacher model. The teacher writes a short, reusable "skill," and the base model uses that skill to double-check only the shaky answers.

Then the same trial happens again: a skill survives only where it measurably helps. In more than half of the places I tried, the AI made things worse, so the agent threw those skills away automatically.

No model weights were touched. Everything the agent "knows" is readable text and settings it learned and tested itself.

Demo

Flowchart of the self-improvement loop: propose a change, test it on unseen data, then keep, revise or reject it, and repeat until further improvement isn't worth it, then freeze and ship.

Technologies I used

  • Databricks: served the LLMs through its AI Gateway. The free tier's rate limits forced me to get smart: batching requests, retrying only on rate-limit errors, and caching every response so the same call never ran twice. Constraints made the design better.
  • GitLab: version control, plus CI validation on every push.
  • Claude Code: my AI pair programmer for building and debugging fast.
  • Python: for everything else.

Hackathon Experience

TAMU Datathon 2026 was pure energy from the opening minute: whiteboards filling up with half-baked ideas, people huddled over laptops sketching architectures, and that moment of "wait… what if the agent checks its own work?" that turned into this whole project. The brainstorming was easily the best part: chasing an idea, watching it fail, and coming back with a better one a few hours later.

Then came the late-night chaos:

  • My first model scored worse than just guessing the most common answer. One tiny smoothing bug made rare answers look likely, so it confidently picked the weirdest option every time.
  • My first AI layer made results worse. That failure is where the core idea, "prove it or lose it," came from.
  • PowerShell misread one of my command-line flags right as the final run started. Nothing like adrenaline at the finish line.

Finding out we had made it to the podium made every one of those 3am bugs worth it. πŸ†

What I'm Taking Away

  1. Self-improvement without verification is just self-deception.
  2. Bigger models don't automatically win. Learning from real examples beat raw general knowledge.
  3. AI is a great reader and a bad guesser of conventions. Use it where it's strong, and gate it everywhere else.
  4. Knowing when to stop is a feature. The best agent isn't the one that tries hardest; it's the one that knows when more effort isn't worth it.

Kudos to my teammates Nidhi, Mihir and Ravi, and huge thanks to the TAMU Datathon organizers for an amazing event!

Top comments (1)

Collapse
 
nishtha_pathak_8c3f89f93b profile image
Nishtha Pathak •

Love this! Worth the win!