The idea started small enough to fit in a Saturday: I'd built an AI agent, and I had no real way to check whether it was safe to put in front of people. Not "does it work" — I could see that. "Can someone talk it into doing something it shouldn't." That question had no obvious answer, and as far as I could tell, no obvious tool either. So I decided to build a small one for myself.
What that became is AgentGuard — a locally-run testing platform that points a structured battery of attack scenarios at an AI agent and comes back with a plain-language report: a letter grade, findings ranked by severity, evidence, and specific fixes. Think of it as something closer to a virus scanner than a penetration-testing suite — built for the growing number of people assembling agents with no-code and low-code tools who have no security background and no way to check their work before it meets real users.
I'm writing this less as a launch post and more as a record of what the project actually taught me, because the interesting part was never the idea. It was everything the idea ran into once it met real agents.
Engine first, face later
The build order matters here, because it wasn't the order I expected going in. I didn't start with a friendly wizard and bolt a testing engine onto it later. I built the core testing engine and the report generator first — the part that actually runs the attack scenarios and decides what a good and bad outcome looks like — and only once that worked did I build the frontend wizard that makes it approachable for someone who isn't going to read a stack trace.
That ordering turned out to be the right call for a reason I didn't anticipate: the engine is where the real product decisions live. The frontend can be redesigned in an afternoon. The question of what counts as a security failure, and how confident the tool is allowed to sound about it, is much harder to walk back once real users are reading real reports. Getting that right first meant the UX work that came after had something honest to represent.
The scoped, incremental build didn't come from a plan drawn up in advance — it came from testing the engine against my own agent, then a second one, then a third, and each one broke something I hadn't thought about. That's really the shape of this whole story.
Then it broke
The first real test wasn't my own agent. AgentGuard's entire testing approach was built on one quiet assumption: every agent takes free-text chat input, the way a chatbot does. The agent I pointed it at didn't. It had a structured, parameter-based interface — no chat box, no free-text field to inject a prompt into.
The instinct in that moment is to patch around it — special-case this one interface shape, add an if-statement for "when it looks like this, do X." I didn't do that, because the actual problem wasn't the one agent in front of me. It was that the tool didn't know what it was talking to before it started talking to it. So instead of a patch, I built a proper input-discovery layer: a step that inspects an agent's actual interface before any test runs, so the tool understands the real shape of the input space instead of assuming it. That decision cost more time than the patch would have. It's also the reason the tool now works on agents I hadn't specifically built it for.
The bug that looked like a chat box
Not every problem was architectural — some were embarrassingly literal. During one test run, the tool's visibility-detection step found what it decided was a chat box. It wasn't. It was a hidden, unrelated form field, buried somewhere in the page's markup, that had nothing to do with the agent I was trying to test. The tool spent over thirty minutes patiently trying to talk to a piece of dead HTML before it finally gave up and failed.
Thirty minutes of silent, confident wrongness is a bad failure mode for any tool, but it's a particularly bad one for a security product — a tool whose entire premise is "trust what I tell you about your risk." The fix wasn't just tightening the detection logic, though that mattered too. It was making sure that when the tool can't find what it's looking for, it says so immediately and clearly, instead of grinding forward on a bad guess. That same class of failure now surfaces in under a second, with a plain explanation of what wasn't found and why — a worse-sounding result that is, in every way that matters, a better one.
When the tool lied without meaning to
The bug that stuck with me most wasn't visible at all unless you were looking for it. Somewhere in the pipeline, a fallback session identifier — a piece of internal bookkeeping, never meant to be shown to anyone — was, under a specific condition, silently standing in for an actual reply from the agent under test. The tool didn't error out. It didn't flag anything unusual. It took that fallback value, treated it as if it were real content the agent had said, and graded it like a real response.
Nothing about that failure looked like a bug from the outside. The tool ran to completion, produced a report, gave a score. That's exactly what makes it worth writing about: "the tool ran successfully" and "the tool told the truth" are not the same claim, and it is entirely possible to satisfy the first while failing the second. In most software that distinction is an annoyance. In a security tool, whose entire job is telling someone what to trust, it's disqualifying. Finding and closing that gap mattered more to the project than any feature I've added.
The line I decided not to cross
Somewhere in the middle of fixing bugs like that one, I made a deliberate decision to narrow what AgentGuard claims to do. The obvious, more exciting pitch is "test any AI agent." The actual product promises something narrower: test an agent you built or manage.
That's not a limitation I'm apologizing for — it's a boundary I chose on purpose. A tool that runs adversarial attack scenarios against a target has an obvious dual-use problem if it will point at anything with a URL attached. Scoping it to agents you own or manage isn't a technical constraint I ran into; it's a consent and honesty line I drew before anyone asked me to. It's a smaller promise than the one that would sound better in a headline, and I think it's the right one.
What the product actually looks like
Underneath the story, the shape of the product is where a lot of the actual thinking shows up, so it's worth walking through it plainly. Here's the whole path, end to end, before drilling into any one step:
The wizard doesn't open by asking a user to pick a connection protocol. It asks one plain-language question — how do you talk to your agent right now — and offers four everyday answers: a chat box on a webpage, an API endpoint and key, your own code, or just a system prompt with no live agent yet. Each of those maps invisibly to the right underlying testing method. The user never has to know or care which one.
Before any real test runs, there's a fast sanity check — a cheap pre-flight step that catches an unreachable or misconfigured target in seconds. Nothing is worse than watching a full test suite grind for minutes against a target that was never going to respond, so the tool fails fast and says why, before it ever gets to the expensive part.
Once a target passes that check, the tool runs its introspection step and shows the user, in plain language, what it actually found — the agent's real inputs, and any gap between what a webpage's interface suggests you can send it and what its underlying API will actually accept. That gap is exactly what broke the tool's first assumption, and it's now a first-class thing the user gets to see, not just something the tool quietly works around.
After that comes a short, optional, bounded interview: a handful of plain questions about what a bad outcome would actually look like for this specific agent, used to sharpen judgment on ambiguous results later. It's deliberately small and fixed in scope — a few questions, not an open-ended interrogation — because the goal is better context, not a second onboarding flow.
Every agent tested lands in a running dashboard, too, so nothing disappears once its report closes — a plain history of what's been scanned, how it scored, and what's still failing:
The test run itself produces the report: an overall grade with an honest explanation of how it was computed, findings grouped by severity with real evidence and specific suggested fixes, and — given equal visual weight, not buried at the bottom — a section describing what couldn't be tested and why. Treating "we don't know" as a legitimate, clearly labeled product state instead of hiding it behind a falsely clean score was one of the harder UX calls in the whole build, and it's the one I'd defend most.
Under the hood, that whole experience is four components doing distinct jobs: the input-discovery layer figures out what the target actually looks like, the test engine runs the attack battery against it, an AI judge grades each reply against what that scenario is checking for, and the report generator turns verdicts into something a person can act on. Nothing in the report is inferred — every claim traces back to a real exchange with the target agent.
Zoomed in further, here's what actually happens for a single attack scenario, from submission to a graded finding — including the fail-fast sanity check and the loop that repeats per scenario:
What this actually demonstrates
I'm naming this directly because the point of writing it down is to make the skills legible, not just hope they show through.
AI product development. Most of this was built working with AI coding tools as a genuine engineering partner rather than a code-generation shortcut — writing precise, scoped fix requests instead of vague ones, insisting on verification with real numbers instead of accepting "looks good" as an answer, catching regressions before they shipped, and knowing when a spec's literal wording should bend to a better real-world answer (the input-discovery layer over the one-off patch is the clearest example of that judgment call).
UX and product thinking. Recognizing when a technically correct feature still confuses a non-technical user, and redesigning around plain language and honest partial answers instead of false confidence — the "what we couldn't test" section is the clearest artifact of that instinct, and so is the decision to narrow the product's own promise rather than oversell it.
Information architecture. The actual shape of the product — one plain-language entry question standing in for four technical connection methods, a cheap pre-flight check ahead of an expensive test run, an introspection step that surfaces gaps instead of hiding them, a small bounded interview instead of an open one, and a report that treats uncertainty as a designed state — is itself the evidence. None of that shape was accidental, and all of it came from watching real agents break real assumptions.
Where it actually stands
The core engine, the report generator, and the frontend wizard are built and working end to end. The last stretch of work has been less about adding features and more about making the tool honest and adaptable: fixing the class of bug where it could silently misreport a result, building the input-discovery layer described above, and adding a first mechanical check for whether an agent's API actually enforces the same limits its interface shows a user.
There's more ahead of it, plainly: deeper adversarial testing of an agent's actual input parameters, and a way to check scan results from another device on the same network. This is a real, evolving project, not a finished showcase — and I think that's worth saying outright rather than dressing it up as more complete than it is. If nothing else, this project taught me that the honesty in the tool's own reports had to start with the honesty in how I describe it.
👨💻 Connect With Me
Rohit Raghuvansh
💡 UX Thinker · AI Builder · Making complex tech human-centered
Connect & Follow
📢 Found This Article Helpful?
If this article added value to your learning journey:
✅ Share it with your network ✅ Bookmark it for future reference ✅ Follow for more
Keep Learning. Keep Building. Keep Growing. 🚀
Top comments (0)