Cover: a real qa-probe run against our own stack, cropped to the summary - internal product detail withheld.
We are building LightShield, a SIEM that is in active demo right now. We built
most of it pair-programming with an AI coding assistant wired in over MCP - it
ran our stack, read the errors, and patched its own code. For a small team that
is a superpower. Until an endpoint failed.
Here is the loop we kept hitting. A route returns a 500, or a 404, or an empty
[]. The assistant looks at the status code and announces the cause with total
confidence. Then it rewrites a handler that was never broken - because a status
code is not a cause, and it had nothing else to go on. So it guessed, and it
guessed wrong, and the diff made things worse.
The thing is, that empty [] had at least six possible causes:
- the database was empty (nothing seeded)
- a feature flag was off
- a contract mismatch between the frontend and the backend
- an auth token that never got attached
- a 428 precondition
- a schema drift
Same symptom, six different fixes. We could bisect to the real one. The AI could
not - it had no ground truth, so it manufactured one.
So we built qa-probe
It analyzes the app, probes the live endpoints, and classifies each failure with
a root cause and a fix hint. Three decoupled, cached phases:
qa-probe analyze # parse source + OpenAPI -> route graph
qa-probe probe # hit live endpoints (HTTP/SSE/WS), record evidence
qa-probe report # classify root cause -> HTML / Markdown / JSON / AI-context
# or just: qa-probe run
It has adapters for FastAPI, Express, Next.js, tRPC, GraphQL, and a generic
fallback, so it discovers your routes instead of you hand-listing them.
The part that actually fixed our problem: every result is falsifiable
Each result carries the evidence (the real request, a bounded response sample,
the timing), a root cause from ~25 categories, and a calibrated confidence -
high, medium, or none. When it cannot tell, it returns none instead of
bluffing. No neural network, no black box - transparent rules plus per-endpoint
stat memory, so you can always read why it landed on a verdict. An AI
consuming this needs to verify the claim, not trust a vibe.
What changed when we fed it to the AI
qa-probe mcp # exposes 8 tools to Claude, Cursor, any MCP client
The assistant stopped reasoning from a status code and started reasoning from
evidence: "empty database, high confidence, here is the response that proves
it." It seeded the DB instead of rewriting the handler. It fixed the right
layer. The guessing basically stopped.
It helped us debug faster. It helped the AI more - because an AI is only as good
as the evidence you hand it, and "the endpoint is failing" is not evidence.
What else it caught once it was in our pipeline
- Security checks (read-only) - auth-bypass, privilege-escalation, PII leaks
- Logic bugs via response assertions
- Adaptive baselines that flag an endpoint deviating from its own history
- Write-flows - opt-in CRUD chains with guaranteed cleanup, off by default
- A feedback loop - label a verdict once and it remembers
Where it fits
- A CI gate that fails the build with a reason, not just a number
- A pre-release smoke test against staging
- The thing you point your AI assistant at so it debugs instead of hallucinating
Try it, break it, contribute
It is early and it is open. The fastest way to help:
- Run it against your own API and tell me what it found - especially if it got a root cause wrong. A wrong verdict with the evidence attached is the most useful bug report I can get, because the classifier is just rules and I can fix the rule.
- File an issue for anything broken or confusing: https://github.com/LS-SIEM-LLP/qa-probe/issues
- Send a PR. Good first contributions are new framework adapters (Hono, NestJS, Fastify), new root-cause rules, and new MCP tools. The adapter and rule interfaces are documented in CONTRIBUTING.md, and there is a checklist on every PR.
One housekeeping note: contributions are sign-off based (DCO) - commit with
git commit -s so the project's licensing stays clean. That is the only hoop.
Get it
We built it for ourselves. It worked well enough that we cleaned it up and
released it under Apache-2.0.
npm i -g qa-probe
Built by LS-SIEM LLP. If you run it against your own API, I would genuinely like
to know what it found - that feedback is how the rules get sharper.
Top comments (2)
This highlights a failure mode that's easy to miss: the model isn't hallucinating because it's incapable of debugging, it's hallucinating because it's being forced to explain symptoms without enough evidence.
An HTTP 500, 404, or empty response can have multiple root causes, but AI systems often feel pressure to provide an answer instead of acknowledging uncertainty. Adding a layer that gathers evidence and assigns confidence before remediation is a much more reliable approach.
We've seen similar patterns while working on AI-assisted debugging and support workflows at IT Path Solutions. The biggest improvements usually come from improving the quality of signals available to the model rather than changing the model itself. Better observability, structured diagnostics, and confidence scoring tend to reduce incorrect fixes far more effectively than prompt tweaks.
The "return none instead of bluffing" principle is especially important. In production environments, knowing what the system doesn't know can be more valuable than a confident but incorrect diagnosis.
Thanks for taking the time to write this out, really thoughtful comment. And yeah, you nailed it. We spent way too long assuming the model was the weak link, and it just wasn't. It was getting a 500 and being asked to explain it with nothing else to work from.
That observability point is the whole thing for me. Instead of trying to make the model smarter about a status code, we handed it the actual request and response plus a deterministic root cause, and the guessing mostly stopped on its own.
The "say none instead of bluffing" part took the longest to get right, oddly enough. But a real "I don't know" is what stopped it from rewriting a perfectly good handler when the database was just empty.
Curious what the signal-quality wins looked like on your end at IT Path. Tracing, structured diagnostics, something else? Always interested in how other people are tackling this.