LLM-powered features have a failure mode traditional software doesn't: they degrade without breaking. The output is still valid JSON, still 200 OK, just worse — and the regression hides until a user complains about something that "used to work."
I built TestPilot because "it worked in testing" is the most common and least useful sentence in LLM QA.
30-second takeaway
- LLM regressions don't throw errors — they drift, and drift is invisible to normal tests.
- The guard is an eval checklist tied to a version, run before every ship.
- Catching the drift at commit time beats catching it in a support ticket.
The regression nobody's watching
A prompt change, a model swap, a system-message tweak — any of them can move behavior in a way unit tests never see. The app still runs. The answers just get softer, or wrong, or off-brand. By the time it's a pattern, it's shipped to everyone.
An eval tied to the version turns that invisible drift into a visible diff you can block on, the way you'd block a failing unit test.
What the tool does
Paste a prompt and its expected behavior; get an eval checklist and a version regression guard. Built for dev and QA teams shipping LLM-powered features. Open it and try — no signup.
TestPilot runs in the browser, no account, no key.
FAQ
What is TestPilot?
A tool that turns a prompt and expected behavior into an eval checklist and a version regression guard for LLM apps. No account, no key.
Does it replace my test suite?
No. It covers the behavior-drift your normal tests can't see. The two are teammates.
Will it call the model for me?
It helps you define what to check; the runs are yours to wire into your pipeline.
Author's Process
This product was built with AI coding tools. The design fight was against turning it into another agent — it stays a guard, not a chatbot.
Top comments (0)