Most "AI testing" tools aim for the same promise: the AI runs everything and tells you it passed.
When something ships broken anyway, "the AI said it passed" is not an answer anyone accepts. A person is accountable.
git-qa is an open-source human-in-the-loop QA runner built around that fact.
An AI agent works through your test sheet; you watch the live screen and sign off each case with one key.
The evidence records who verified what.
What it does
- Reads a test sheet — a TSV you can write in any spreadsheet: step, expected result
- Drives the target — Android devices and emulators (adb), web pages (Chrome, Edge, Firefox, Safari), desktop apps on macOS and Windows, and iPhone / iPad (USB mirroring; tapping through WebDriverAgent)
- Shows you the live screen while it works, with a marker where it found the expected text
-
Lets you place the verdict with one key —
Dpass,Ffail -
Keeps the evidence —
run.jsonplus the screen at the moment of the verdict, per case
The vocabulary is the point
| Value | Meaning |
|---|---|
VERIFIED |
A person put their name on it |
AUTO_PASS |
The AI passed it. Nobody looked |
FAIL |
It failed |
BLOCKED |
Could not run |
SKIP |
Not run |
A plain PASS would merge the first two. That merge is exactly what git-qa exists to prevent.
Letting Claude Code test on a real device
git-qa ships an MCP server. An agent such as Claude Code can take screenshots, read the screen,
list what can be pressed and tap it — on Android, web, desktop or iPhone.
You can run a sheet unattended overnight (--no-ui) and review only the failures in the morning.
The pass is still yours to give.
It also refuses sheets that verify nothing: if the expected text was already on screen before the step,
the case is stopped as BLOCKED with a note on how to fix the sheet.
Try it
- Download from the distribution page (macOS signed and notarized; Windows x64 / ARM64, unsigned) and install Node 22+
- Enter a handle, pick a device / URL / desktop app, pick a test sheet
- Start, watch, press
D/F
It is beta and MIT-licensed. Bug reports and requests: Issues.
Top comments (1)
The BLOCKED rule for a sheet that verifies nothing is the part I'd copy first. We run UI automation on real iPhones through WebDriverAgent and hit the mirror image of it: a check that passes because the screen changes without the step doing anything.
We confirmed a feed swipe by hashing a screenshot before and after. On a playing video the hash differs on every frame, so a swipe that bounced back to the same video was logged as a page turn. The fix was to compare something only the event can cause (the caption and account text on the overlay), which sounds close to what your expected-text marker does.
Two things from the iPhone side that may matter for the WDA path: system prompts sit outside the app's element tree, so a tap can report success while a prompt swallowed it, and the same app build can open a different screen on a small model than on a big one.
Does git-qa handle "the screen changed, but not because of the step" on screens that animate, or is that left to the sheet author?