Most "AI testing" tools aim for the same promise: the AI runs everything and tells you it passed.
When something ships broken anyway, "the AI said it passed" is not an answer anyone accepts. A person is accountable.
git-qa is an open-source human-in-the-loop QA runner built around that fact.
An AI agent works through your test sheet; you watch the live screen and sign off each case with one key.
The evidence records who verified what.
What it does
- Reads a test sheet — a TSV you can write in any spreadsheet: step, expected result
- Drives the target — Android devices and emulators (adb), web pages (Chrome, Edge, Firefox, Safari), desktop apps on macOS and Windows, and iPhone / iPad (USB mirroring; tapping through WebDriverAgent)
- Shows you the live screen while it works, with a marker where it found the expected text
-
Lets you place the verdict with one key —
Dpass,Ffail -
Keeps the evidence —
run.jsonplus the screen at the moment of the verdict, per case
The vocabulary is the point
| Value | Meaning |
|---|---|
VERIFIED |
A person put their name on it |
AUTO_PASS |
The AI passed it. Nobody looked |
FAIL |
It failed |
BLOCKED |
Could not run |
SKIP |
Not run |
A plain PASS would merge the first two. That merge is exactly what git-qa exists to prevent.
Letting Claude Code test on a real device
git-qa ships an MCP server. An agent such as Claude Code can take screenshots, read the screen,
list what can be pressed and tap it — on Android, web, desktop or iPhone.
You can run a sheet unattended overnight (--no-ui) and review only the failures in the morning.
The pass is still yours to give.
It also refuses sheets that verify nothing: if the expected text was already on screen before the step,
the case is stopped as BLOCKED with a note on how to fix the sheet.
Try it
- Download from the distribution page (macOS signed and notarized; Windows x64 / ARM64, unsigned) and install Node 22+
- Enter a handle, pick a device / URL / desktop app, pick a test sheet
- Start, watch, press
D/F
It is beta and MIT-licensed. Bug reports and requests: Issues.
Top comments (2)
The BLOCKED rule for a sheet that verifies nothing is the part I'd copy first. We run UI automation on real iPhones through WebDriverAgent and hit the mirror image of it: a check that passes because the screen changes without the step doing anything.
We confirmed a feed swipe by hashing a screenshot before and after. On a playing video the hash differs on every frame, so a swipe that bounced back to the same video was logged as a page turn. The fix was to compare something only the event can cause (the caption and account text on the overlay), which sounds close to what your expected-text marker does.
Two things from the iPhone side that may matter for the WDA path: system prompts sit outside the app's element tree, so a tap can report success while a prompt swallowed it, and the same app build can open a different screen on a small model than on a big one.
Does git-qa handle "the screen changed, but not because of the step" on screens that animate, or is that left to the sheet author?
Thanks, the video-hash story is exactly the trap this is about.
Short answer: git-qa never compares screenshots or asks "did the screen change", so an animating screen cannot produce a false pass on its own. Each row has one expectation, "X is shown", and the only check after the step is whether X is in the screen text. On iPhone that text comes from OCR of the real screen, not from the app's element tree.
The one thing the tool enforces is: if X was already on screen before the step, the row is BLOCKED as a sheet defect, not a product failure. "The screen changed, but not because of the step" is left to the author: pick text that only this step can produce. Your overlay-caption fix is the same move, just chosen by hand. We don't try to detect causation.
Your comment did expose one hole, though. The "already there" check reads the screen once, before the step. Text that appears on its own, like a video caption, a rotating banner or a timed notice, can be absent at that read and show up afterwards for reasons unrelated to the step, and the row would pass. The tool cannot catch that, so I have added it to the sheet-writing guide as a rule: never use text that changes by itself as an expectation.
On the iPhone points: a tap never "reports success" in git-qa. The step runs, then the expectation decides. A system prompt that swallowed the tap shows up as the expected text not appearing, and the prompt itself is in the screenshot and the OCR text that the human reviewer sees. Targets are found by on-screen text rather than coordinates, so layout shifts between small and large models are usually fine. A genuinely different screen ends up BLOCKED or FAIL with the screenshot attached, and the human makes the call. The final verdict is always a person's.