One thing I keep running into when reviewing a PR is this: before writing a test, I first have to work out what the change is actually supposed to prove.
My first instinct was to generate E2E test drafts from diffs. It looked useful on paper. In practice, a lot of the output was obvious: click something, then check the success screen.
So I am trying to start one step earlier.
I have been building QAMap, a small local CLI that reads commits and diffs, tries to describe the behavior a change touches, and points out QA questions that may be missing around failures, boundaries, and state changes.
It runs locally. It does not upload source code or make LLM calls.
It is still early, and it definitely gets things wrong. Right now, I am more interested in finding those weak spots than pretending it can understand every product domain.
https://github.com/IvoryCanvas/qamap
https://github.com/ivory-code
Top comments (3)
This is a good point. Tests are only as useful as the question they are answering.
Before the test, someone has to understand the change, the risk, the affected behavior, and what failure would actually look like. That is where a lot of PR review work hides.
AI can help, but only if it preserves the reasoning path instead of jumping straight to a green check.
Thank you. That distinction is exactly what I’m trying to explore.
I agree that a green test without a traceable reasoning path can create false confidence. I’m working on making QAMap show how a diff leads to affected behavior, risk, and failure scenarios before producing an optional test draft.
I really appreciate how clearly you articulated this.
That is a useful direction. The optional test draft matters because it keeps the reasoning layer separate from the artifact. If QAMap can show the path from diff to affected behavior before producing code, the test stops looking like magic output and starts looking like a traceable consequence.