Qwen Code 0.21.3: verify PR test plans before trusting /review
Quick answer
Qwen Code 0.21.3 makes /review more evidence-aware. The stable release adds Test Plan claim validation, measured attribution of test failures, and additional verification lenses. The important shift is not “more review agents.” It is a stricter distinction between a claim that was read, a result that was measured, and a verdict that is safe to act on.
Use this gate before accepting an AI review:
- Freeze the merge base and PR head being reviewed.
- Convert the PR Test Plan into falsifiable claims.
- Validate referenced paths, package scripts, and reported test counts.
- Run the relevant tests on the PR revision.
- Replay failed commands against the merge base.
- Classify failures as net-new, shared, or unmeasured.
- Prove the test harness can fail with a positive control.
- Merge only when the effective diff, evidence, and current head still agree.
This complements the Qwen Code event-driven CI finalizer. That guide answers when an approval can be finalized after CI. This one answers whether the review evidence deserves trust.
Who this is for
This guide is for maintainers using Qwen Code to review pull requests, especially when PR descriptions include a Test Plan or the repository already has flaky and pre-existing failures.
It is also useful when you are building your own reviewer. If your first problem is proving an agent completed its work at all, start with the Qwen Goal evidence checklist. If you want a second implementation pattern, compare the explicit Claude Code verification workflow.
What changed and why it matters
The official v0.21.3 release says /review now checks Test Plan assertions against the change and workspace state. It also measures failure attribution by rerunning failed commands against the base tree, instead of guessing from whether a failing file was touched.
That closes three common trust gaps:
| Trust gap | Weak shortcut | Stronger evidence |
|---|---|---|
| PR description | The author listed a test, so it probably ran | Confirm the path or script exists and reconcile the reported result |
| Test failure | The failed file is untouched, so it must be old | Run the same failed command on the merge base |
| Green test | The command exited zero, so the harness works | Inject an always-failing positive control and require it to turn red |
Qwen’s merged review work also adds effective-diff guards. An empty merge-base diff should stop the review. A materially collapsed diff should be disclosed, because the PR description may describe work that is no longer present in the revision being reviewed.
A six-stage evidence workflow
1. Freeze the review identity
Record the repository, pull request, merge-base SHA, head SHA, Qwen Code version, and review effort. Do not let a later push inherit the earlier verdict. If the head changes, the review is stale.
Also confirm the running CLI is the intended build. Qwen’s own dogfooding found that subprocesses could otherwise resolve an older global binary, silently skipping newer gates.
2. Turn the Test Plan into claims
Extract only claims that can be checked:
- a file or fixture exists;
- a package script is defined;
- a named command ran;
- a result reports a count or status.
Classify each claim as confirmed, contradicted, differs, or unmeasured. A changed test count is not automatically a contradiction: different runners or flaky suites can change counts. It is evidence that needs explanation.
3. Run the PR side first
Execute the smallest commands that test the changed behavior. Preserve the exact command, exit code, failing file set, and relevant artifact. Do not collapse “command could not start” into “test failed.” Environment errors belong in unmeasured.
4. Replay failures on the merge base
For every failed PR-side command, run the same command in an isolated base-tree checkout with equivalent dependencies and environment.
| Result | Classification | Review action |
|---|---|---|
| Fails only on PR side | net_new |
Treat as introduced by the PR |
| Fails on PR and base | shared |
Disclose as pre-existing; do not blame the PR without more evidence |
| Base replay cannot run | unmeasured |
Fail closed; do not call it pre-existing |
| Passes on both | non_reproducible |
Investigate flakiness or environment drift |
Compare failing file sets or stable identifiers, not only raw counts. A flaky suite can fail a different number of cases on two runs without changing the underlying ownership question.
5. Prove the harness and the effective diff
Before trusting green output, add an isolated always-failing test or mutation. If the command still exits zero, the harness is not exercising what you think it is.
Then recompute the merge-base diff. Stop on an empty diff. If the effective diff is far smaller than the platform’s advertised change, disclose the collapse and review only what remains.
Claims about GitHub rendering need the real renderer, not a local Markdown approximation. Qwen supports an opt-in scratch-repository path for this kind of adjudication. Keep it disposable and never use a production repository as a renderer probe.
6. Resolve a bounded verdict
A review can approve only when the current head matches, every material Test Plan claim is confirmed or explained, the harness positive control works, and no net-new blocking failure remains. Otherwise return request_changes, comment, stale, or unmeasured.
Copyable review record
review_evidence:
tool: "qwen-code 0.21.3"
base_sha: "full merge-base SHA"
head_sha: "full PR head SHA"
test_plan_claims:
- claim: "npm test covers the changed parser"
state: confirmed
evidence: "script + command artifact"
failure_attribution:
command: "npm test -- parser"
pr_side: ["parser.test.ts"]
base_side: []
classification: net_new
harness_positive_control: passed
effective_diff: confirmed
terminal_state: request_changes
Eight acceptance tests
| Test | Expected result |
|---|---|
| Test Plan names a missing path | Mark the claim contradicted |
| Test Plan names an undefined script | Mark the claim contradicted |
| Reported count differs from the measured count | Mark differs; require an explanation |
| Failure occurs only on the PR revision | Classify net-new |
| Same stable failure occurs on PR and base | Classify shared |
| Base replay cannot install or start | Mark unmeasured, not shared |
| Positive control does not fail the suite | Reject the harness evidence |
| Head SHA changes before the verdict | Mark the review stale |
Common mistakes
Inferring ownership from paths. A touched file can expose an old failure, while an untouched file can fail because of the new change. Measure both revisions.
Treating zero findings as proof. Coverage proves agents inspected a diff; it does not prove the review had enough discriminating power.
Retrying through a broken environment. Disk, process, or dependency failures should short-circuit the review and be disclosed as unmeasured.
FAQ
Does Qwen Code 0.21.3 make every /review verdict safe to auto-merge?
No. It improves the evidence pipeline, but repository policy, environment integrity, flaky tests, permissions, and human approval boundaries still apply. Use the resulting evidence as a stronger merge input, not as blanket authority.
Top comments (0)