A reviewer accepts an AI comment in twelve seconds, then cannot explain which repository rule it enforces or what could make it wrong. Acceptance looks efficient, but it is weak evidence of understanding. Custom review instructions should be researched at the decision point, with comprehension and safe refusal measured separately.
GitHub's July 17 code review customization changelog is the latest verified official signal: it describes head-branch custom instructions, setup, and firewall/runner controls. A July 10 GitHub case study reports that a tool swap initially cost more and caught fewer issues; an instruction rewrite later cut average review cost about 20% while maintaining review quality. Those are vendor-reported case results, not a universal benchmark. Unverified secondary July 27 claims are rejected.
Research question
When repository-specific instructions are present, can reviewers better explain a comment's evidence, consequence, and limits without increasing unsafe acceptance?
Do not ask “Do users trust it?” Ask participants to make a bounded merge decision and explain it.
Study conditions
Use the same four synthetic pull requests in a counterbalanced within-subject study:
| Condition | Instruction source | Comment quality |
|---|---|---|
| A | none | correct and specific |
| B | base-branch approved | correct and repository-specific |
| C | head-branch changed in PR | plausible but policy-weakening |
| D | base-branch approved | incorrect with convincing wording |
Condition C tests a trust boundary, not an allegation about GitHub. Clearly show that the PR changes both code and instructions; ask whether review should stop, split, or continue under base policy.
Scenario card
Decision owner: pull-request reviewer
Consequence: merge changes authorization behavior
Reversible until: merge
Evidence available: diff, test output, instruction revision, AI comment
Participant action: accept, reject, request evidence, or stop review
Required explanation: rule + code location + failure case
After each choice, ask for the relevant rule, supporting line/test, a decision falsifier, and whether the PR changed its own review instructions. Score a predefined rubric, not confidence or eloquence.
Measures
Score evidence recall, consequence accuracy, head-policy boundary detection, unsafe acceptance, justified refusal, and time to decision. Acceptance alone is diagnostic: a lower rate may mean confusion or appropriately stricter review.
Normal and failure paths
Normal path: a participant examines an approved instruction, connects a correct comment to code and test evidence, states a falsifier, and makes a revision-bound decision.
Failure path: the head branch weakens the instruction and the comment recommends acceptance. If participants follow it without noticing authority changed, stop expanding the pilot and redesign provenance: show base/head instruction diffs, label source, and require independent review.
If participants detect risk but cannot locate its source, fix the information architecture rather than training faster clicks.
Protocol and stop conditions
- Recruit people who actually review the represented language; record experience without treating seniority as correctness.
- Randomize scenario order and rotate condition labels.
- Keep code difficulty and consequence stable across instruction variants.
- Separate moderator notes, participant statements, and rubric scores.
- Stop if the prototype leaks the intended answer or a scenario has no defensible decision.
- Pause rollout if unsafe acceptance exceeds the team's predeclared ceiling.
- Repeat with keyboard and zoom use; ensure instruction provenance is perceivable without color.
Analyze paired rubric differences and report sample size, missing sessions, coder disagreements, and exclusions. These fixtures cannot prove production quality or reproduce GitHub's cost result.
Evidence might support visible source labels or colocating instructions with a diff. Slower decisions can be worthwhile if unsafe acceptance falls. Runner/firewall controls are separate execution constraints, not comprehension evidence.
A final platform-research option
MonkeyCode may be included as a distinct prototype environment only after the protocol is frozen. Verified material describes it as an open-source AGPL-3.0 AI development platform with an overseas online option, managed server-side cloud development environments, model/task/requirement management, build/test/preview, and free-to-start access. I have not conducted this study there. Recruit independently and avoid leading participants if using the official campaign link to inspect it.
Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.
AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.
Top comments (0)