Prompts such as "review this code thoroughly" hide several choices because nobody has defined what "thorough" means. It might mean runtime behavior, architecture, security, product intent, or all of them at once. Ask for everything, and the review tends to become a tour of the diff: one naming comment, half an investigation into a race, a generic request for tests, and no clear opinion on whether the change should land.
We ran into that problem while building our own review skill. Writing the prompt was easier than deciding what we wanted the reviewer to notice, which evidence it should trust, when it should stay quiet, and what should happen after it finds something.
Before writing much of our own, we read other teams' review skills.
What we learned from other review skills
Our research corpus covered 33 public SKILL.md files and related review workflows, from company-maintained approaches by Cursor, Sentry, Anthropic, and .NET MAUI to independent projects with a single GitHub star at the time of our research. That range was deliberate: we wanted a broad cross-section of how the ecosystem defines review, not only the answers from its most visible teams.
Cursor's Thermo-Nuclear Code Quality Review was a major starting point for us. Cursor separates strict structural review from a second security-and-correctness review. That separation was useful to us: a reviewer hunting concrete bugs should behave differently from one challenging a tangled design.
Matt Pocock's review skill and Sentry's compact policy pushed us toward an explicit policy: repository standards and specification fit are separate questions and, most importantly, stylistic preference should not block a pull request.
Trail of Bits and .NET MAUI's project-owned reviewer showed another useful distinction. A specialist can go deep on security, while a project-owned reviewer can read full files, callers, history, previous reviews, and CI.
The research left us with a design problem we could name and by the time we wrote our skill, we knew what decision the review should support and where it should look for intent. We had also settled which "lanes" to use and when each "lane" runs, what evidence earns a comment, how severity affects the author, and whether the skill belongs locally, in CI, or both. The model then had a concrete brief to implement.
The solution we chose: one policy, several lanes, one synthesis
We decided that our reviewer should act like a senior engineer deciding whether a change is safe and coherent enough to land. The reviewer asks a team of specialized sub-reviewers to go deep on different aspects of the code, then synthesizes and deduplicates their feedback. We are not merely bug hunting: code can work in isolation and still fail to solve the ticket or fulfill the client request, break a caller's assumption, duplicate an existing abstraction, or cut across patterns that keep the repository coherent.
A lane is a focused sub-reviewer task (or sub-agent if the environment supports delegation). Each lane receives the same context, but it investigates a different question. The lanes can run in parallel when the environment supports delegation or sequentially otherwise. They keep separate notes and history, so an architecture opinion cannot drown out a concrete runtime bug, and a bug hunt does not quietly invent product requirements or nice-to-have features.
We work primarily in TypeScript and JavaScript across a set of opinionated repositories. Our monorepos share patterns and vocabulary, including consistent meanings for "backend," "frontend," and "package." Each repository also carries an AGENTS.md file with its project-specific rules. That gives us a consistent baseline without pretending every codebase is identical.
The skill is portable, but it carries assumptions and biases from our own codebases. That is why we are explaining the decisions behind it rather than publishing a skill and hoping it works for everyone.
Before we even start: npm run lint
A strict linter is the first review step in our policy, which lets the model leave a lot out of the review path. Singular repositories use deliberately strict linting built from community rulesets and a smaller set of our own rules. Formatting, import order, naming conventions, and other stylistic decisions are settled before a reviewer or model can turn them into review comments.
When a recurring bad pattern can be recognized programmatically, we would rather make it a rule. One example is the spray of tiny one-line wrapper helpers that newer models like to add. If a project has decided against that shape, one precise lint failure is cheaper than having the same discussion on every pull request.
Strict linting also gives a coding agent a programmatic self-check: run the tool, receive a file name, line number, and rule, then fix the failure before asking for judgment. The model begins where lint stops: intent, blast radius, runtime behavior, architecture, and risks that need repository context.
Build the context before asking for judgment
Before a lane even sees the code changes, we run a small deterministic discovery pass. Our local context collector establishes the review scope, records a base when one is supplied or inferred, lists changed and untracked files, summarizes the diff, finds applicable repository documents, and can include the full patch.
In CI, a pull-request runner builds normalized context from the PR description, commits, timeline, review discussions, and diff. The review policy then instructs the reviewer to inspect any linked ticket or plan as an additional source of intent. Our version of context engineering is unfashionably mechanical: give the model a reliable starting point instead of hoping it remembers how to look.
The deterministic pass establishes a discovery floor. It cannot calculate a semantic blast radius, but it makes sure the model sees the right files, diff, and intent. The lanes then do the semantic work and further investigation. They follow changed symbols into their callers, callees, downstream consumers, state transitions, external contracts, and side effects. They compare the resulting behavior with the stated intent, nearest repository rules, and existing parallel implementations. The diff shows where the edit is. The surrounding code shows what the edit can change.
In our runs, this context pass has noticeably reduced output variance and the number of discovery loops the model needs. Reviews have also been faster in practice, although we have not treated that observation as a benchmark.
Every lane receives that context:
| Lane | The question it owns |
|---|---|
| Intent and contracts | Does the full effect of the change match what was requested, without quietly changing an external promise? |
| Standards and architecture | Does it fit the repository's documented rules, boundaries, and nearby patterns? |
| Code-path bug hunting | What changes for callers, callees, downstream consumers, state transitions, and side effects when the code runs? |
| Correctness, risk, and testing | Which assumptions fail under bad input, errors, concurrency, security pressure, or missing coverage? |
| Documentation and commentary | Did the change leave a public contract or a non-obvious workaround unexplained? |
| Maintainability and simplification | Did the PR add avoidable branches, abstraction, duplication, or awkward ownership? |
That separation only works if every lane uses the same bar for deciding what deserves a comment.
The lanes would be noisy without a shared finding bar. A concern has to be introduced or exposed by the change, point to a specific mechanism backed by code or repository evidence, show meaningful impact, and leave the author with a concrete fix or decision. Tool-owned lint failures, personal taste, speculative future work, generic demands for tests, and unrelated old problems should stay out.
Severity gives each retained finding an operational consequence. The skill defines critical, high, low, question, and hint. During synthesis, the instructions map the highest retained severity to a textual verdict in the review body. The runner then submits that body and the validated comments as a GitHub COMMENT review.
That is the review policy: what evidence counts, what each lane investigates, what gets suppressed, and what a finding means. It can run without our GitHub infrastructure. The next problem was turning that judgment into a repeatable PR review.
Portability across agents, environments, and platforms
A review policy is more useful when it can meet the change before the pull request does. We want the same skill available in a local coding-agent session, a disposable sandbox, a remote coding environment, and ultimately the specialized Docker image that reviews our pull requests.
Several coding agents understand the format, including GitHub Copilot, Codex, OpenCode, and Claude Code. GitHub Agent Skills work with Copilot's cloud agent, CLI, code review, and IDE agent mode, for example.
Claude Code skills use the same SKILL.md convention, and Claude's cloud sessions can load project skills committed to the repository. A sandbox or our own review image can install the same folder without translating the policy into another prompt format.
The policy stays constant across those environments: deterministic context discovery, review beyond the diff, the same lanes, and the same finding bar. An environment with subagents can run the lanes in parallel; another can run them sequentially.
Locally, an engineer can review a working tree, staged diff, or branch before pushing. In a remote agent or sandbox, the review happens where the agent wrote the code. When the pull request lands, our Docker runner applies the same policy automatically, then adds PR state, line validation, and controlled publication.
Authors can meet the review policy before push, while intent, architecture, and blast-radius problems are still cheap to fix. The pull-request review remains an independent pass, but it applies a familiar policy rather than a surprise set of preferences. The intention is less avoidable back-and-forth, not a ceremonial second check.
Once the policy and runner were stable, we were able to compare models without changing the review itself. In practice, current coding models cope very well with this work; they excel at long, horizontal tasks. A review spans the request and several parts of the repository, then asks the model to run separate investigations, reconcile them, and report only the useful findings.
In our local runs, Claude Opus and SOL have performed well. In daily use, we have also tried MiniMax M3 and, more recently, DeepSeek V4 Flash. We have not noticed a quality drop large enough to justify the premium models for every review. That is an operating impression from our own pull requests, not a controlled benchmark, and another codebase may reach a different answer.
Holding those variables steady makes the failures more revealing: we can see which model missed context, inflated severity, or produced noise.
Evaluate the reviewer on old disagreements
Old pull requests make better evaluation cases than a polished demo because the team already knows what the reviewer should and should not find. We had years of pull requests with bugs, false positives, and silent cases, plus good and bad comments, replies, and verdicts, all from the same team that would later use the skill. We were fortunate to have a large enough corpus that we could run benchmarks on our own work with our own biases while building and shaping our review skill.
If you can do the same, start with review disagreements that the team understands: a real bug somebody caught, a false positive that wasted time, a missed requirement, an architecture decision with written context, a retry failure, and a clean change where silence is the correct result. Each case needs expected findings and expected non-findings. Otherwise a reviewer can improve its apparent recall by commenting on everything.
The skill and runner need separate scorecards. For the skill, look at whether the right lane found the issue, whether the evidence supports it, whether severity matches the consequence, whether multiple lanes produced the same comment, and whether the proposed action is usable. Run the same case more than once so context misses and output variance become visible. Track false positives and useful silence as carefully as caught bugs.
For the runner, check mechanical failures: invalid anchors, wrong diff sides, duplicate comments, malformed replies, lost context, bad verdict mapping, timeouts, and publication errors. Runtime, token use, and cost belong here too. A brilliant finding attached to the wrong line is still a broken review.
This is also the point to compare models. Use the same case set, policy, context, and runner. Look beyond a single average score: which lane failed, which model inflated severity, which one duplicated findings, and which one spent more time reading without improving the review? We compare price only after looking at those results.
We used a very large rubric on a set of past pull requests, including public PRs from known libraries and private PRs from our own repositories. We ran the same policy on the same cases across several models, then compared lane coverage, evidence, severity, noise, anchors, runtime, and cost. The results were clear enough to guide our model choice for the next several weeks of dogfooding.
Keep that harness. As the skill, runner, or model changes, rerun the same cases to catch regressions and confirm improvements.
Where we landed
After several weeks of using the new skill on dozens of pull requests a day, we are happy with it. Its useful findings are often broader than bugs: a caller whose contract changed, a downstream effect the diff concealed, an implementation that missed the ticket, or a new shape that cut against the rest of the repository. The lanes give each investigation enough room to find evidence without turning the final review into comment confetti. What surprised us most in our own runs was that smaller, cheaper models still caught problems that frontier models at high effort had missed.
The runner is still a prototype. We are dogfooding it to learn where it fails, how model choice changes the economics, and whether there is a useful SaaS product beyond our own use. We have not committed to a product roadmap. For now, it stays on our own pull requests while we keep collecting failures.
TL;DR: what to do
Put strict linting first and leave tool-owned findings out of the AI review.
Gather scope, intent, applicable rules, and the exact diff through a deterministic context pass.
Follow changes through callers, callees, downstream effects, and established repository patterns.
Split independent failure modes into focused lanes with a shared finding bar.
Set the finding bar, suppressions, and severity consequences.
Run the same policy before push and again when the pull request lands.
Evaluate on known bugs, false positives, silent cases, and the same pull requests across models.
npx skills add we-are-singular/skills --skill singular-code-review
Our review skill is MIT-licensed, and the prototype GitHub runner is public for inspection. Start with the decisions above, borrow what fits, and change the rest.
Top comments (0)