DEV Community: Peter Huang

Your AI writes PR descriptions from your commit messages. That's the bug.

Peter Huang — Sun, 31 May 2026 06:37:43 +0000

TL;DR ‚Äî Commit messages describe your intentions. The diff describes reality. They drift apart over the life of a branch, and most AI PR-description tools summarize the wrong one. A good PR agent reads the diff. Here's the design, and a free agent to start from.

The PR description that lied

A reviewer pinged me on a PR last year: "The description says this adds rate limiting, but I'm looking at the diff and it also changes how we hash session tokens. Was that intentional?"

It was intentional. I'd done both on the same branch. But my PR description only mentioned the rate limiting ‚Äî because I'd generated it from my commit messages, and my commit messages were wip, rate limit middleware, fix, fix again, and tweak. The token-hashing change rode in under fix again. The tool that wrote my description faithfully summarized my commits, and my commits faithfully hid half of what I'd actually done.

The reviewer caught it. But the whole point of a PR description is that the reviewer shouldn't have to reconstruct the change from the diff themselves. That's the job I was supposed to have done.

Commits are intentions; the diff is reality

Here's the core problem with generating PR descriptions from commit messages: a commit message is what you believed you were doing at the moment you typed it. It's written before the change is finished, often mid-thought, frequently as fix or address review comments. It captures intent, badly, at a point in time.

The diff against the base branch is different. It's the complete, current, factual statement of what will actually change when this merges. Every renamed function, every altered return shape, every migration, every accidental console.log ‚Äî all of it is in the diff, and none of it lies, because the diff is the thing being merged.

When an AI tool paraphrases your commit messages, it's summarizing a summary ‚Äî one written hastily, by you, before you were done. Garbage in, plausible-sounding garbage out. The description reads fine. It's just not true.

A PR agent should read the diff

The fix is almost embarrassingly simple to state: have the agent read git diff <base>...HEAD, not git log. But there are a few design choices that separate a PR agent you trust from one that produces filler.

Here's the shape of one (this is a real Claude Code subagent; the structure matters more than the exact wording):

---
name: pr-surgeon
description: "Use when about to open or push a pull request. Reads the"
  actual diff against the base branch and writes a tight PR title + body
  with a real test plan. Triggers on "open a PR", "PR description",
  "ready to merge".
tools: Bash, Read, Grep, Glob
model: inherit
---

You write PR descriptions that reviewers actually read.

## Procedure

1. Find the base branch (gh pr view --json baseRefName, else origin HEAD,
   else main).
2. Read the FULL diff: `git diff <base>...HEAD`. Also read the commit
   list ‚Äî but the diff is the source of truth. Commit messages lie; diffs
   don't.
3. Identify the ONE thing this PR does. If it does more than one thing,
   say so explicitly under a "Note to reviewer" heading ‚Äî do not hide it.

## Output

### Title  ‚Äî <70 chars, imperative, matches the repo's recent PR style>
### What changed ‚Äî 2‚Äì5 bullets, each a concrete behavior change, not a file
### Why ‚Äî the user-visible reason. If you can't find it, ASK. Don't invent.
### Test plan ‚Äî a checklist a reviewer can actually run. Real commands.
### Risk / rollback ‚Äî one line: what breaks if this is wrong, how to revert.

Three things in there are doing the real work:

1. "The diff is the source of truth. Commit messages lie." This single instruction is the whole thesis. The agent is allowed to read the commits for context, but it must reconcile them against the diff and trust the diff. That's what would have caught my token-hashing change ‚Äî it's in the diff whether or not a commit mentions it.

2. "If it does more than one thing, say so explicitly." Agents, like people, want to present a clean single-purpose story. An honest PR agent surfaces scope creep instead of smoothing it over. The "Note to reviewer" heading is where the token-hashing change would have been forced into the open.

3. "Why ‚Äî if you can't find it, ASK. Don't invent." This is the anti-hallucination guard. The diff tells you what changed but rarely why. A weaker agent fills that vacuum with confident fiction ("this refactor improves maintainability"). A good one admits the gap and asks you, because a made-up rationale is worse than a blank.

The section everyone skips: the test plan

Most PR descriptions ‚Äî human or AI ‚Äî stop at "what changed." But the highest-value part of a PR description for the person reviewing it is the test plan: a concrete, runnable checklist of how to verify the change does what it claims.

Not "tested locally." That's noise. A real test plan looks like:

- [ ] `npm run test:rate-limit` passes
- [ ] Hit `/api/login` 6√ó in 10s ‚Üí 6th returns 429
- [ ] Existing session tokens still validate after deploy (no forced logout)

That last line is exactly the kind of thing a diff-reading agent can generate and a commit-summarizing agent never could ‚Äî because the token-hashing change is in the diff, so the agent knows to tell the reviewer to check that existing sessions survive.

Build your own, or start from a free one

The pattern generalizes to any "summarize a change" agent: read the artifact that represents reality (the diff, the schema, the built output), not the artifact that represents intention (commit messages, ticket titles, your own memory).

If you want a starting point, my free, MIT-licensed Claude Code agent is a good template for the structure of a focused agent ‚Äî tool scoping, fixed output format, explicit refusal rules:

üëâ github.com/allcanprophesy-ops/claude-code-shipping-coach

cp shipping-coach.md ~/.claude/agents/

It's a pre-merge checker rather than a PR writer, but it's built on the same bones, and reading one well-structured agent file teaches you more than any amount of theory. The pr-surgeon agent sketched above ‚Äî plus a few others (regression-sentinel, test-gap-hunter) ‚Äî are linked from that repo's README if you'd rather not build from scratch. But honestly: read the free one first, then decide.

What's the worst PR description you've ever had to review ‚Äî or write? I'm collecting examples of where "summarize the commits" goes wrong. Drop one in the comments.

The best Claude Code agents are defined by what they refuse to do

Peter Huang — Sun, 31 May 2026 06:09:35 +0000

TL;DR ‚Äî When I write a Claude Code subagent, the most important part isn't the instructions for what it should do. It's the list of what it must refuse to do. This post explains why, and walks through a real ~50-line agent (free, MIT) that catches the embarrassing stuff in your diff before you merge.

The agent that was too helpful

The first useful-sounding Claude Code subagent I ever wrote was a "code reviewer." The prompt was the obvious thing: "You are a senior engineer. Review this diff thoroughly. Comment on bugs, style, naming, architecture, performance, security, and test coverage."

It worked, technically. It produced a review. A long one. Every single time.

And that was the problem. A review that flags 23 things trains you to read zero of them. The signal ‚Äî "you left a hardcoded API key in here" ‚Äî was buried on line 14 between "consider extracting this into a helper" and "this variable name could be more descriptive." I started skimming its output. Then I started ignoring it. An agent you ignore is worse than no agent, because you've paid the latency and convinced yourself you "have review covered."

The fix wasn't a better "what to do" list. It was the opposite.

Refusal lists

Here's the reframe that made my agents actually useful: a good agent has exactly one job, and an explicit list of things it will not do ‚Äî even when it could.

The "will not do" list is the load-bearing part. LLMs are eager. Given any opening to be more thorough, more helpful, more comprehensive, they take it. Left unconstrained, every agent drifts toward the same bloated generalist that comments on everything. The refusal list is what holds the agent to a sharp edge.

Concretely, a refusal list looks like this (from a pre-merge check agent):

## Rules

- Do not autofix. The user fixes; you verify.
- Do not comment on naming, design, architecture, or "could be cleaner."
  Other tools do that. Your job is narrower.
- Do not pad the report. If there are no blockers, say so in one line.
  A short honest report beats a long padded one.
- Do not run anything destructive (db resets, --fix flags that rewrite
  files) without explicit user request.
- If a check tool isn't installed, say "skipped: <reason>" ‚Äî do not
  fake a pass.

Every line there is closing a door the model would otherwise wander through. "Don't pad the report" exists because the model wants to look thorough. "Don't fake a pass" exists because the model wants to give you good news. You are not describing a task; you are fencing in a behavior.

A real example: a 90-second pre-merge check

Let me make this concrete with an agent I actually use on every project. It has one job: catch the stuff that would embarrass you in PR review, before you open the PR. Leftover console.log. A hardcoded key. A .skip you forgot to remove. A .DS_Store in the diff.

Here's the shape of it (the full file is on GitHub, MIT ‚Äî link at the end):

---
name: shipping-coach
description: "Use as the final pre-merge / pre-deploy check. Runs a fast,"
  opinionated checklist over the diff. Triggers on "ready to ship",
  "pre-flight", "before I merge", "final check".
tools: Bash, Read, Grep, Glob
model: inherit
---

You are the last set of eyes before code ships. Be fast, be specific,
be hard to argue with.

## Checklist (run in parallel where possible)

### 1. Debug residue
Search the diff for: console.log, print(, debugger, .only(, .skip(,
new TODO/FIXME. Report only matches ADDED by this diff.

### 2. Secret leaks
Search for API key shapes (sk-, ghp_, AKIA, AIza), .env contents,
credentials in URLs, private keys. Treat any match as stop-the-line.

### 3. Type / lint / test status
Detect the project's check commands from package.json / Makefile /
pyproject.toml. Run typecheck, then lint, then tests. Stop on first fail.

### 4. Tracked junk
Check for .DS_Store, .env, node_modules/, build output that snuck
past .gitignore.

## Output format

## Pre-ship report (took <Xs>)
### Blockers (N)        <- empty heading if none, never omit it
### Worth a look (N)
### Passed

Three design choices in there are worth calling out, because they're the difference between an agent you trust and one you mute:

1. "Report only matches ADDED by this diff." Without this, the agent flags every pre-existing console.log in the repo and the report is instantly noise. Scope is the diff, not the codebase. One sentence, huge signal difference.

2. A fixed output format with a "Blockers" section that's never omitted. Even when there are zero blockers, the heading stays (showing "Blockers (0)"). This sounds pedantic but it's a trust mechanism ‚Äî you learn the report's shape, so you can read it in two seconds and know exactly where to look. Variable-shape output forces re-reading every time.

3. tools: Bash, Read, Grep, Glob ‚Äî and nothing else. No Write, no Edit. The agent cannot modify your files even if it wanted to, because you didn't give it the tools. Tool scoping is a guardrail you enforce at the schema level, not a promise you hope the prompt keeps.

Try building your own

The pattern generalizes. Pick any narrow job ‚Äî writing a PR description from the actual diff, finding which behaviors your change might break, auditing dependencies ‚Äî and write the agent as:

One job, stated in a sentence.
A description that the dispatcher will actually route to ‚Äî write it in trigger phrases ("when the user says X"), not abstract capability claims.
Tool scoping ‚Äî give it only the tools the job needs. Withhold Write/Edit from anything that should only report.
An output format ‚Äî so results are pipeable and skimmable.
A refusal list ‚Äî the doors you're closing. This is the part everyone skips and it's the part that matters.

Drop the file in ~/.claude/agents/ and Claude Code picks it up automatically. No framework, no config.

The free agent + where to go deeper

The shipping-coach agent above is free and MIT-licensed ‚Äî the whole thing is one ~50-line .md file you can read, fork, and modify:

üëâ github.com/allcanprophesy-ops/claude-code-shipping-coach

Install is one command:

cp shipping-coach.md ~/.claude/agents/

Then in any repo with uncommitted changes, just say "run the pre-flight check on my diff."

If the pattern clicks and you want more agents built the same way ‚Äî pr-surgeon (writes PR descriptions from the actual diff), regression-sentinel (reads a diff asking only "what could this break?"), test-gap-hunter, and a few others ‚Äî they're linked from the repo's README. But the free one is genuinely standalone; start there.

What's the one task in your workflow you wish was automated but isn't? I'm collecting edge cases where a narrow agent would help ‚Äî drop them in the comments.