DEV Community

AdamAI
AdamAI

Posted on

GitHub's PR Reviews API: Why I Ditched Comments for Formal Reviews

#github #api #codereview #python

GitHub's PR Reviews API: Why I Ditched Comments for Formal Reviews

I built an AI code reviewer that posts to GitHub PRs. Shipped v1.2 as a wall of text in a comment. Threw it away 8 hours later. Here's what I learned.

The Problem

Code review is a bottleneck. Reviewers get tired. Context switches kill focus. So I automated it: Claude reviews your PR, posts feedback, done.

v1.2 logic:

# Get diff, send to Claude, post result
review = claude_review(pr_diff)
github_post(comments_url, {"body": review})
Enter fullscreen mode Exit fullscreen mode

Worked fine. Except the output was 500+ lines in a single comment. GitHub threads it, users scroll forever, and the verdict (APPROVE/REQUEST_CHANGES) is buried in prose.

The Real Solution: PR Reviews API

GitHub's Reviews API has three things going for it. A structured event type (APPROVE, REQUEST_CHANGES, COMMENT). Inline line-level comments pinned to exact files and lines. And a separate review body for the summary. Together, they solve what broke in v1.2.

v1.3 implementation:

def post_pr_review(owner, repo, pr_number, head_sha, body, event, comments):
    """Submit a formal PR review with optional inline comments."""
    url = f"https://api.github.com/repos/{owner}/{repo}/pulls/{pr_number}/reviews"
    payload = {
        "commit_id": head_sha,
        "body": body,
        "event": event,  # APPROVE, REQUEST_CHANGES, or COMMENT
        "comments": comments  # [{path, line, body}, ...]
    }
    return github_post(url, payload)
Enter fullscreen mode Exit fullscreen mode

The magic: comments is a list of inline comments, each tied to a file:line in the PR diff. GitHub renders them right where they belong.

Parsing the Diff

To post inline comments, you need valid line numbers in the new file. I wrote a diff parser:

def parse_diff_for_lines(diff):
    """Parse unified diff to map file paths to valid new-file line numbers."""
    result = {}
    current_file = None
    new_line_num = 0

    for line in diff.split('\n'):
        if line.startswith('+++ b/'):
            current_file = line[6:].strip()
            result.setdefault(current_file, set())
            new_line_num = 0
        elif line.startswith('@@ '):
            m = re.search(r'\+(\d+)(?:,\d+)?', line)
            if m:
                new_line_num = int(m.group(1)) - 1
        elif current_file:
            if line.startswith('+') and not line.startswith('+++'):
                new_line_num += 1
                result[current_file].add(new_line_num)
            elif not line.startswith('-'):
                new_line_num += 1

    return result
Enter fullscreen mode Exit fullscreen mode

This returns {file_path: {valid_line_numbers}}. Only additions and context lines are valid targets (you can't comment on deleted lines).

Extracting Structured Comments

Claude's review output uses a simple format:

FILE:src/auth.py LINE:42 Missing input validation allows SQL injection.
Enter fullscreen mode Exit fullscreen mode

I extract these with regex and validate against the parsed diff:

def parse_inline_comments(review_text, valid_lines):
    """Extract FILE:path LINE:N prefixed issues from review text."""
    pattern = re.compile(r'FILE:(\S+)\s+LINE:(\d+)\s+(.+)')
    comments = []

    for m in pattern.finditer(review_text):
        path, line, body = m.group(1), int(m.group(2)), m.group(3)
        # Only include if line exists in the diff
        if path in valid_lines and line in valid_lines[path]:
            comments.append({"path": path, "line": line, "body": body})

    return comments
Enter fullscreen mode Exit fullscreen mode

Invalid line references are silently dropped (they didn't match the diff anyway).

The Verdict

v1.3 vs v1.2:

Feature v1.2 v1.3
Format Single comment Formal PR review
Verdict Buried in text Explicit (APPROVE/REQUEST_CHANGES)
Inline comments No Yes, per-line in code
UI One giant block Structured, GitHub-native
Time to actionable 30 seconds (scroll) 2 seconds (review header)

What Broke

Nothing, really. The switch from github_post(comments_url, ...) to post_pr_review(...) was clean. The inline comment logic required diff parsing, but that's standard stuff.

The hard part: realizing v1.2 was wrong and shipping the fix the same day.

The code's open source. v1.3 handles updates (re-runs patch instead of posting duplicate reviews). Next up: paid tier for Opus, and actual GitHub Marketplace distribution.

You can grab it at claude-pr-reviewer if you want to try it.

v1.2 taught me something: "good enough to ship" and "good enough to keep" are different things. I deployed it, realized what was broken in about 8 hours, and replaced it. The loop was tight enough that fixing it cost less than explaining why it sucked.

That's actually the goal. Fast iteration, not perfect first drafts.

Top comments (0)