Clone the PR branch

Here's the thing about bounty hunting for AI agents: it's becoming a full-time job. And not the fun kind.

The MRWK bounty round 12 just opened. 40 MRWK per accepted review. 30 max awards. That's 1,200 MRWK floating around for people willing to read other people's PRs and actually verify the code works.

Sound familiar? You've got a pile of open PRs, a bounty program that says "review with evidence," and a bunch of agents or contributors submitting vague garbage like "looks good to me" or "tested locally." That's not evidence. That's noise.

The Problem With "Evidence"

Here's what's breaking. The bounty rules say accepted review claims must:

• Be by someone other than the PR author
• Link a GitHub PR review or concise PR comment
• Include concrete evidence: files inspected, behavior checked, commands run, CI reviewed, or an actionable finding
• Be distinct from already paid review claims

That last one is the killer. Duplicate, vague claims get rejected. So you have to be specific. You have to show your work. And if you're doing this manually across 30 PRs, you're burning hours clicking through diffs, running builds, checking CI logs.

I've been there. You open a PR, skim the diff, run the tests, write a comment that says "verified the file parsing logic in src/parser.ts handles edge cases, ran npm test, all 47 tests pass." That's the kind of evidence that gets accepted.

But here's the reality: you're not going to manually review 30 PRs with that level of detail. Not if you want to keep your sanity or your day job.

The Manual Way (Still Works, Costs Time)

Let's say you want to review a PR properly. Here's what the boring, reliable process looks like:

# Clone the PR branch
git fetch origin pull/123/head:pr-123
git checkout pr-123

# Inspect the actual files changed
git diff main...pr-123 -- src/api/route.ts

# Run the test suite
npm test

# Check for lint issues
npm run lint

Then you write a comment that references the specific files, the commands you ran, and what you found. That's the gold standard. It gets accepted.

The problem? You can do maybe 5-10 of these properly in a day. The bounty wants 30. And the next round will want 30 more.

The AI Agent Problem

Now, if you're building an agent to do this automatically — and let's be real, that's why you're reading this — you've hit the wall. Your agent can clone repos, run tests, even write comments. But when it fails, you have no idea why.

Here's the scenario that'll make you pull your hair out:

1. Your agent clones a PR branch
2. It runs the tests
3. It writes a review comment with "evidence"
4. The bounty reviewer rejects it as vague

Why? Because your agent didn't actually verify the right things. Maybe it ran the wrong test suite. Maybe it didn't check the CI status. Maybe it hallucinated a file path that doesn't exist.

And you're left with logs that say "agent finished". No trace of what it actually did. No way to know which step produced the bad evidence.

The Fix: Trace Your Agent's Steps

This is where time-travel debugging saves your ass. Instead of guessing what your agent did, you capture every step.

import { TracePilot } from 'tracepilot-sdk';
import OpenAI from 'openai';

const tp = new TracePilot('tp_live_YOUR_KEY');
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function reviewPR(prNumber: number) {
  await tp.startTrace('pr-review-agent');

  const messages = [
    { role: 'user', content: `Review PR #${prNumber}. Clone it, run tests, verify behavior.` }
  ];

  // Step 1: Agent plans the review
  const { result: plan, spanId: planSpan } = await tp.wrapOpenAI(
    () => openai.chat.completions.create({ 
      model: 'gpt-4o', 
      messages,
      tools: [{ type: 'function', function: { name: 'cloneRepo', parameters: ... } }]
    }),
    messages
  );

  // Step 2: Agent clones and inspects
  const { result: cloneResult, spanId: cloneSpan } = await tp.wrapToolCall(
    'clone-and-inspect',
    () => cloneAndInspect(prNumber),
    planSpan,
    2
  );

  // Step 3: Agent runs tests
  const { result: testResult } = await tp.wrapToolCall(
    'run-tests',
    () => runTests(prNumber),
    cloneSpan,
    3
  );

  // Step 4: Agent writes review
  const reviewMessages = [...messages, plan.choices[0].message, 
    { role: 'tool', content: JSON.stringify({ clone: cloneResult, tests: testResult }) }];

  const { result: review } = await tp.wrapOpenAI(
    () => openai.chat.completions.create({ model: 'gpt-4o', messages: reviewMessages }),
    reviewMessages,
    cloneSpan,
    4
  );

  return review.choices[0].message.content;
}

Now when your agent writes a vague review, you open the dashboard. You see exactly which step produced the hallucinated file path. You see the exact prompt that led to the bad output. You fork that step, fix the prompt, and replay.

No redeployment. No "I'll add more logging and hope." You see the exact state of the agent when it failed.

The One-Line Change

If you're using the Vercel AI SDK, it's even simpler:

import { tracepilot } from "@/lib/tracepilot";
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: openai("gpt-4o"),
    messages,
    ...tracepilot.instrument(), // ← captures everything
  });

  return result.toDataStreamResponse();
}

That's it. Every tool call, every token, every latency spike. Captured.

The Bottom Line

Bounty hunting with agents is a real use case. But your agent is only as good as its ability to produce verifiable evidence. And when it fails — and it will fail — you need to know exactly why.

Manual review still works. But if you're scaling to 30 PRs per round, you need visibility into your automation. Otherwise you're just generating rejected claims at scale.

The next time your agent produces a vague review, don't add more logging. Trace the execution. Fork the failing step. Fix it. Move on.

That's the difference between an agent that writes "LGTM" and one that actually earns the bounty.

---

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord