I gave myself 48 hours to validate an AI code review tool. Here's how I built it.

I wanted to test a simple idea: what if every pull request you opened got an instant AI code review?

Not a replacement for human reviewers — just a first pass that catches the obvious stuff before your teammates even look at it.

So I gave myself 48 hours to build it, ship it, and see if anyone actually finds it useful.

What it does

CodeReview.ai is a GitHub App. You install it on a repo, and every time a PR is opened, it:

1. Receives a webhook from GitHub
2. Pulls the diff via the GitHub API
3. Sends it to GPT-3.5-turbo with a code review prompt
4. Posts the review as a PR comment within seconds

That's it. No dashboard, no config files, no pricing tiers. Install and go.

The stack

• TypeScript end to end
• Vercel serverless functions for the webhook handler
• Octokit for GitHub API calls
• OpenAI API (GPT-3.5-turbo) for the review
• PostHog for tracking installs and usage
• Static HTML + Tailwind for the landing page

The whole thing is two API routes: one for the GitHub webhook, one for waitlist signups.

What I'm measuring

I set four metrics to decide if this is worth continuing:

1. More than 15 installs in 48 hours
2. At least 5 repos that receive 2+ reviews
3. Waitlist signup rate above 10% of installs
4. Sentiment not negative across feedback channels

If I hit 3 of 4, I keep building. If not, I scrap it and move on.

Honest limitations

• Diffs are truncated to 8,000 characters. Large PRs won't get full coverage.
• GPT-3.5-turbo is fast and cheap but not as sharp as GPT-4. Review depth has a ceiling.
• 50 reviews per day per installation. I'm a solo dev and my OpenAI bill needs to stay sane.

Try it

If you want to kick the tires:

• Landing page: https://codereview-ai-v2.vercel.app
• Install: https://github.com/apps/codereview-ai-alpha/installations/new

If you install it and it's annoying rather than helpful, I genuinely want to hear that. The whole point of the sprint is to find out.