So, I've been wondering this —
How predictable is programming?
With tools like GitHub Copilot being pretty decent at writing code, surely reviewing should be possible too. And if you think about it, reviewing pull requests mainly pattern matching.
TLDR: Yep, it can, with crazy accuracy.
Okay, let's add a little bit of context first. On average, most code contributions (about 65%) get merged without any fixes/changes as a result of the review.
Most teams aim to ship code in small incremental pull requests, but no matter how seemingly safe a change is, you have to wait for review.
But why?
Well, if an AI can accurately predict which contributions would be approved by humans, it can save developers a ton of time.
Saving review time aside, the part I'm really excited about is this:
If we can stop rubber stamping every single PR, we can start paying more attention to the tricky ones.
How + demo
During a hackaton at work we built Codeball — a code review AI which approves safe Pull Requests that a human would have approved.
We trained it on over 1M Pull Requests, so it has seen tons of failure (shitty code) and success (decent code) patterns, making it really good at telling the difference.
You can play with it at codeball.ai where you can do "historical" tests on GitHub repos to see what the AI would have done.
Results and what's next
People have been testing Codeball on all sorts of pull requests resulting in around 99% accuracy. To quote a comment from a Hacker News thread:
Codeball is like a strict bartender who only serves you when they are absolutely sure you're old enough. You may still be overage but Codeball's not serving you.
You can add Codeball to your repository as a GitHub Action (takes like 2 minutes).
Thanks for reading!
~ Kiril
Top comments (0)