DEV Community: Kiril Videlov

You are waiting for code review more than you should

Kiril Videlov — Fri, 05 Aug 2022 06:37:00 +0000

If you are reading this, chances are, you are in some way involved with the creation of software. You are probably well aware that code reviews are considered a best practice.

What you may not know is that, 64.3% of Pull Requests are merged without any code changes as a result of the review. This means that only in 1/3 of reviews yielded a change to the proposed code.

🗃 Dataset

We have been collecting open source data for a while, and in this post we will share insights from analyzing 2,340,078 PRs spread across 7,836 organizations. The dataset is specifically derived from repositories that exhibit teamwork. Thankfully, with so many software companies embracing open source, there are plenty of examples of different styles of collaboration.

🔮 Code review outcomes

Generally speaking, the fate of a PR being reviewed will follow one of these two paths:

Merged as is (aka LGTM'd) — 1,504,914 PRs in the dataset
Merged with changes as a result of feedback (comments, suggestions etc.) — 835,164 PRs in the dataset

⏳ Time to merge PRs

Let's look at how much people wait for their PRs to be merged.

On average, the time between a PR being opened and it being merged is 128 hours (~5 days). If we look only at the PRs that were LGTM'd this number is 74 hours (~3 days) and for the other 35.7% of PRs the average lead time is 225 hours (~9 days).

Here are two histograms with the probability distributions of merge times for LGTM'd and non-LGTM'd PRs.

Of course, the PRs that are LGTM-merged are either simpler, smaller or simply low risk, so the lead time is significantly lower. However, the cumulative wait time for changes that were already good to go is significant.

In this dataset, the total time waiting for Pull Requests to be merged (without any changes as a result of the review) is 12,718 years. For the PRs that got corrections, the time waiting adds up to 21,437 years.

The astronomical absolute values aside, we can see that a whooping 37% of the time waited was more or less unnecessary.

Looking at the trend over the past three years, it is evident lead times are going down.

A very similar trend is also observed by GitHub themselves. This is because more and more teams are choosing to ship small and often, adopting techniques like Trunk based development and Continuous delivery.

🎭 What is code review anyways

Knowledge sharing aside, code review is meant to act as a quality gate. A good review should encompass multiple aspects of the code (e.g. fit for purpose, free of errors etc.), but, let's be honest, in practice, reviews are transactional.

As an author:

You need your code reviewed so that you can ship it and continue iterating.

As a reviewer:

You want to unblock your teammate and get back to your own code.

Of course, in both cases, we don't want to compromise on quality, but not all code changes hold the same amount of impact and risk. I'd go as far as claiming that code reviews at most teams are largely pattern matching — e.g.: "Have I seen this pattern succeed before?", "Is the author familiar with this subsystem?" etc. When a sufficient number of mental tick boxes are checked, we are ready to declare LGTM.

🤖 Automation

Programmers love to automate their work — think of all the formatters, linters and static analysis tools in existence. Just for fun, here's a comparison between car automation and code automation.

Codeball is a new and ballsy attempt at taking automation further. It simulates the developer intuition and approves safe pull requests, saving teams time.

The best way of illustrating the impact of such advanced automation is by looking at how it affected the Pull Request merge times for a team using it.

While there always are tricky Pull Requests that require human review, Codeball identifies and approves a large proportion of PRs that would have been LGTM'd by a human. Naturally, this team spends less time dealing with such PRs.

If this sounds too good to be true, this is because a technological innovation is upon us! Go and test it out on your own repository!

Thanks for reading!

Kiril

Can AI do Code Review?

Kiril Videlov — Wed, 01 Jun 2022 13:38:00 +0000

So, I've been wondering this —

How predictable is programming?

With tools like GitHub Copilot being pretty decent at writing code, surely reviewing should be possible too. And if you think about it, reviewing pull requests mainly pattern matching.

TLDR: Yep, it can, with crazy accuracy.

Okay, let's add a little bit of context first. On average, most code contributions (about 65%) get merged without any fixes/changes as a result of the review.

Most teams aim to ship code in small incremental pull requests, but no matter how seemingly safe a change is, you have to wait for review.

But why?

Well, if an AI can accurately predict which contributions would be approved by humans, it can save developers a ton of time.

Saving review time aside, the part I'm really excited about is this:

If we can stop rubber stamping every single PR, we can start paying more attention to the tricky ones.

How + demo

During a hackaton at work we built Codeball — a code review AI which approves safe Pull Requests that a human would have approved.

We trained it on over 1M Pull Requests, so it has seen tons of failure (shitty code) and success (decent code) patterns, making it really good at telling the difference.

You can play with it at codeball.ai where you can do "historical" tests on GitHub repos to see what the AI would have done.

Results and what's next

People have been testing Codeball on all sorts of pull requests resulting in around 99% accuracy. To quote a comment from a Hacker News thread:

Codeball is like a strict bartender who only serves you when they are absolutely sure you're old enough. You may still be overage but Codeball's not serving you.

You can add Codeball to your repository as a GitHub Action (takes like 2 minutes).

Thanks for reading!
~ Kiril

Simple & efficient code collaboration

Kiril Videlov — Wed, 09 Mar 2022 17:45:50 +0000

There is this big debate out in the dev community about collaborating on code —

Code reviews vs. Pair programming

Surely though, there must be more alternatives. For the longest time, I have been wondering how a code collaboration tool, specifically designed for teams at work could look like.

This is what prompted me to build an alternative. I would love to hear what you think!

Sturdy is a low overhead code collaboration platform for fast moving teams. It is, of course, open-source. Here are the key ideas —

👨‍💻 High-level operations around developer intent
🙌 Enable early & fast feedback on code
🔌 Compatible with Git and GitHub

It is specifically built for collaborating on code at work, and aims to remove most of the overhead associated with contributing code in a team setting.

🙋 Why build a new platform?

I think it's fair to say that the way we code at work is not the same as open-source. The biggest difference is in how closely developers collaborate with one another — at work, we do standups, plan together, and share knowledge.

On top of that...

Waiting for code reviews sucks!

For developers adopting DevOps and Continuous Delivery, shipping small, incremental changes is crucial. At the same time, the default pull request workflow has a fixed overhead per contribution — pulling code, creating a branch, staging, committing, pushing etc., not to mention getting a review! This is very pronounced when trying to integrate your code multiple times per day.

🧑‍💻 How is Sturdy different from code reviews?

With Sturdy, you work in the open within your team. This means you can discover and interact with draft code in your team as it is written. Those team drafts are a bit like live pull requests (think Figma or Google Docs, but using your local editor).

You can easily get a copy of someone else's draft code to your computer, explore it in your IDE and edit the code to automatically create suggestions.

But why?

This enables early, proactive feedback. It means you can help each other in a meaningful way, communicate better, and when everything goes well, your code is pre-reviewed.
Contributing code should not feel like submitting homework!

💁 Sounds a lot like pair programming. How is Sturdy different?

Pair programming is great for knowledge sharing and moving fast, but the tradeoff is flexibility — you have to be there in the moment.

Some developers describe Sturdy as —

asynchronous pair programming

This is because, similar to pairing, everybody in the team can see, try, and give suggestions to draft code. The key difference is that feedback exchange can happen whenever it suits individuals, and doesn't require you to be in a call.

🤷 Is this compatible with my existing setup?

Yes! Sturdy uses Git as a database but not as a protocol. This means preserving compatibility of the output, while gaining flexibility to design new ways of interacting with code.

It also has a GitHub bridge integration, which allows you to use it side by side with your existing setup, CI/CD and other integrations.

💻 Can I try it by myself, without involving my team?

Yes, absolutely! Your own workflow will be faster for it. It's not necessary for an entire team to adopt Sturdy to benefit from a more leveraged VCS experience.

With the GitHub bridge set up, by default, Sturdy follows changes on the default (main) branch and allows you to quickly create pull requests.

🐣 Can I try it? How about getting involved?

For sure! You can try the cloud hosted Sturdy or run your own instance with this Docker one-liner:

docker run --interactive \
    --pull always \
    --publish 30080:80 \
    --volume "$HOME/.sturdydata:/var/data" \
    getsturdy/server

If you choose to host yourself, be sure to check out the docs.

Sturdy is open-source and we love receiving feedback, bug reports, issues, and even code. The project uses some cool tech - Go, Vue 3, GraphQL, be sure to check out the code :). While you are there, consider giving it a star ⭐️, it would make my day 🙏!

Thanks! I will be happy to answer any questions in the comments.

— Kiril