Comparing CodiumAI PR-Agent to Copilot for Pull Request

The Power of Collaboration in Code and the Role of Pull Requests in Collaboration

Collaboration lies at the heart of successful software development. The power of collaboration is particularly evident in the collective creation of code. In the dynamic realm of programming – where diverse skill sets converge – the synergy of collaborative efforts enhances the quality, efficiency, reliability, and ingenuity of code production. At the forefront of this collaborative landscape are pull requests (PRs). In this era of interconnected development, understanding the significance of collaboration and appreciating the role of pull requests is essential for building robust and scalable software solutions.

At best, PRs are pivotal mechanisms that facilitate the seamless integration of individual contributions, not only serving as gateways for incorporating changes into a codebase, but also playing a crucial role in fostering communication, code review, and continuous improvement. At worst – and, unfortunately, often in practice – PRs are a tedious purgatory of nits, miscommunication, misalignment, and requests for missing tests.

Comic via MonkeyUser

Using AI to Make Pull Requests Better

What if we could find a way to alleviate some of the painful dysfunctions that arise during the process of opening and reviewing pull requests? Fortunately, we are engineers, and solving problems is what we do best!

Artificial Intelligence (AI) has the potential to offer us new avenues of efficiency in the realm of collaborative coding. With machine learning algorithms and natural language processing capabilities, AI tools can autonomously analyze code changes, identify potential nits or coding style issues, and even suggest improvements, thereby expediting the review process. Moreover, AI-driven systems can assist in mitigating miscommunications by providing contextual understanding, bridging the gap between different coding styles, and ensuring a harmonized codebase. By automating routine checks and flagging missing tests, AI not only streamlines the pull request journey but also allows developers to focus on the more intricate aspects of the code, fostering a collaborative environment that thrives on innovation and problem-solving. In other words, AI can help engineers spend more time doing what they love.

In response to the promise of AI – particularly when it comes to pull requests – GitHub Copilot released Copilot for Pull Requests and CodiumAI released PR-Agent. Both GitHub Copilot and CodiumAI are artificial intelligence-powered tools for developers intended to make code generation more efficient. While these tools offer similar features, they are fundamentally different and in this post, we will consider those differences and take a quick tour of each tool.

Proprietary vs. Open-source

GitHub Copilot is a cloud-based artificial intelligence tool developed by GitHub and OpenAI to assist users of popular integrated development environments (IDEs) by autocompleting code. It was launched in October 2021 and – according to GitHub – “85% of developers felt more confident in their code quality when authoring code with GitHub Copilot and GitHub Copilot Chat.” This is probably good news since GitHub Copilot users have to pay to use it. Copilot is only available via subscription at $10/month for individuals.

In March 2023, GitHub launched Copilot for Pull Requests to limited release. (There is currently a waitlist.) Owned by Microsoft, GitHub Copilot does not support any other Git-hosting platform.

CodiumAI uses generative AI to help developers build code logic tests automatically. CodiumAI focuses on code integrity by generating tests that help decipher how code behaves, discover edge cases and suspicious behaviors, and ultimately produce more robust code. Like GitHub Copilot, CodiumAI is available as an extension for Visual Code Studio and JetBrains. Unlike GitHub Copilot, CodiumAI is free for individuals.

CodiumAI also offers an AI solution for pull requests called PR-Agent. PR-Agent is open-source, supports other popular Git-hosting platforms besides GitHub, and is available without a waitlist.

Like a lot of engineers I know, I’ve had pull requests languish in review across multiple sprints. It’s often a demoralizing rite of passage for a developer, so I was curious to see how Copilot for Pull Requests and PR-Agent could help expedite reviews and get sprints – and spirits – back on track.

Creating an Example Pull Request

For this example, I am using a pull request I created in my testing-strategies-for-python repo. I want to add a database to my Build-a-Pug application and I have created a (branch)[https://github.com/liz-acosta/testing-strategies-for-python/tree/add-database-ai-pr-example] and pull request to do so. You can view the pull request here.

GitHub Copilot for Pull Requests

GitHub Copilot for Pull Requests operates within the GitHub UI for accounts subscribed to Copilot. Copilot for Pull Requests is triggered by using specified Markers within the comments of a PR.

The current available Markers are:

• copilot:all all the content, in one go!
• copilot:summary a one-paragraph summary of the changes in the pull request.
• copilot:walkthrough a detailed list of changes, including links to the relevant pieces of code.
• copilot:poem a poem about the changes in the pull request.

Unfortunately, at the time of this writing, I am still on the waitlist for Copilot for Pull Requests, so I can’t test it out with my own repo, but here’s a demonstration provided by GitHub.

Demonstration of Copilot for Pull Requests via GitHub

CodiumAI PR-Agent

CodiumAI’s PR-Agent supports functionalities across the most popular Git-hosting platforms in addition to GitHub. PR-Agent offers flexibility in its implementation: It can be installed and executed locally via a CLI, run using a Docker image, or integrated with a specific Git platform workflow. Or you can get started using it right away with the PR-Agent Git plugin – no waitlist necessary.

PR-Agent supports the following commands:

• /describe automatically generates a PR description with a title, type, summary, code walkthrough and labels.
• /review provides adjustable feedback about the PR’s main theme, type, relevant tests, security issues, score, and various suggestions for the PR content.
• /ask answers free-text questions about the PR.
• /improve provides committable code suggestions for improving the PR.
• /update_changelog automatically updates the CHANGELOG.md file with the PR changes.
• /similar_issue automatically retrieves and presents similar issues.
• /add_docs automatically adds documentation to un-documented functions and classes in the PR.
• /generate_labels automatically suggests custom labels based on the PR code changes.

To use the PR-Agent Git plugin, all you need to do is comment on a PR with one of the commands and preface it with @CodiumAI-Agent.

For example, commenting with @CodiumAI-Agent /review on my example PR first elicits an eyes emoji response from CodiumAI-Agent (which indicates acknowledgment of the command) and then a new comment with the following:

PR Analysis

• 🎯 Main theme: Adding database functionality to the application
• 📝 PR summary: This PR introduces a database to the application by using SQLAlchemy. It also defines a new model, Pug, and configures the application to use a SQLite database. Additionally, it removes a GitHub action and adds a new dependency to the Pipfile.
• 📌 Type of PR: Enhancement
• 🧪 Relevant tests added: No
• ⏱️ Estimated effort to review [1-5]: 3, because the PR introduces a significant new feature (a database), but the changes are relatively straightforward and well-structured. However, the lack of tests for the new functionality increases the review effort.

• 🔒 Security concerns: Yes, because the SECRET_KEY is hardcoded in the code. This could potentially expose sensitive information. It's recommended to use environment variables for such data.

  PR Feedback

• 💡 General suggestions: It's great to see that you've added a database to the application. However, it would be beneficial to include tests for the new functionality. This would help ensure that the database and the new Pug model are working as expected. Also, consider adding error handling for potential database errors.

See the rest of the review here.

And because I believe that the more documentation the better, I decided to use @CodiumAI-Agent /add_docs and received a new comment with the following:

Suggestion: Proposed documentation

"""
This is a class representing a Pug in the database.
It has the following attributes:
- id: The unique identifier for the Pug.
- name: The name of the Pug.
- age: The age of the Pug.
- home: The home of the Pug.
- puppy_dinner: The time for the Pug's dinner.
- created_at: The time when the Pug's record was created.
- image: The image of the Pug.
"""
class Pug(db.Model):

See the rest of the proposed documentation here.

But most impressive was the /ask command: @CodiumAI-Agent /ask What would be the best way to refactor the Pug class into the Pug database model?

Question: What would be the best way to refactor the Pug class into the Pug database model?

Answer:
The Pug class has been refactored into a database model quite well in this PR. Here are some suggestions to further improve it:

1. Separation of Concerns: The Pug class is currently defined in the app.py file. It would be better to move this class to a separate file, say models.py. This way, the application setup and the database models are separated, making the code more organized and easier to maintain.

2. Adding Relationships: If there are other tables in the database that have a relationship with the Pug table, these relationships should be defined in the Pug model. For example, if there is a Owner table, and each Pug has an owner, then a relationship should be defined in the Pug model like so: owner = db.relationship('Owner', backref='pugs').

3. Adding Class Methods: It could be useful to add class methods for common database operations. For example, a method to add a new Pug to the database, or to find a Pug by its name. This would encapsulate these operations within the Pug class, and make the rest of the code cleaner and more readable.

4. Adding Validation: You could add validation to the Pug model to ensure that only valid data is saved to the database. For example, you could add a check to ensure that the age is a positive integer, or that the name is not empty.

See the rest of the answer here.

While this may not be the best answer or the answer I was looking for, it’s a good answer to start working from, which is what pair programming is all about and would help me improve this pull request.

Final Thoughts on Copilot for Pull Requests vs. PR-Agent

At the moment, CodiumAI’s PR-Agent is not only more accessible and flexible, it also offers more commands and all for free for individual developers. In contrast, GitHub Copilot for Pull Requests is currently waitlist only, supports no other Git-hosting platform but GitHub, does not offer as many commands, and costs $10/month even for individual developers.

Until I get access to GitHub Copilot for Pull Request, a complete and accurate assessment may not be possible, but I can say this: as an open-source project readily accessible, CodiumAI’s PR-Agent is much more aligned with the spirit of collaborative innovation that pull requests represent.