Automate GitHub PR Reviews with LangChain Agents

LLMs have unlocked countless opportunities to tackle once unsolvable problems, thanks to their exceptional reasoning and decision-making capabilities. Among their many strengths, one of the most significant is their general code understanding, which can be leveraged to build tools that write, re-write, and review code.
Building on this capability, in this article, you will create an AI Agent that reviews GitHub pull requests, posts them as a comment in GitHub, and sends a summary of it to a configured Slack channel. For this, you will use LangChain Agents and Composio tools.

Key Objectives

• Understand what Composio is.
• Learn about LangChain and building agents with it.
• Understand the workflow of the PR agent.
• Learn how to build the agent with LangChain and Composio.

What is Composio?

Composio is an open-source platform that provides tools and integrations for building AI agents. Many applications like Slack, GitHub, Linear, etc., require complex user authentication and authorization mechanisms, and adding these integrations to your agentic workflow can be quite challenging. Composio addresses this by offering built-in user authentication and authorization management to streamline your AI application development workflow. It also lets you add applications with only a few lines of code. Composio offers an easy way to integrate your application with AI agents.

Composio supports authentication mechanisms such as OAuth, JWT, ApiKey, and basic authentication. It handles the authentication and authorization of your users, enabling the agents to integrate tools to perform actions on behalf of your users.

For this walkthrough, you need to understand two key Composio concepts:

• Actions: In Composio, actions are tasks performed on behalf of the users. For instance, if you have configured a GitHub integration, you can perform actions like starting a repository, updating the README file, etc. Composio wraps all the GitHub API features and optimizes them for LLM tool calls.
• Triggers: Triggers are predefined conditions that, when met, initiate actions from your agents. Composio offers a built-in webhook to capture trigger events. The webhooks receive a payload from integrations in real time, letting you perform actions on event data. For example, if a slack_receive_message trigger is configured for your Slack integration, the Slack app will send the event data like text, time, and channel ID to the webhook at the backend.

What is LangChain?

LangChain is an open-source framework for AI-powered applications. It offers LLM chains, vector stores, graph stores, databases, document loaders, parsers, and many more components to build a complete backend for AI applications. Because of its versatility and popularity, it has become the default choice for building AI-powered systems.

In this article, you will use LangChain Agents and other components with Composio tools to build the PR agent.

Workflow Overview

The workflow involves a Slack bot, your GitHub app integration, a webhook at the backend, and a LangChain Agent.

With Composio, you can integrate a Slack bot and connect to GitHub. The Slack integration allows you to send and receive messages within Slack channels, while the GitHub integration enables you to fetch pull request diffs.

When someone makes a pull request to the configured repository, the trigger activates and sends the event data to the backend webhook. The event payload is then forwarded to the LangChain agent. Following the provided instructions, the agent reviews the code, summarizes it, posts the review as a comment on the pull request, and also sends the summary to the configured Slack channel.

Prerequisites

To complete this tutorial, you will need a Composio account and access to the GPT-4 API. You can create a free Composio account here. Consider using Mixtral 8x7B from Groq as an alternative OpenAI GPT-4.

Get the API keys for both Composio and the LLM provider. For Composio, click on the Settings tab to view your API key.

Building the PR Agent

Now that you have grasped the basics of Composio and understood the workflow, let's build the agent. But before that, let’s set up the development environment.

Step 1: Setting up Development Environment

Create a virtual environment using Python Venv:

python -m venv myenv
source myenv/bin/activate  # On Windows use `myenv\Scripts\activate`

Install the following libraries.

pip install composio-core composio-langchain /
langchain-openai/
python-dotenv

Step 2: Setting Environment Variables

Create a *.env* file and add the following environment variables:

COMPOSIO_API_KEY=your Composio API key
OPENAI_API_KEY=your OpenAI API key
SLACK_CHANNEL_ID=slack channel ID, you want the summary posted

To authenticate your Composio account, run the following command and follow the login flow:

composio login

Step 3: Defining Tools and LLM

Now, import the libraries and define the required tools and LLM.

import os
from dotenv import load_dotenv
from composio_langchain import Action, ComposioToolSet
from langchain import hub
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain_openai import ChatOpenAI

from composio.client.collections import TriggerEventData

load_dotenv()

# Initialize the ComposioToolSet
composio_toolset = ComposioToolSet()

# Define the tools
pr_agent_tools = composio_toolset.get_actions(
    actions=[
        Action.GITHUB_GET_CODE_CHANGES_IN_PR,
        Action.GITHUB_PULLS_CREATE_REVIEW_COMMENT,
        Action.GITHUB_ISSUES_CREATE,
        Action.SLACKBOT_CHAT_POST_MESSAGE,
    ]
)

# Initialize the language model
llm = ChatOpenAI(model="gpt-4")

We are using four different actions from Composio. Each action performs a single specific task:

• Action.GITHUB_GET_CODE_CHANGES_IN_PR: Retrieves the code changes in a GitHub pull request.
• Action.GITHUB_PULLS_CREATE_REVIEW_COMMENT: Creates a review comment on a GitHub pull request.
• Action.GITHUB_ISSUES_CREATE: Creates a new issue in a GitHub repository.
• Action.SLACKBOT_CHAT_POST_MESSAGE: Sends a message to a Slack channel using a Slack bot.

Step 4: Defining the LangChain Agent

Next, define the OpenAI functions agent from LangChain with a system prompt to provide context about the workflow, LLM, and tools.

code_review_assistant_prompt = """
        You are an experienced code reviewer.
        Your task is to review the provided file diff and give constructive feedback.

        Follow these steps:
        1. Identify if the file contains significant logic changes.
        2. Summarize the changes in the diff in clear and concise English, within 100 words.
        3. Provide actionable suggestions if there are any issues in the code.

        Once you have decided on the changes, for any TODOs, create a GitHub issue.
        And send the summary of the PR review to """+os.environ['CHANNEL_ID']+""" channel on Slack. Slack doesn't have markdown so send a plain text message.
        Also, add the comprehensive review to the PR as a comment.
"""

prompt = hub.pull("hwchase17/openai-functions-agent")
combined_prompt = prompt+code_review_assistant_prompt
query_agent = create_openai_functions_agent(llm, pr_agent_tools, combined_prompt)
agent_executor = AgentExecutor(agent=query_agent, tools=pr_agent_tools, verbose=True)

print("Assistant is ready")

Step 5: Defining the Event Listener

Finally, define the event listener. This will be used to capture event payloads from the GitHub trigger. Composio's built-in event listener webhook can be configured to pick event data only from relevant trigger events.

# Create a trigger listener
listener = composio_toolset.create_trigger_listener()

@listener.callback(filters={"trigger_name": "github_pull_request_event"})
def review_new_pr(event: TriggerEventData) -> None:
    # Using the information from Trigger, execute the agent
    code_to_review = str(event.payload)
    query_task = f"Review the following code changes: {code_to_review}"
    # Execute the agent
    res = agent_executor.invoke({"input": query_task})
    print(res)

print("Listener started!")
print("Create a pr to get the review")
listener.listen()

In the code above, the callback function review_new_pr is invoked, when a PR is raised in the repository. The function receives the event data which is then passed to agent_executor. The agent executes the task as explained earlier.
Putting everything together.

import os
from dotenv import load_dotenv
from composio_langchain import Action, ComposioToolSet
from langchain import hub
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain_openai import ChatOpenAI

from composio.client.collections import TriggerEventData

load_dotenv()

# Initialize the ComposioToolSet
composio_toolset = ComposioToolSet()

# Define the code review assistant prompt
code_review_assistant_prompt = """
        You are an experienced code reviewer.
        Your task is to review the provided file diff and give constructive feedback.

        Follow these steps:
        1. Identify if the file contains significant logic changes.
        2. Summarize the changes in the diff in clear and concise English, within 100 words.
        3. Provide actionable suggestions if there are any issues in the code.

        Once you have decided on the changes, for any TODOs, create a GitHub issue.
        And send the summary of the PR review to """+os.environ['CHANNEL_ID']+""" channel on Slack. Slack doesn't have markdown so send a plain text message.
        Also, add the comprehensive review to the PR as a comment.
"""

# Define the tools
pr_agent_tools = composio_toolset.get_actions(
    actions=[
        Action.GITHUB_GET_CODE_CHANGES_IN_PR,
        Action.GITHUB_PULLS_CREATE_REVIEW_COMMENT,
        Action.GITHUB_ISSUES_CREATE,
        Action.SLACKBOT_CHAT_POST_MESSAGE,
    ]
)

# Initialize the language model
llm = ChatOpenAI(model="gpt-4")

prompt = hub.pull("hwchase17/openai-functions-agent")
combined_prompt = prompt+code_review_assistant_prompt
query_agent = create_openai_functions_agent(llm, pr_agent_tools, combined_prompt)
agent_executor = AgentExecutor(agent=query_agent, tools=pr_agent_tools, verbose=True)

print("Assistant is ready")

# Create a trigger listener
listener = composio_toolset.create_trigger_listener()

@listener.callback(filters={"trigger_name": "github_pull_request_event"})
def review_new_pr(event: TriggerEventData) -> None:
    # Using the information from Trigger, execute the agent
    code_to_review = str(event.payload)
    query_task = f"Review the following code changes: {code_to_review}"
    # Execute the agent
    res = agent_executor.invoke({"input": query_task})
    print(res)

print("Listener started!")
print("Create a pr to get the review")
listener.listen()

Now, once everything is set up, run the Python file. Make sure you have set up the Slack bot correctly in your channel.

pr agent - Gifyu

Image pr agent hosted in Gifyu

gifyu.com

Link to the GitHub repository: GitHub PR Agent, Also, check implementations with other frameworks like CrewAI, LlamaIndex, Autogen, and OpenAI.

Next Steps

In this tutorial, you learned how to build a GitHub PR agent using LangChain and Composio. However, you can customize the agent for your personal needs. For example, you can automate the entire review process by adding code reviews for individual code blocks in the PR diff. You can build this with Composio’s comprehensive set of actions and triggers. Check out the actions in the dashboard and play around to get a sense of how each of them works.