DEV Community: AG2

Get Communicating with Discord, Slack, and Telegram

AG2 Blogger — Fri, 28 Feb 2025 16:24:47 +0000

Welcome DiscordAgent, SlackAgent, and TelegramAgent

We want to help you focus on building workflows and enhancing agents, so we’re building reference agents to get you going quicker.

Say hello to three new AG2 communication agents - DiscordAgent, SlackAgent, and TelegramAgent, here so that you can use an agentic application to send and retrieve messages from messaging platforms.

New agents need new tools

Underpinning these new agents are AG2 tools for sending and retrieving messages.

Why is it important that these are tools? Simple, you can add the tools to your existing agents or build a mega communications agent with all the tools.

Additionally, as the agents are tool-based, you can add your tools to these agents to make them even more capable.

Here are our three new agents and the pair of new tools that they use (and you can use with your own agents):

Agent	Send Tool	Retrieve Tool
`DiscordAgent`	`DiscordSendTool`	`DiscordRetrieveTool`
`SlackAgent`	`SlackSendTool`	`SlackRetrieveTool`
`TelegramAgent`	`TelegramSendTool`	`TelegramRetrieveTool`

Capabilities

The send tools all have the same functionality, they take a message and post it to the configured channel/group/bot. If a message is longer than the platforms permitted message length, the tool will split the message into multiple messages.

All the retrieve tools can:

Retrieve the latest X messages from a channel/group/bot.
Retrieve messages since a given date.
Retrieve messages since a given message ID.
Retrieve a message given its ID.

The Telegram retrieve tool also has the ability to retrieve messages using a search string (and this can be done in combination with the retrieval options above).

Configuration

In this first iteration of the tools, each tool is configured for a specific channel/group/bot. This configuration is applied when you create the tool.

Discord, Slack, and Telegram all have their own authentication and channel/group/bot identifiers. See this Notebook for more guidance on establishing those authentication details and identifiers.

Here are examples of how to create the tools, passing in the configuration:

discord_send_tool = DiscordSendTool(
    bot_token=my_bot_token,
    guild_name=my_guild_name,
    channel_name=my_channel_name,
    )

slack_send_tool = SlackSendTool(
    bot_token=my_bot_token,
    channel_id=my_channel_id,
    )

telegram_send_tool = TelegramRetrieveTool(
    api_id=my_api_id,
    api_hash=my_api_hash,
    chat_id=my_group_chat_id,
    )

# And the same for DiscordRetrieveTool, SlackRetrieveTool, and TelegramRetrieveTool

Or, the agents:

discord_agent = DiscordAgent(
    name="discord_agent",
    llm_config=llm_config,
    bot_token=my_bot_token,
    guild_name=my_guild_name,
    channel_name=my_channel_name,
    )

slack_agent = SlackAgent(
    name="slack_agent",
    llm_config=llm_config,
    bot_token=my_bot_token,
    channel_id=my_channel_id,
    )

telegram_agent = TelegramAgent(
    name="telegram_agent",
    llm_config=llm_config,
    api_id=my_api_id,
    api_hash=my_api_hash,
    chat_id=my_group_chat_id,
    )

If you need to send to or retrieve messages from multiple channels, create additional tool instances for each channel and load up your agents or have an agent for each channel.

Agent vs Tools

You could create a ConversableAgent and add the send and retrieve tools and you would almost have the same functionality as the new agents.

Apart from the simplicity of creating an agent with the tools already registered, an important component of the new agents is that the platform’s messaging requirements will be appended to their system message.

They cover aspects like maximum message length, preferred format (e.g. Markdown), and provide tips like using emojis.

Agent	System Message guidance
`DiscordAgent`	2,000 character limit, Markdown, bold/italic/code, use emojis
`SlackAgent`	40,000 character limit, Markdown, bold/italic/code, emojis, notification formats
`TelegramAgent`	4,096 character limit, HTML, mentions and emojis

These are enabled by default, but you can turn these off by setting has_writing_instructions to False when creating the agent.

Another important consideration is the execution of these tools. In AG2 the tool execution is typically handled by a separate agent that will follow the agent in the conversation (unless its an agent in a swarm whereby tools are executed automatically). So, you will need to register the tools with another agent for execution. See the demo below for how to do this.

Installation

Each messaging platform requires installation of a package specifically for that platform.

To use the Discord agent or tools install AG2 with the commsagent-discord extra:

pip install ag2[openai,commsagent-discord]

Slack:

pip install ag2[openai,commsagent-slack]

Telegram:

pip install ag2[openai,commsagent-telegram]

You’re all set! Now you can start sending to and retrieving messages from these platforms.

Demo

Here’s a quick example of how to use the new agents and tools. If you’re after more examples, see this Notebook.

In this demo we have a Slack channel where AG2 bugs are noted. Those bugs need to go to a Discord channel for them to be raised as a ticket and also to Telegram to inform the community about it. Finally, we should send a message back to Slack to say that we’ve actioned the steps.

We’ll start with creating an equivalent DiscordAgent using the tools so you now how to construct your own communication agent. For Slack and Telegram, we’ll use SlackAgent and TelegramAgent.

# Imports
from autogen import ConversableAgent # We'll build our DiscordAgent from ConversableAgent
from autogen.agents.experimental import SlackAgent, TelegramAgent
from autogen.tools.experimental.messageplatform import DiscordSendTool, DiscordRetrieveTool
from autogen import GroupChatManager, GroupChat

# LLM Configuration
# OpenAI GPT-4o mini for all agents
# Put your OpenAI API key in the OPENAI_API_KEY environment key
llm_config = {"api_type": "openai", "model": "gpt-4o-mini"}

All your tokens and identifiers, next.

# Tokens and Ids
my_discord_bot_token = ""
my_discord_guild_name = ""
my_discord_channel_name = ""

my_telegram_api_id = ""
my_telegram_api_hash = ""
my_telegram_chat_id = ""

my_slack_bot_token = ""
my_slack_channel_id = ""

Creating an equivalent DiscordAgent using tools.

# Manually creating a DiscordAgent, start with a ConversableAgent and detailed system message
discord_agent = ConversableAgent(
    name="discord_agent",
    llm_config=llm_config,
    system_message=(
        "You are managing a Discord channel for the developers of AG2.\n"
        "When you get a new issue ask them to create a ticket for it.\n"
        "\nFormat guidelines for Discord:\n"
        "1. Max message length: 2000 characters\n"
        "2. Supports Markdown formatting\n"
        "3. Use double asterisks (**) for bold text, single asterisks (*) for italic text, and triple backticks (```



 ```) for code blocks.\n"
        "4. Consider using appropriate emojis when suitable\n"
    ),
)

# Create the tools
discord_send_tool = DiscordSendTool(bot_token=my_discord_bot_token, guild_name=my_discord_guild_name, channel_name=my_discord_channel_name)
discord_retrieve_tool = DiscordRetrieveTool(bot_token=my_discord_bot_token, guild_name=my_discord_guild_name, channel_name=my_discord_channel_name)

# Register the tools for LLM recommendation with the agent
discord_agent.register_for_llm()(discord_send_tool)
discord_agent.register_for_llm()(discord_retrieve_tool)

# Now you have an equivalent to DiscordAgent

Our Slack and Telegram agents.

# Slack and Telegram agents
slack_agent = SlackAgent(
    name="slack_agent",
    system_message=(
        "You are managing a Slack channel for announcements of bugs in AG2.\n"
        "Retrieve the latest message from the channel."
    ),
    llm_config=llm_config,
    bot_token=my_slack_bot_token,
    channel_id=my_slack_channel_id,
)

telegram_agent = TelegramAgent(
    name="telegram_agent",
    system_message=(
        "You are managing a Telegram group chat for users of AG2.\n"
        "Keep them updated on any bugs that are found.\n"
        "Ask them to help fix it if they can."
        ),
    llm_config=llm_config,
    api_id=my_telegram_api_id,
    api_hash=my_telegram_api_hash,
    chat_id=my_telegram_chat_id,
)

To execute the tools, we create a tool executor and associate the tools to allow it to execute them.

# Create an agent to execute the tools
tool_executor = ConversableAgent(
    name="tool_executor",
    system_message=(
        "You execute send and retrieve functions for Discord, Slack, and Telegram platforms.\n"
        "Respond with 'TERMINATE' when finished."
        ),
    llm_config=llm_config,
    human_input_mode="NEVER",
    )

# Register the tools with the tool executor, here are the Discord tools
discord_send_tool.register_for_execution(tool_executor)
discord_retrieve_tool.register_for_execution(tool_executor)

# For SlackAgent and TelegramAgent we get the registered tools
# and register them for execution with the tool executor
for tool in slack_agent.tools + telegram_agent.tools:
    tool.register_for_execution(tool_executor)

Here’s our GroupChat and GroupChatManager to manage the agents.

groupchat = GroupChat(
    agents=[tool_executor, discord_agent, slack_agent, telegram_agent],
    messages=[],
    )

gcm = GroupChatManager(
    name="coordinator",
    groupchat=groupchat,
    llm_config=llm_config,
    is_termination_msg=lambda x: (x["content"] is not None) and "TERMINATE" in x["content"],
    )

Let’s go!

tool_executor.initiate_chat(
    recipient=gcm,
    message=(
        "Get the latest bug report from Slack, send it to Discord for a ticket to be raised, and then notify users of AG2 on Telegram.\n"
        "Once that has been done, send a message to Slack to say that a ticket has been raised and the community notified.\n"
        "Each time a message is retrieved or prepared for send, use the tool_executor to execute that send/retrieve."
        ),
    )

Here’s the message we had in Slack that it retrieved:

AG2 workflow:

tool_executor (to coordinator):

Get the latest bug report from Slack, send it to Discord for a ticket to be raised, and then notify users of AG2 on Telegram.
Once that has been done, send a message to Slack to say that a ticket has been raised and the community notified.
Each time a message is retrieved or prepared for send, use the tool_executor to execute that send/retrieve.

--------------------------------------------------------------------------------

Next speaker: slack_agent


>>>>>>>> USING AUTO REPLY...
slack_agent (to coordinator):

***** Suggested tool call (call_ZqtAzdntJkhX5dZOTCzuFnBi): slack_retrieve *****
Arguments:
{"maximum_messages":1}
*******************************************************************************

--------------------------------------------------------------------------------

Next speaker: tool_executor


>>>>>>>> EXECUTING FUNCTION slack_retrieve...
Call ID: call_ZqtAzdntJkhX5dZOTCzuFnBi
Input arguments: {'maximum_messages': 1}
tool_executor (to coordinator):

***** Response from calling tool (call_ZqtAzdntJkhX5dZOTCzuFnBi) *****
{'message_count': 1, 'messages': [{'user': 'U0892JKRLM9', 'type': 'message', 'ts': '1738791823.519679', 'client_msg_id': '1eea96fe-1949-4585-bba4-a67609ff3787', 'text': "We have a bug with the Swarm Orchestration, it's crashing if two agents have the same name.", 'team': 'T088N5HLRTR', 'blocks': [{'type': 'rich_text', 'block_id': '8PxkJ', 'elements': [{'type': 'rich_text_section', 'elements': [{'type': 'text', 'text': "We have a bug with the Swarm Orchestration, it's crashing if two agents have the same name."}]}]}]}], 'start_time': 'latest'}
**********************************************************************

--------------------------------------------------------------------------------

Next speaker: discord_agent


>>>>>>>> USING AUTO REPLY...
discord_agent (to coordinator):

***** Suggested tool call (call_XqB52fkcV8gsHkEwdQhlzOMH): discord_send *****
Arguments:
{"message":"🚨 **New Bug Report:**\n\n*We have a bug with the Swarm Orchestration, it's crashing if two agents have the same name.*\n\nPlease create a ticket for this issue!"}
*****************************************************************************

--------------------------------------------------------------------------------

Next speaker: tool_executor


>>>>>>>> EXECUTING FUNCTION discord_send...
Call ID: call_XqB52fkcV8gsHkEwdQhlzOMH
Input arguments: {'message': "🚨 **New Bug Report:**\n\n*We have a bug with the Swarm Orchestration, it's crashing if two agents have the same name.*\n\nPlease create a ticket for this issue!"}
tool_executor (to coordinator):

***** Response from calling tool (call_XqB52fkcV8gsHkEwdQhlzOMH) *****
Message sent successfully (ID: 1336843063695380533):
🚨 **New Bug Report:**

*We have a bug with the Swarm Orchestration, it's crashing if two agents have the same name.*

Please create a ticket for this issue!
**********************************************************************

--------------------------------------------------------------------------------

Next speaker: telegram_agent


>>>>>>>> USING AUTO REPLY...
telegram_agent (to coordinator):

***** Suggested tool call (call_4NxGOLt0gK91Mu2YOSuIIWjj): telegram_send *****
Arguments:
{"message":"🚨 <b>Latest Bug Report:</b>\n\n<i>We have a bug with the Swarm Orchestration, it's crashing if two agents have the same name.</i>\n\nIf anyone has insights or can help fix this issue, please step forward! Your collaboration is greatly appreciated! 💪"}
******************************************************************************

--------------------------------------------------------------------------------

Next speaker: tool_executor


>>>>>>>> EXECUTING FUNCTION telegram_send...
Call ID: call_4NxGOLt0gK91Mu2YOSuIIWjj
Input arguments: {'message': "🚨 <b>Latest Bug Report:</b>\n\n<i>We have a bug with the Swarm Orchestration, it's crashing if two agents have the same name.</i>\n\nIf anyone has insights or can help fix this issue, please step forward! Your collaboration is greatly appreciated! 💪"}
tool_executor (to coordinator):

***** Response from calling tool (call_4NxGOLt0gK91Mu2YOSuIIWjj) *****
Message sent successfully (ID: 112):
🚨 <b>Latest Bug Report:</b>

<i>We have a bug with the Swarm Orchestration, it's crashing if two agents have the same name.</i>

If anyone has insights or can help fix this issue, please step forward! Your collaboration is greatly appreciated! 💪
**********************************************************************

--------------------------------------------------------------------------------

Next speaker: slack_agent


>>>>>>>> USING AUTO REPLY...
slack_agent (to coordinator):

***** Suggested tool call (call_xBqfhiam27uRoNwHoPzDrH75): slack_send *****
Arguments:
{"message":"📢 A ticket has been raised for the reported bug, and the community has been notified on Telegram. Thank you for your attention!"}
***************************************************************************

--------------------------------------------------------------------------------

Next speaker: tool_executor


>>>>>>>> EXECUTING FUNCTION slack_send...
Call ID: call_xBqfhiam27uRoNwHoPzDrH75
Input arguments: {'message': '📢 A ticket has been raised for the reported bug, and the community has been notified on Telegram. Thank you for your attention!'}
tool_executor (to coordinator):

***** Response from calling tool (call_xBqfhiam27uRoNwHoPzDrH75) *****
Message sent successfully (ID: 1738798634.772939):
📢 A ticket has been raised for the reported bug, and the community has been notified on Telegram. Thank you for your attention!
**********************************************************************

--------------------------------------------------------------------------------

Next speaker: tool_executor

tool_executor (to coordinator):

All tasks completed successfully.

TERMINATE

--------------------------------------------------------------------------------

Discord message posted (based on Markdown as per instructions):

Telegram message posted (based on HTML):

Finally, our return message to Slack (based on Markdown):

Feedback

These tools are currently in our experimental namespace, indicating that we have tested the functionality but their interface may change.

We’d love for you to try them but please use them with that in mind. Let us know how you go with them!

What’s next?

We’re continuing to build new agents, creating more tools, and enhancing agent capabilities.

There are many messaging platforms and we’re starting with Discord, Slack, and Telegram. If you want your AG2 agents to communicate with more messaging platforms we’d love to help you build new agents, see our Contributor Guide and chat with the team and collaborators on Discord.

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

DeepResearchAgent - Your Shortcut for Faster Research

AG2 Blogger — Fri, 14 Feb 2025 12:07:34 +0000

Authors: Robert Jambrecic, Tvrtko Sternak

Introduction

We are excited to unveil DeepResearchAgent, a powerful autonomous research capability within our AG2 framework. Inspired by OpenAI’s Deep Research, our agent is designed to tackle complex, multi-step research tasks efficiently, synthesizing insights from diverse online sources.

With DeepResearchAgent, we take a significant step toward the future of autonomous knowledge synthesis—freeing up valuable time and enabling deeper, data-driven decision-making

Installation

DeepResearchAgent is built on top of Browser Use, which requires Python 3.11 or higher.

To get started with the DeepResearchAgent agent, follow these steps:

Install AG2 with the browser-use extra:
```
pip install ag2[browser-use]
```
If you have been using autogen or pyautogen, all you need to do is upgrade it using:
```
pip install -U autogen[browser-use]
```
or
```
pip install -U pyautogen[browser-use]
```
as pyautogen, autogen, and ag2 are aliases for the same PyPI package.

Set up Playwright:

# Installs Playwright and browsers for all OS
playwright install
# Additional command, mandatory for Linux only
playwright install-deps

For running the code in Jupyter, use nest_asyncio to allow nested event loops.
```
pip install nest_asyncio
```

You’re all set! Now you can start solving complex tasks.

Imports

import os
import nest_asyncio

from autogen.agents.experimental import DeepResearchAgent

nest_asyncio.apply()

Starting the Conversation

This code initializes DeepResearchAgent and asks it to find out What was the impact of DeepSeek on stock prices and why? using GPT-4o.

We recommend using gpt-4o

Models like gpt-4o-mini often don’t perform well enough for complex tasks.

llm_config = {
    "config_list": [{"model": "gpt-4o", "api_key": os.environ["OPENAI_API_KEY"]}],
}

agent = DeepResearchAgent(
    name="DeepResearchAgent",
    llm_config=llm_config,
)

message = "What was the impact of DeepSeek on stock prices and why?"

result = agent.run(
    message=message,
    tools=agent.tools,
    max_turns=2,
    user_input=False,
    summary_method="reflection_with_llm",
)

print(result.summary)

The agent will break down the initial task into smaller, manageable subtasks and tackle them sequentially. Once all subtasks are completed, a final summary will be generated, for example:

The investigation into the impact of DeepSeek on stock prices reveals several key insights:

1. **DeepSeek's Role and Functionality:**
 - DeepSeek is a renowned Chinese AI company, known for its cutting-edge, open-source language models that compete with those of major U.S. tech giants. The models offer significant advancements in reasoning and optimization of tasks requiring logical inference and problem-solving, utilizing methods such as "Chain of Thought" and reinforcement learning.

2. **Impact on Stock Prices:**
 - The introduction and application of DeepSeek have led to significant repercussions in stock markets. The tech industry experienced a selloff, notably impacting companies like Nvidia, with stock valuations decreasing markedly—from 50 times to 36 times estimated earnings. Major indexes, including Nasdaq and S&P 500, faced declines of 3.1% and 1.5%, respectively.

3. **Factors behind the Changes in Stock Prices:**
 - The shifts in stock prices are attributed to several factors:
 - **Technological Advancements:** DeepSeek's technological breakthroughs have triggered notable market changes.
 - **Impact on Major Tech Companies:** There have been significant market-cap losses for companies like Nvidia due to concerns about competition from DeepSeek.
 - **Market Volatility:** Concerns over Chinese AI enhancements have induced market-wide volatility.
 - **Investment Opportunities:** DeepSeek's advancements have sparked interest in investing in undervalued Chinese AI stocks, indicating potential growth areas.
 - **Financial Market Reactions:** The rapid changes induced by DeepSeek's developments have caused noticeable financial market fluctuations, particularly affecting Wall Street.

Overall, DeepSeek's emergence has profoundly influenced stock prices by fostering both immediate market anxiety and potential long-term investment opportunities.
Reasoning: The answer comprehensively covers the impact of DeepSeek on stock prices and the reasons behind these changes. It is well-structured, providing a clear understanding of DeepSeek's functions, its effects on the stock market, and the underlying factors contributing to these effects. Each aspect of the investigation aligns with the subquestions previously addressed, ensuring completeness and coherence. Thus, this summary effectively communicates the investigation's findings.

Conclusion

The DeepResearchAgent within the AG2 framework helps automate complex research tasks by gathering information from different online sources. It breaks down big questions into smaller tasks and works through them one by one. After completing all the steps, the agent creates a detailed summary, making it easier for users to handle research challenges and find useful insights quickly.

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for DeepResearchAgent! Or maybe you would like to see more features or improvements? In any case, do join our Discord server for discussion.

Adding Browsing Capabilities to AG2

AG2 Blogger — Thu, 06 Feb 2025 18:36:17 +0000

Author: Robert Jambrecic

Introduction

Previously, in our Cross-Framework LLM Tool Integration guide, we combined tools from frameworks like LangChain, CrewAI, and PydanticAI to enhance AG2.

Now, we’re taking AG2 even further by integrating Browser Use and Crawl4AI, enabling agents to navigate websites, extract dynamic content, and interact with web pages. This unlocks new possibilities for automated data collection, web automation, and more.

`Browser Use` Integration

Installation

Browser Use requires Python 3.11 or higher.

To get started with the Browser Use integration in AG2, follow these steps:

Install AG2 with the browser-use extra:
```
pip install ag2[browser-use]
```
If you have been using autogen or pyautogen, all you need to do is upgrade it using:
```
pip install -U autogen[browser-use]
```
or
```
pip install -U pyautogen[browser-use]
```
as pyautogen, autogen, and ag2 are aliases for the same PyPI package.

Set up Playwright:

# Installs Playwright and browsers for all OS
playwright install
# Additional command, mandatory for Linux only
playwright install-deps

For running the code in Jupyter, use nest_asyncio to allow nested event loops.
```
pip install nest_asyncio
```

You’re all set! Now you can start using browsing features in AG2.

Imports

import os
import nest_asyncio

from autogen import AssistantAgent, UserProxyAgent
from autogen.tools.experimental import BrowserUseTool

nest_asyncio.apply()

Agent Configuration

Configure the agents for the interaction.

config_list defines the LLM configurations, including the model and API key.
UserProxyAgent simulates user inputs without requiring actual human interaction (set to NEVER).
AssistantAgent represents the AI agent, configured with the LLM settings.

Browser Use supports the following models: Supported Models

We had great experience with OpenAI, Anthropic, and Gemini. However, DeepSeek and Ollama haven’t performed as well.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
assistant = AssistantAgent(name="assistant", llm_config=llm_config)

Web Browsing with Browser Use

The BrowserUseTool llows agents to interact with web pages—navigating, searching, and extracting information.

To see the agent’s activity in real-time, set headless to False in the browser_config. If True, the browser runs in the background.

browser_use_tool = BrowserUseTool(
    llm_config=llm_config,
    browser_config={"headless": False},
)

browser_use_tool.register_for_execution(user_proxy)
browser_use_tool.register_for_llm(assistant)

Initiate Chat

Now, let’s run a task where the assistant searches Reddit for “AG2,” clicks the first post, and retrieves the first comment.

result = user_proxy.initiate_chat(
    recipient=assistant,
    message="Go to Reddit, search for 'ag2' in the search bar, click on the first post and return the first comment.",
    max_turns=2,
)

user_proxy (to assistant):

Go to Reddit, search for 'ag2' in the search bar, click on the first post and return the first comment.

--------------------------------------------------------------------------------
assistant (to user_proxy):

***** Suggested tool call (call_kHzzd6KnbDpGatDyN5Pm2hLv): browser_use *****
Arguments:
{"task":"Go to Reddit, search for 'ag2', click on the first post and return the first comment."}
****************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION browser_use...
Call ID: call_kHzzd6KnbDpGatDyN5Pm2hLv
Input arguments: {'task': "Go to Reddit, search for 'ag2', click on the first post and return the first comment."}
INFO     [agent] 🚀 Starting task: Go to Reddit, search for 'ag2', click on the first post and return the first comment.
INFO     [agent]
📍 Step 1
INFO     [agent] 🤷 Eval: Unknown - No previous actions to evaluate.
INFO     [agent] 🧠 Memory:
INFO     [agent] 🎯 Next goal: Navigate to Reddit
INFO     [agent] 🛠️  Action 1/1: {"go_to_url":{"url":"https://www.reddit.com"}}
INFO     [controller] 🔗  Navigated to https://www.reddit.com
INFO     [agent]
📍 Step 2
INFO     [agent] 🤷 Eval: Unknown - Initial navigation to Reddit completed successfully.
INFO     [agent] 🧠 Memory: Navigated to Reddit and located the search box.
INFO     [agent] 🎯 Next goal: Search for 'ag2'
INFO     [agent] 🛠️  Action 1/2: {"input_text":{"index":4,"text":"ag2"}}
INFO     [agent] 🛠️  Action 2/2: {"click_element":{"index":4}}
INFO     [controller] ⌨️  Input "ag2" into index 4
INFO     [controller] Something new appeared after action 1 / 2
INFO     [agent]
📍 Step 3
INFO     [agent] 👍 Eval: Successful in inputting search term 'ag2'.
INFO     [agent] 🧠 Memory: Input 'ag2' into the search box on Reddit.
INFO     [agent] 🎯 Next goal: Click the search button for 'ag2'.
INFO     [agent] 🛠️  Action 1/1: {"click_element":{"index":12}}
INFO     [controller] 🖱️  Clicked button with index 12:
INFO     [agent]
📍 Step 4
INFO     [agent] 👍 Eval: Successful in searching for 'ag2' and reached the results page.
INFO     [agent] 🧠 Memory: Landed on Reddit search results for 'ag2'.
INFO     [agent] 🎯 Next goal: Click on the first post.
INFO     [agent] 🛠️  Action 1/1: {"click_element":{"index":14}}
INFO     [controller] 🖱️  Clicked button with index 14: AG2 vs Autogen, which one to use?
INFO     [agent]
📍 Step 5
INFO     [agent] 👍 Eval: Successful in clicking the first post for 'ag2'.
INFO     [agent] 🧠 Memory: Accessed the chosen post 'AG2 vs Autogen, which one to use?' and the comments section is visible.
INFO     [agent] 🎯 Next goal: Extract the first comment from the comments section.
INFO     [agent] 🛠️  Action 1/1: {"extract_content":{"include_links":false}}
INFO     [controller] 📄  Extracted page as text
: Go to AI_Agentsr/AI_Agentsr/AI_AgentsA place for discussion around the use of AI Agents and related tools. AI Agents are LLMs that have the ability to "use tools" or "execute functions" in an autonomous or semi-autonomous (also known as human-in-the-loop) fashion.

Follow our event calendar: https://lu.ma/oss4ai

Join us on Discord! https://discord.gg/6tGkQcFjBY55KMembers29Online•22 days agoXodnilAG2 vs Autogen, which one to use?DiscussionI’m trying to decide betweenAG2andAutoGenfor building a multi-agent system. Both seem powerful, but I’m not sure which one fits my needs better. It's so confusing really.From what I’ve seen:AG2: Focuses on stability and backward compatibility, with features like StateFlow and Reasoner agents. But how does it handle structured outputs and multi-agent workflows?AutoGen: Known for advanced multi-agent collaboration and human-in-the-loop functionality. It integrates well with LLMs, but is it beginner-friendly?Which one would you recommend and why?ThanksAomI_V•Promoted🔥 Create AI Agents with ZERO code or hassle: click to connect your tools, click to add knowledge, click to launch!app.omnimind.aiAdd a commentSort by:BestOpen comment sort optionsBestTopNewControversialOldQ&Amacronancer•22d agoMy impression was that AG2 was a successor architecture for Autogen.I could be wrong, and I have not had the time to read into these very deeply.ReplyreplyCtiPath•21d agoYou’re correct. In November, the developers of AutoGen branched off to form AG2. Much of AG2 is still AutoGen under the hood. But they are making some improvements like the new Captain Agent.ReplyreplyMore repliesLong_Complex_4395•22d agoI am not really conversant with frameworks for AI agents, so I may not understand their capabilities. The first rule of thumb for using any framework is what do you want to achieve?So, what are you building?Do you have a workflow for what you intend to build?What are your expectations?These questions would guide you to frame your question better for people to actually help you.Replyreplyfreudweeks•21d agoAG2 seems much more active. I think autogen is now in a support mode, while MSFT's next version, autogen 0.4 is what they're actively creating features for but is unstable. Since the founders of autogen went to ag2, I think it's a safe enough bet to go to ag2. You can always pick up autogen 0.4 when it comes out later.Replyreply

INFO     [agent]
📍 Step 6
INFO     [agent] 🤷 Eval: Extracted page content containing the first comment.
INFO     [agent] 🧠 Memory: Extracted comments from the selected post on Reddit, including the first comment.
INFO     [agent] 🎯 Next goal: Complete the task.
INFO     [agent] 🛠️  Action 1/1: {"done":{"text":"First comment: 'My impression was that AG2 was a successor architecture for Autogen. I could be wrong, and I have not had the time to read into these very deeply.'"}}
INFO     [agent] 📄 Result: First comment: 'My impression was that AG2 was a successor architecture for Autogen. I could be wrong, and I have not had the time to read into these very deeply.'
INFO     [agent] ✅ Task completed successfully
user_proxy (to assistant):

***** Response from calling tool (call_kHzzd6KnbDpGatDyN5Pm2hLv) *****
{"extracted_content":["🔗  Navigated to https://www.reddit.com","⌨️  Input \"ag2\" into index 4","🖱️  Clicked button with index 12: ","🖱️  Clicked button with index 14: AG2 vs Autogen, which one to use?","📄  Extracted page as text\n: Go to AI_Agentsr/AI_Agentsr/AI_AgentsA place for discussion around the use of AI Agents and related tools. AI Agents are LLMs that have the ability to \"use tools\" or \"execute functions\" in an autonomous or semi-autonomous (also known as human-in-the-loop) fashion.\n\nFollow our event calendar: https://lu.ma/oss4ai\n\nJoin us on Discord! https://discord.gg/6tGkQcFjBY55KMembers29Online•22 days agoXodnilAG2 vs Autogen, which one to use?DiscussionI’m trying to decide betweenAG2andAutoGenfor building a multi-agent system. Both seem powerful, but I’m not sure which one fits my needs better. It's so confusing really.From what I’ve seen:AG2: Focuses on stability and backward compatibility, with features like StateFlow and Reasoner agents. But how does it handle structured outputs and multi-agent workflows?AutoGen: Known for advanced multi-agent collaboration and human-in-the-loop functionality. It integrates well with LLMs, but is it beginner-friendly?Which one would you recommend and why?ThanksAomI_V•Promoted🔥 Create AI Agents with ZERO code or hassle: click to connect your tools, click to add knowledge, click to launch!app.omnimind.aiAdd a commentSort by:BestOpen comment sort optionsBestTopNewControversialOldQ&Amacronancer•22d agoMy impression was that AG2 was a successor architecture for Autogen.I could be wrong, and I have not had the time to read into these very deeply.ReplyreplyCtiPath•21d agoYou’re correct. In November, the developers of AutoGen branched off to form AG2. Much of AG2 is still AutoGen under the hood. But they are making some improvements like the new Captain Agent.ReplyreplyMore repliesLong_Complex_4395•22d agoI am not really conversant with frameworks for AI agents, so I may not understand their capabilities. The first rule of thumb for using any framework is what do you want to achieve?So, what are you building?Do you have a workflow for what you intend to build?What are your expectations?These questions would guide you to frame your question better for people to actually help you.Replyreplyfreudweeks•21d agoAG2 seems much more active. I think autogen is now in a support mode, while MSFT's next version, autogen 0.4 is what they're actively creating features for but is unstable. Since the founders of autogen went to ag2, I think it's a safe enough bet to go to ag2. You can always pick up autogen 0.4 when it comes out later.Replyreply\n","First comment: 'My impression was that AG2 was a successor architecture for Autogen. I could be wrong, and I have not had the time to read into these very deeply.'"],"final_result":"First comment: 'My impression was that AG2 was a successor architecture for Autogen. I could be wrong, and I have not had the time to read into these very deeply.'"}
**********************************************************************

--------------------------------------------------------------------------------
assistant (to user_proxy):

The first comment on the Reddit post regarding 'ag2' is:

"My impression was that AG2 was a successor architecture for Autogen. I could be wrong, and I have not had the time to read into these very deeply."

TERMINATE

`Crawl4AI` Integration

The integration process follows a straightforward approach.

Installation

To integrate Crawl4AI with AG2, follow these steps:

Install AG2 with the crawl4ai extra:
```
pip install ag2[crawl4ai]
```
If you have been using autogen or pyautogen, all you need to do is upgrade it using:
```
pip install -U autogen[crawl4ai]
```
or
```
pip install -U pyautogen[crawl4ai]
```
as pyautogen, autogen, and ag2 are aliases for the same PyPI package.

Set up Playwright:

# Installs Playwright and browsers for all OS
playwright install
# Additional command, mandatory for Linux only
playwright install-deps

For running the code in Jupyter, use nest_asyncio to allow nested event loops.
```
pip install nest_asyncio
```

Once installed, you’re ready to start using the browsing features in AG2.

Imports

import os

import nest_asyncio
from pydantic import BaseModel

from autogen import AssistantAgent, UserProxyAgent
from autogen.tools.experimental import Crawl4AITool

nest_asyncio.apply()

Agent Configuration

Configure the agents for the interaction.

config_list defines the LLM configurations, including the model and API key.
UserProxyAgent simulates user inputs without requiring actual human interaction (set to NEVER).
AssistantAgent represents the AI agent, configured with the LLM settings.

Crawl4AI is built on top of LiteLLM and supports the same models as LiteLLM.

We had great experience with OpenAI, Anthropic, Gemini and Ollama. However, as of this writing, DeepSeek is encountering some issues.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
assistant = AssistantAgent(name="assistant", llm_config=llm_config)

Web Browsing with `Crawl4AI`

Crawl4AITool offers three integration modes:

Basic Web Scraping (No LLM):

crawlai_tool = Crawl4AITool()

Web Scraping with LLM Processing:

crawlai_tool = Crawl4AITool(llm_config=llm_config)

LLM Processing with Schema for Structured Data:

class Blog(BaseModel):
    title: str
    url: str


crawlai_tool = Crawl4AITool(llm_config=llm_config, extraction_model=Blog)

We’ll proceed with the most advanced option: LLM Processing with Structured Data Schema.

After creating the tool instance, register the agents:

crawlai_tool.register_for_execution(user_proxy)
crawlai_tool.register_for_llm(assistant)

Initiate Chat

message = "Extract all blog posts from https://docs.ag2.ai/docs/blog"
result = user_proxy.initiate_chat(
    recipient=assistant,
    message=message,
    max_turns=2,
)

user_proxy (to assistant):

Extract all blog posts from https://docs.ag2.ai/docs/blog

--------------------------------------------------------------------------------
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
assistant (to user_proxy):

***** Suggested tool call (call_R1DUz9LKPxE3t0wiy94OotAR): crawl4ai *****
Arguments:
{"url":"https://docs.ag2.ai/docs/blog","instruction":"Extract all blog posts including titles, dates, and links."}
*************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION crawl4ai...
Call ID: call_R1DUz9LKPxE3t0wiy94OotAR
Input arguments: {'url': 'https://docs.ag2.ai/docs/blog', 'instruction': 'Extract all blog posts including titles, dates, and links.'}
[INIT].... → Crawl4AI 0.4.247
[FETCH]... ↓ https://docs.ag2.ai/docs/blog... | Status: True | Time: 8.28s
[SCRAPE].. ◆ Processed https://docs.ag2.ai/docs/blog... | Time: 23ms
12:16:26 - LiteLLM:INFO: utils.py:2825 -
LiteLLM completion() model= gpt-4o-mini; provider = openai
INFO:LiteLLM:
LiteLLM completion() model= gpt-4o-mini; provider = openai
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
12:17:04 - LiteLLM:INFO: utils.py:1030 - Wrapper: Completed Call, calling success_handler
INFO:LiteLLM:Wrapper: Completed Call, calling success_handler
[EXTRACT]. ■ Completed for https://docs.ag2.ai/docs/blog... | Time: 38.029349499964155s
[COMPLETE] ● https://docs.ag2.ai/docs/blog... | Status: True | Total: 46.34s
user_proxy (to assistant):

***** Response from calling tool (call_R1DUz9LKPxE3t0wiy94OotAR) *****
[
    {
        "title": "RealtimeAgent with Gemini API",
        "url": "https://docs.ag2.ai/docs/blog/2025-01-29-RealtimeAgent-with-gemini/index",
        "error": false
    },
    {
        "title": "Tools with ChatContext Dependency Injection",
        "url": "https://docs.ag2.ai/docs/blog/2025-01-22-Tools-ChatContext-Dependency-Injection/index",
        "error": false
    },
    {
        "title": "Streaming input and output using WebSockets",
        "url": "https://docs.ag2.ai/docs/blog/2025-01-10-WebSockets/index",
        "error": false
    },
    {
        "title": "Real-Time Voice Interactions over WebRTC",
        "url": "https://docs.ag2.ai/docs/blog/2025-01-09-RealtimeAgent-over-WebRTC/index",
        "error": false
    },
    {
        "title": "Real-Time Voice Interactions with the WebSocket Audio Adapter",
        "url": "https://docs.ag2.ai/docs/blog/2025-01-08-RealtimeAgent-over-websocket/index",
        "error": false
    },
    {
        "title": "Tools Dependency Injection",
        "url": "https://docs.ag2.ai/docs/blog/2025-01-07-Tools-Dependency-Injection/index",
        "error": false
    },
    {
        "title": "Cross-Framework LLM Tool Integration with AG2",
        "url": "https://docs.ag2.ai/docs/blog/2024-12-20-Tools-interoperability/index",
        "error": false
    },
    {
        "title": "ReasoningAgent Update - Beam Search, MCTS, and LATS for LLM Reasoning",
        "url": "https://docs.ag2.ai/docs/blog/2024-12-20-Reasoning-Update/index",
        "error": false
    },
    {
        "title": "Introducing RealtimeAgent Capabilities in AG2",
        "url": "https://docs.ag2.ai/docs/blog/2024-12-20-RealtimeAgent/index",
        "error": false
    },
    {
        "title": "Knowledgeable Agents with FalkorDB Graph RAG",
        "url": "https://docs.ag2.ai/docs/blog/2024-12-06-FalkorDB-Structured/index",
        "error": false
    },
    {
        "title": "ReasoningAgent - Tree of Thoughts with Beam Search in AG2",
        "url": "https://docs.ag2.ai/docs/blog/2024-12-02-ReasoningAgent2/index",
        "error": false
    },
    {
        "title": "Agentic testing for prompt leakage security",
        "url": "https://docs.ag2.ai/docs/blog/2024-11-27-Prompt-Leakage-Probing/index",
        "error": false
    },
    {
        "title": "Building Swarm-based agents with AG2",
        "url": "https://docs.ag2.ai/docs/blog/2024-11-17-Swarm/index",
        "error": false
    },
    {
        "title": "Introducing CaptainAgent for Adaptive Team Building",
        "url": "https://docs.ag2.ai/docs/blog/2024-11-15-CaptainAgent/index",
        "error": false
    },
    {
        "title": "Unlocking the Power of Agentic Workflows at Nexla with Autogen",
        "url": "https://docs.ag2.ai/docs/blog/2024-10-23-NOVA/index",
        "error": false
    },
    {
        "title": "AgentOps, the Best Tool for AutoGen Agent Observability",
        "url": "https://docs.ag2.ai/docs/blog/2024-07-25-AgentOps/index",
        "error": false
    },
    {
        "title": "Enhanced Support for Non-OpenAI Models",
        "url": "https://docs.ag2.ai/docs/blog/2024-06-24-AltModels-Classes/index",
        "error": false
    },
    {
        "title": "AgentEval: A Developer Tool to Assess Utility of LLM-powered Applications",
        "url": "https://docs.ag2.ai/docs/blog/2024-06-21-AgentEval/index",
        "error": false
    },
    {
        "title": "Agents in AutoGen",
        "url": "https://docs.ag2.ai/docs/blog/2024-05-24-Agent/index",
        "error": false
    },
    {
        "title": "AutoDefense - Defend against jailbreak attacks with AutoGen",
        "url": "https://docs.ag2.ai/docs/blog/2024-03-11-AutoDefense/index",
        "error": false
    },
    {
        "title": "What's New in AutoGen?",
        "url": "https://docs.ag2.ai/docs/blog/2024-03-03-AutoGen-Update/index",
        "error": false
    },
    {
        "title": "StateFlow - Build State-Driven Workflows with Customized Speaker Selection in GroupChat",
        "url": "https://docs.ag2.ai/docs/blog/2024-02-29-StateFlow/index",
        "error": false
    },
    {
        "title": "FSM Group Chat -- User-specified agent transitions",
        "url": "https://docs.ag2.ai/docs/blog/2024-02-11-FSM-GroupChat/index",
        "error": false
    },
    {
        "title": "Anny: Assisting AutoGen Devs Via AutoGen",
        "url": "https://docs.ag2.ai/docs/blog/2024-02-02-AutoAnny/index",
        "error": false
    },
    {
        "title": "AutoGen with Custom Models: Empowering Users to Use Their Own Inference Mechanism",
        "url": "https://docs.ag2.ai/docs/blog/2024-01-26-Custom-Models/index",
        "error": false
    },
    {
        "title": "AutoGenBench -- A Tool for Measuring and Evaluating AutoGen Agents",
        "url": "https://docs.ag2.ai/docs/blog/2024-01-25-AutoGenBench/index",
        "error": false
    },
    {
        "title": "Code execution is now by default inside docker container",
        "url": "https://docs.ag2.ai/docs/blog/2024-01-23-Code-execution-in-docker/index",
        "error": false
    },
    {
        "title": "All About Agent Descriptions",
        "url": "https://docs.ag2.ai/docs/blog/2023-12-29-AgentDescriptions/index",
        "error": false
    },
    {
        "title": "AgentOptimizer - An Agentic Way to Train Your LLM Agent",
        "url": "https://docs.ag2.ai/docs/blog/2023-12-23-AgentOptimizer/index",
        "error": false
    },
    {
        "title": "AutoGen Studio: Interactively Explore Multi-Agent Workflows",
        "url": "https://docs.ag2.ai/docs/blog/2023-12-01-AutoGenStudio/index",
        "error": false
    },
    {
        "title": "Agent AutoBuild - Automatically Building Multi-agent Systems",
        "url": "https://docs.ag2.ai/docs/blog/2023-11-26-Agent-AutoBuild/index",
        "error": false
    },
    {
        "title": "How to Assess Utility of LLM-powered Applications?",
        "url": "https://docs.ag2.ai/docs/blog/2023-11-20-AgentEval/index",
        "error": false
    },
    {
        "title": "AutoGen Meets GPTs",
        "url": "https://docs.ag2.ai/docs/blog/2023-11-13-OAI-assistants/index",
        "error": false
    },
    {
        "title": "EcoAssistant - Using LLM Assistants More Accurately and Affordably",
        "url": "https://docs.ag2.ai/docs/blog/2023-11-09-EcoAssistant/index",
        "error": false
    },
    {
        "title": "Multimodal with GPT-4V and LLaVA",
        "url": "https://docs.ag2.ai/docs/blog/2023-11-06-LMM-Agent/index",
        "error": false
    },
    {
        "title": "AutoGen's Teachable Agents",
        "url": "https://docs.ag2.ai/docs/blog/2023-10-26-TeachableAgent/index",
        "error": false
    },
    {
        "title": "Retrieval-Augmented Generation (RAG) Applications with AutoGen",
        "url": "https://docs.ag2.ai/docs/blog/2023-10-18-RetrieveChat/index",
        "error": false
    },
    {
        "title": "Use AutoGen for Local LLMs",
        "url": "https://docs.ag2.ai/docs/blog/2023-07-14-Local-LLMs/index",
        "error": false
    },
    {
        "title": "MathChat - An Conversational Framework to Solve Math Problems",
        "url": "https://docs.ag2.ai/docs/blog/2023-06-28-MathChat/index",
        "error": false
    },
    {
        "title": "Achieve More, Pay Less - Use GPT-4 Smartly",
        "url": "https://docs.ag2.ai/docs/blog/2023-05-18-GPT-adaptive-humaneval/index",
        "error": false
    },
    {
        "title": "Does Model and Inference Parameter Matter in LLM Applications? - A Case Study for MATH",
        "url": "https://docs.ag2.ai/docs/blog/2023-04-21-LLM-tuning-math/index",
        "error": false
    }
]
**********************************************************************

--------------------------------------------------------------------------------
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
assistant (to user_proxy):

I have extracted all the blog posts from the specified URL. Below is the list of titles along with their corresponding links:

1. [RealtimeAgent with Gemini API](https://docs.ag2.ai/docs/blog/2025-01-29-RealtimeAgent-with-gemini/index)
2. [Tools with ChatContext Dependency Injection](https://docs.ag2.ai/docs/blog/2025-01-22-Tools-ChatContext-Dependency-Injection/index)
3. [Streaming input and output using WebSockets](https://docs.ag2.ai/docs/blog/2025-01-10-WebSockets/index)
4. [Real-Time Voice Interactions over WebRTC](https://docs.ag2.ai/docs/blog/2025-01-09-RealtimeAgent-over-WebRTC/index)
5. [Real-Time Voice Interactions with the WebSocket Audio Adapter](https://docs.ag2.ai/docs/blog/2025-01-08-RealtimeAgent-over-websocket/index)
6. [Tools Dependency Injection](https://docs.ag2.ai/docs/blog/2025-01-07-Tools-Dependency-Injection/index)
7. [Cross-Framework LLM Tool Integration with AG2](https://docs.ag2.ai/docs/blog/2024-12-20-Tools-interoperability/index)
8. [ReasoningAgent Update - Beam Search, MCTS, and LATS for LLM Reasoning](https://docs.ag2.ai/docs/blog/2024-12-20-Reasoning-Update/index)
9. [Introducing RealtimeAgent Capabilities in AG2](https://docs.ag2.ai/docs/blog/2024-12-20-RealtimeAgent/index)
10. [Knowledgeable Agents with FalkorDB Graph RAG](https://docs.ag2.ai/docs/blog/2024-12-06-FalkorDB-Structured/index)
11. [ReasoningAgent - Tree of Thoughts with Beam Search in AG2](https://docs.ag2.ai/docs/blog/2024-12-02-ReasoningAgent2/index)
12. [Agentic testing for prompt leakage security](https://docs.ag2.ai/docs/blog/2024-11-27-Prompt-Leakage-Probing/index)
13. [Building Swarm-based agents with AG2](https://docs.ag2.ai/docs/blog/2024-11-17-Swarm/index)
14. [Introducing CaptainAgent for Adaptive Team Building](https://docs.ag2.ai/docs/blog/2024-11-15-CaptainAgent/index)
15. [Unlocking the Power of Agentic Workflows at Nexla with Autogen](https://docs.ag2.ai/docs/blog/2024-10-23-NOVA/index)
16. [AgentOps, the Best Tool for AutoGen Agent Observability](https://docs.ag2.ai/docs/blog/2024-07-25-AgentOps/index)
17. [Enhanced Support for Non-OpenAI Models](https://docs.ag2.ai/docs/blog/2024-06-24-AltModels-Classes/index)
18. [AgentEval: A Developer Tool to Assess Utility of LLM-powered Applications](https://docs.ag2.ai/docs/blog/2024-06-21-AgentEval/index)
19. [Agents in AutoGen](https://docs.ag2.ai/docs/blog/2024-05-24-Agent/index)
20. [AutoDefense - Defend against jailbreak attacks with AutoGen](https://docs.ag2.ai/docs/blog/2024-03-11-AutoDefense/index)
21. [What's New in AutoGen?](https://docs.ag2.ai/docs/blog/2024-03-03-AutoGen-Update/index)
...

TERMINATE

--------------------------------------------------------------------------------

Conclusion

With Browser Use and Crawl4AI, AG2 gets a serious upgrade for web browsing and data extraction. Browser Use makes it easy to navigate and interact with web pages, while Crawl4AI helps scrape and structure data, with or without AI processing.

Whether you need quick web access, detailed data extraction, or AI-powered insights, these tools make it simple to integrate real-time web content into your projects. Now you’re all set to build smarter, more automated agents that can browse, extract, and use web data with ease. 🚀

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for Browser Use and Crawl4AI! Or maybe you would like to see more features or improvements? In any case, do join our Discord server for discussion.

Riding the Web with WebSurferAgent

AG2 Blogger — Wed, 05 Feb 2025 16:30:14 +0000

Author: Robert Jambrecic

Introduction

In our Adding Browsing Capabilities to AG2 guide, we explored how to build agents with basic web surfing capabilities. Now, let’s take it to the next level with WebSurferAgent—a powerful agent that comes with built-in web browsing tools right out of the box!

With WebSurferAgent, your agents can seamlessly browse the web, retrieve real-time information, and interact with web pages—all with minimal setup.

WebSurferAgent with BrowserUseTool

Installation

To get started with the Browser Use integration in AG2, follow these steps:

Install AG2 with the browser-use extra:
```
pip install ag2[browser-use]
```

Set up Playwright:

playwright install
playwright install-deps

For running the code in Jupyter, use nest_asyncio to allow nested event loops.
```
pip install nest_asyncio
```

You’re all set! Now you can start using browsing features in AG2.

Imports

import os

import nest_asyncio

from autogen.agentchat import UserProxyAgent
from autogen.agents import WebSurferAgent

nest_asyncio.apply()

Configure WebSurferAgent with Browser Use

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

WebSurferAgent is the one responsible for browsing the web and retrieving information. The web_tool="browser_use" tells the agent to use the BrowserUseTool to surf the web.

After creating the WebSurferAgent there are two ways to start the chat session:

Recommended: Using the `run` Method

The new run method simplifies the process by eliminating the need for manual UserProxyAgent creation.

✅ Easier setup – No need to manually register tools

websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="browser_use")

websurfer.run(
    message="Get info from https://docs.ag2.ai/docs/Home",
    tools=websurfer.tools,
    max_turns=2,
    user_input=False,
)

Manual Setup: Using `initiate_chat` Method

This method requires manually creating a UserProxyAgent and registering tools for execution.

⚠️ More setup required
⚠️ Must manually register tools

websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="browser_use")
user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
# WebSurferAgent has a list of tools which are registered for LLM
# We need to register the tools for execution with the UserProxyAgent
for tool in websurfer.tools:
    tool.register_for_execution(user_proxy)

user_proxy.initiate_chat(
    recipient=websurfer,
    message="Get info from https://docs.ag2.ai/docs/Home",
    max_turns=2,
)

Output

user_proxy (to WebSurfer):

Get info from https://docs.ag2.ai/docs/Home

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
WebSurfer (to user_proxy):

***** Suggested tool call (call_rl777jGrOGhc68goW5142urK): browser_use *****
Arguments:
{"task":"Get info from https://docs.ag2.ai/docs/Home"}
****************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION browser_use...
Call ID: call_rl777jGrOGhc68goW5142urK
Input arguments: {'task': 'Get info from https://docs.ag2.ai/docs/Home'}
INFO     [agent] 🚀 Starting task: Get info from https://docs.ag2.ai/docs/Home
INFO     [agent]
📍 Step 1
INFO     [agent] 🤷 Eval: Unknown - No previous actions to evaluate.
INFO     [agent] 🧠 Memory:
INFO     [agent] 🎯 Next goal: Navigate to the specified URL to gather information
INFO     [agent] 🛠️  Action 1/1: {"go_to_url":{"url":"https://docs.ag2.ai/docs/Home"}}
INFO     [controller] 🔗  Navigated to https://docs.ag2.ai/docs/Home
INFO     [agent]
📍 Step 2
INFO     [agent] 🤷 Eval: Unknown - No previous actions to evaluate.
INFO     [agent] 🧠 Memory:
INFO     [agent] 🎯 Next goal: Extract useful information regarding AG2 from the home page.
INFO     [agent] 🛠️  Action 1/1: {"extract_content":{"include_links":true}}
INFO     [controller] 📄  Extracted page as markdown
: [AG2 home page![light logo](https://mintlify.s3.us-
west-1.amazonaws.com/ag2ai/logo/ag2.svg)![dark logo](https://mintlify.s3.us-
west-1.amazonaws.com/ag2ai/logo/ag2-white.svg)](/)

Search or ask...

⌘K

Search...A

Navigation

Home

AG2

[Home](/docs/home/Home)[User Guide](/docs/user-guide/quick-start)[API
References](/docs/api-reference/autogen/overview)[Use Cases](/docs/use-
cases/use-cases/customer-service)[Contribute](/contributor-
guide/contributing)[FAQs](/faq/FAQ)[Ecosystem](/ecosystem/agentops)[Blog](/docs/blog/2025-01-29-RealtimeAgent-
with-gemini/index)

##### Home

  * [AG2](/docs/home/Home)

![AG2 Logo](https://mintlify.s3.us-
west-1.amazonaws.com/ag2ai/static/img/ag2.svg)

## AG2

The Open Source Agent OS

[Getting Started - 3 Minute](/docs/user-guide/quick-start)

###



Key Features

##

![Multi-Agent Conversation Framework](https://mintlify.s3.us-
west-1.amazonaws.com/ag2ai/static/img/conv_2.svg)**Multi-Agent Conversation
Framework**

AG2 provides multi-agent conversation framework as a high-level abstraction.
With this framework, one can conveniently build LLM workflows.

##

![Easily Build Diverse Applications](https://mintlify.s3.us-
west-1.amazonaws.com/ag2ai/static/img/autogen_app.svg)**Easily Build Diverse
Applications**

AG2 offers a collection of working systems spanning a wide range of
applications from various domains and complexities.

##

![Enhanced LLM Inference & Optimization](https://mintlify.s3.us-
west-1.amazonaws.com/ag2ai/static/img/extend.svg)**Enhanced LLM Inference &
Optimization**

AG2 supports enhanced LLM inference APIs, which can be used to improve
inference performance and reduce cost.

###



Explore content

## [Quick StartStart building your AG2 application.](/docs/user-guide/quick-
start)## [ConceptsWork through the key concepts of AG2 including
ConversableAgent, GroupChat, Swarm, and tools.](/docs/user-guide/basic-
concepts/installing-ag2)## [Advanced ConceptsAdvance to RAG, Code Execution,
and more complex GroupChats and Swarms.](/docs/user-guide/advanced-concepts)##
[Use CasesTry out use case workflows including Customer Service, Travel
Planning, and Game Design.](/docs/use-cases/use-cases)## [Notebook ExamplesA
collection of interactive notebooks across all AG2 topics.](/docs/use-
cases/notebooks/Notebooks)## [API ReferenceDelve into the AG2 API
reference.](/docs/api-reference)## [How to ContributeGet involved with AG2 by
adding what you need and making the framework even better!](/contributor-
guide/contributing)

###



Popular resources

##

[Foundation Capital Interview with Dr. Chi
Wang](https://www.youtube.com/watch?v=RLwyXRVvlNk)

##

[Learn AG2 on DeepLearningAI](https://www.youtube.com/watch?v=TBNTH-fwGPE)

On this page

  * Key Features
  * Explore content
  * Popular resources



INFO     [agent]
📍 Step 3
INFO     [agent] 👍 Eval: Success - Successfully extracted information from the AG2 home page.
INFO     [agent] 🧠 Memory: Extracted information includes the key features and content related to AG2.
INFO     [agent] 🎯 Next goal: Complete the task and summarize the extracted information.
INFO     [agent] 🛠️  Action 1/1: {"done":{"text":"Extracted information from AG2 home page:\n\n### AG2\n\n**The Open Source Agent OS**  \n[Getting Started - 3 Minute](https://docs/user-guide/quick-start)\n\n#### Key Features\n\n- **Multi-Agent Conversation Framework**: AG2 provides a high-level abstraction for building LLM workflows.\n- **Easily Build Diverse Applications**: Offers a collection of systems across various domains and complexities.\n- **Enhanced LLM Inference & Optimization**: Supports APIs to improve inference performance and reduce costs.\n\n#### Explore Content\n- [Quick Start](https://docs/user-guide/quick-start)\n- [Concepts](https://docs/user-guide/basic-concepts/installing-ag2)\n- [Advanced Concepts](https://docs/user-guide/advanced-concepts)\n- [Use Cases](https://docs/use-cases/use-cases)\n- [API Reference](https://docs/api-reference)\n- [How to Contribute](https://contributor-guide/contributing)  \n\n#### Popular Resources\n- [Foundation Capital Interview with Dr. Chi Wang](https://www.youtube.com/watch?v=RLwyXRVvlNk)  \n- [Learn AG2 on DeepLearningAI](https://www.youtube.com/watch?v=TBNTH-fwGPE)  \n"}}
INFO     [agent] 📄 Result: Extracted information from AG2 home page:

### AG2

**The Open Source Agent OS**
[Getting Started - 3 Minute](https://docs/user-guide/quick-start)

#### Key Features

- **Multi-Agent Conversation Framework**: AG2 provides a high-level abstraction for building LLM workflows.
- **Easily Build Diverse Applications**: Offers a collection of systems across various domains and complexities.
- **Enhanced LLM Inference & Optimization**: Supports APIs to improve inference performance and reduce costs.

#### Explore Content
- [Quick Start](https://docs/user-guide/quick-start)
- [Concepts](https://docs/user-guide/basic-concepts/installing-ag2)
- [Advanced Concepts](https://docs/user-guide/advanced-concepts)
- [Use Cases](https://docs/use-cases/use-cases)
- [API Reference](https://docs/api-reference)
- [How to Contribute](https://contributor-guide/contributing)

#### Popular Resources
- [Foundation Capital Interview with Dr. Chi Wang](https://www.youtube.com/watch?v=RLwyXRVvlNk)
- [Learn AG2 on DeepLearningAI](https://www.youtube.com/watch?v=TBNTH-fwGPE)

INFO     [agent] ✅ Task completed successfully
user_proxy (to WebSurfer):

***** Response from calling tool (call_rl777jGrOGhc68goW5142urK) *****
{"extracted_content":["🔗  Navigated to https://docs.ag2.ai/docs/Home","📄  Extracted page as markdown\n: [AG2 home page![light logo](https://mintlify.s3.us-\nwest-1.amazonaws.com/ag2ai/logo/ag2.svg)![dark logo](https://mintlify.s3.us-\nwest-1.amazonaws.com/ag2ai/logo/ag2-white.svg)](/)\n\nSearch or ask...\n\n⌘K\n\nSearch...\n\nNavigation\n\nHome\n\nAG2\n\n[Home](/docs/home/Home)[User Guide](/docs/user-guide/quick-start)[API\nReferences](/docs/api-reference/autogen/overview)[Use Cases](/docs/use-\ncases/use-cases/customer-service)[Contribute](/contributor-\nguide/contributing)[FAQs](/faq/FAQ)[Ecosystem](/ecosystem/agentops)[Blog](/docs/blog/2025-01-29-RealtimeAgent-\nwith-gemini/index)\n\n##### Home\n\n  * [AG2](/docs/home/Home)\n\n![AG2 Logo](https://mintlify.s3.us-\nwest-1.amazonaws.com/ag2ai/static/img/ag2.svg)\n\n## AG2\n\nThe Open Source Agent OS\n\n[Getting Started - 3 Minute](/docs/user-guide/quick-start)\n\n###\n\n\n\nKey Features\n\n##\n\n![Multi-Agent Conversation Framework](https://mintlify.s3.us-\nwest-1.amazonaws.com/ag2ai/static/img/conv_2.svg)**Multi-Agent Conversation\nFramework**\n\nAG2 provides multi-agent conversation framework as a high-level abstraction.\nWith this framework, one can conveniently build LLM workflows.\n\n##\n\n![Easily Build Diverse Applications](https://mintlify.s3.us-\nwest-1.amazonaws.com/ag2ai/static/img/autogen_app.svg)**Easily Build Diverse\nApplications**\n\nAG2 offers a collection of working systems spanning a wide range of\napplications from various domains and complexities.\n\n##\n\n![Enhanced LLM Inference & Optimization](https://mintlify.s3.us-\nwest-1.amazonaws.com/ag2ai/static/img/extend.svg)**Enhanced LLM Inference &\nOptimization**\n\nAG2 supports enhanced LLM inference APIs, which can be used to improve\ninference performance and reduce cost.\n\n###\n\n\n\nExplore content\n\n## [Quick StartStart building your AG2 application.](/docs/user-guide/quick-\nstart)## [ConceptsWork through the key concepts of AG2 including\nConversableAgent, GroupChat, Swarm, and tools.](/docs/user-guide/basic-\nconcepts/installing-ag2)## [Advanced ConceptsAdvance to RAG, Code Execution,\nand more complex GroupChats and Swarms.](/docs/user-guide/advanced-concepts)##\n[Use CasesTry out use case workflows including Customer Service, Travel\nPlanning, and Game Design.](/docs/use-cases/use-cases)## [Notebook ExamplesA\ncollection of interactive notebooks across all AG2 topics.](/docs/use-\ncases/notebooks/Notebooks)## [API ReferenceDelve into the AG2 API\nreference.](/docs/api-reference)## [How to ContributeGet involved with AG2 by\nadding what you need and making the framework even better!](/contributor-\nguide/contributing)\n\n###\n\n\n\nPopular resources\n\n##\n\n[Foundation Capital Interview with Dr. Chi\nWang](https://www.youtube.com/watch?v=RLwyXRVvlNk)\n\n##\n\n[Learn AG2 on DeepLearningAI](https://www.youtube.com/watch?v=TBNTH-fwGPE)\n\nOn this page\n\n  * Key Features\n  * Explore content\n  * Popular resources\n\n\n","Extracted information from AG2 home page:\n\n### AG2\n\n**The Open Source Agent OS**  \n[Getting Started - 3 Minute](https://docs/user-guide/quick-start)\n\n#### Key Features\n\n- **Multi-Agent Conversation Framework**: AG2 provides a high-level abstraction for building LLM workflows.\n- **Easily Build Diverse Applications**: Offers a collection of systems across various domains and complexities.\n- **Enhanced LLM Inference & Optimization**: Supports APIs to improve inference performance and reduce costs.\n\n#### Explore Content\n- [Quick Start](https://docs/user-guide/quick-start)\n- [Concepts](https://docs/user-guide/basic-concepts/installing-ag2)\n- [Advanced Concepts](https://docs/user-guide/advanced-concepts)\n- [Use Cases](https://docs/use-cases/use-cases)\n- [API Reference](https://docs/api-reference)\n- [How to Contribute](https://contributor-guide/contributing)  \n\n#### Popular Resources\n- [Foundation Capital Interview with Dr. Chi Wang](https://www.youtube.com/watch?v=RLwyXRVvlNk)  \n- [Learn AG2 on DeepLearningAI](https://www.youtube.com/watch?v=TBNTH-fwGPE)  \n"],"final_result":"Extracted information from AG2 home page:\n\n### AG2\n\n**The Open Source Agent OS**  \n[Getting Started - 3 Minute](https://docs/user-guide/quick-start)\n\n#### Key Features\n\n- **Multi-Agent Conversation Framework**: AG2 provides a high-level abstraction for building LLM workflows.\n- **Easily Build Diverse Applications**: Offers a collection of systems across various domains and complexities.\n- **Enhanced LLM Inference & Optimization**: Supports APIs to improve inference performance and reduce costs.\n\n#### Explore Content\n- [Quick Start](https://docs/user-guide/quick-start)\n- [Concepts](https://docs/user-guide/basic-concepts/installing-ag2)\n- [Advanced Concepts](https://docs/user-guide/advanced-concepts)\n- [Use Cases](https://docs/use-cases/use-cases)\n- [API Reference](https://docs/api-reference)\n- [How to Contribute](https://contributor-guide/contributing)  \n\n#### Popular Resources\n- [Foundation Capital Interview with Dr. Chi Wang](https://www.youtube.com/watch?v=RLwyXRVvlNk)  \n- [Learn AG2 on DeepLearningAI](https://www.youtube.com/watch?v=TBNTH-fwGPE)  \n"}
**********************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
WebSurfer (to user_proxy):

Here's the information extracted from the AG2 home page:

### AG2
**The Open Source Agent OS**
[Getting Started - 3 Minute](https://docs/user-guide/quick-start)

#### Key Features
- **Multi-Agent Conversation Framework**: AG2 provides a high-level abstraction for building LLM (Large Language Model) workflows.
- **Easily Build Diverse Applications**: Offers a collection of systems across various domains and complexities.
- **Enhanced LLM Inference & Optimization**: Supports enhanced inference APIs to improve performance and reduce costs.

#### Explore Content
- [Quick Start](https://docs/user-guide/quick-start)
- [Concepts](https://docs/user-guide/basic-concepts/installing-ag2)
- [Advanced Concepts](https://docs/user-guide/advanced-concepts)
- [Use Cases](https://docs/use-cases/use-cases)
- [API Reference](https://docs/api-reference)
- [How to Contribute](https://contributor-guide/contributing)

#### Popular Resources
- [Foundation Capital Interview with Dr. Chi Wang](https://www.youtube.com/watch?v=RLwyXRVvlNk)
- [Learn AG2 on DeepLearningAI](https://www.youtube.com/watch?v=TBNTH-fwGPE)

--------------------------------------------------------------------------------

WebSurferAgent with Crawl4AITool

Installation

To integrate Crawl4AI with AG2, follow these steps:

Install AG2 with the crawl4ai extra:
```
pip install ag2[crawl4ai]
```

Set up Playwright:

playwright install
playwright install-deps

For running the code in Jupyter, use nest_asyncio to allow nested event loops.
```
pip install nest_asyncio
```

Once installed, you’re ready to start using the browsing features in AG2.

Imports

import os

import nest_asyncio

from autogen.agentchat import UserProxyAgent
from autogen.agents import WebSurferAgent

nest_asyncio.apply()

Configure WebSurferAgent with Crawl4AI

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

The only difference from the previous example is that the web_tool parameter must be set to crawl4ai in order for the Crawl4AITool to be used.

As in previous example, there are two ways to start the chat session:

Recommended: Using the `run` Method

The new run method simplifies the process by eliminating the need for manual UserProxyAgent creation.

✅ Easier setup – No need to manually register tools

# `web_tool` parameter must be set to `crawl4ai` in order for the `Crawl4AITool` to be used.
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="crawl4ai")

websurfer.run(
    message="Get info from https://docs.ag2.ai/docs/Home",
    tools=websurfer.tools,
    max_turns=2,
    user_input=False,
)

Manual Setup: Using `initiate_chat` Method

This method requires manually creating a UserProxyAgent and registering tools for execution.

⚠️ More setup required
⚠️ Must manually register tools

user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="crawl4ai")

websurfer_tools = websurfer.tools
# WebSurferAgent has a list of tools which are registered for LLM
# We need to register the tools for execution with the UserProxyAgent
for tool in websurfer_tools:
    tool.register_for_execution(user_proxy)

user_proxy.initiate_chat(
    recipient=websurfer,
    message="Get info from https://docs.ag2.ai/docs/Home",
    max_turns=2,
)

Output

user_proxy (to WebSurfer):

Get info from https://docs.ag2.ai/docs/Home

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
WebSurfer (to user_proxy):

***** Suggested tool call (call_UvcLMFR8osM9AsyxVdmzgmBs): crawl4ai *****
Arguments:
{"url":"https://docs.ag2.ai/docs/Home","instruction":"Extract the main sections and any key information such as features, usage guidelines, and any other relevant details."}
*************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION crawl4ai...
Call ID: call_UvcLMFR8osM9AsyxVdmzgmBs
Input arguments: {'url': 'https://docs.ag2.ai/docs/Home', 'instruction': 'Extract the main sections and any key information such as features, usage guidelines, and any other relevant details.'}
[INIT].... → Crawl4AI 0.4.247
[FETCH]... ↓ https://docs.ag2.ai/docs/Home... | Status: True | Time: 1.36s
[SCRAPE].. ◆ Processed https://docs.ag2.ai/docs/Home... | Time: 47ms
INFO:httpx:HTTP Request: GET https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json "HTTP/1.1 200 OK"
/Users/robert/projects/ag2/.venv-crawl4ai/lib/python3.11/site-packages/pydantic/_internal/_config.py:345: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
14:23:43 - LiteLLM:INFO: utils.py:2825 -
LiteLLM completion() model= gpt-4o-mini; provider = openai
INFO:LiteLLM:
LiteLLM completion() model= gpt-4o-mini; provider = openai
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
14:23:57 - LiteLLM:INFO: utils.py:1030 - Wrapper: Completed Call, calling success_handler
INFO:LiteLLM:Wrapper: Completed Call, calling success_handler
[EXTRACT]. ■ Completed for https://docs.ag2.ai/docs/Home... | Time: 15.169699541991577s
[COMPLETE] ● https://docs.ag2.ai/docs/Home... | Status: True | Total: 16.59s
user_proxy (to WebSurfer):

***** Response from calling tool (call_UvcLMFR8osM9AsyxVdmzgmBs) *****
[
    {
        "index": 0,
        "tags": [
            "introduction"
        ],
        "content": [
            "# AG2",
            "The Open Source Agent OS"
        ],
        "error": false
    },
    {
        "index": 1,
        "tags": [
            "getting_started"
        ],
        "content": [
            "[Getting Started - 3 Minute](https://docs.ag2.ai/docs/</docs/user-guide/quick-start>)"
        ],
        "error": false
    },
    {
        "index": 2,
        "tags": [
            "key_features"
        ],
        "content": [
            "Key Features",
            "**Multi-Agent Conversation Framework**",
            "AG2 provides multi-agent conversation framework as a high-level abstraction. With this framework, one can conveniently build LLM workflows.",
            "**Easily Build Diverse Applications**",
            "AG2 offers a collection of working systems spanning a wide range of applications from various domains and complexities.",
            "**Enhanced LLM Inference & Optimization**",
            "AG2 supports enhanced LLM inference APIs, which can be used to improve inference performance and reduce cost."
        ],
        "error": false
    },
    {
        "index": 3,
        "tags": [
            "explore_content"
        ],
        "content": [
            "## [Quick StartStart building your AG2 application.](https://docs.ag2.ai/docs/</docs/user-guide/quick-start>)",
            "## [ConceptsWork through the key concepts of AG2 including ConversableAgent, GroupChat, Swarm, and tools.](https://docs.ag2.ai/docs/</docs/user-guide/basic-concepts/installing-ag2>)",
            "## [Advanced ConceptsAdvance to RAG, Code Execution, and more complex GroupChats and Swarms.](https://docs.ag2.ai/docs/</docs/user-guide/advanced-concepts>)",
            "## [Use CasesTry out use case workflows including Customer Service, Travel Planning, and Game Design.](https://docs.ag2.ai/docs/</docs/use-cases/use-cases>)",
            "## [Notebook ExamplesA collection of interactive notebooks across all AG2 topics.](https://docs.ag2.ai/docs/</docs/use-cases/notebooks/Notebooks>)",
            "## [API ReferenceDelve into the AG2 API reference.](https://docs.ag2.ai/docs/</docs/api-reference>)",
            "## [How to ContributeGet involved with AG2 by adding what you need and making the framework even better!](https://docs.ag2.ai/docs/</contributor-guide/contributing>)"
        ],
        "error": false
    },
    {
        "index": 4,
        "tags": [
            "popular_resources"
        ],
        "content": [
            "Popular resources",
            "[Foundation Capital Interview with Dr. Chi Wang](https://docs.ag2.ai/docs/<https:/www.youtube.com/watch?v=RLwyXRVvlNk>)",
            "[Learn AG2 on DeepLearningAI](https://docs.ag2.ai/docs/<https:/www.youtube.com/watch?v=TBNTH-fwGPE>)"
        ],
        "error": false
    }
]
**********************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
WebSurfer (to user_proxy):

Here is the extracted information from the AG2 documentation website:

### Introduction
- **AG2**: The Open Source Agent OS

### Getting Started
- [Quick Start - 3 Minute Guide](https://docs.ag2.ai/docs/user-guide/quick-start)

### Key Features
1. **Multi-Agent Conversation Framework**: AG2 provides a framework for multi-agent conversations, allowing users to build LLM workflows conveniently.
2. **Easily Build Diverse Applications**: AG2 comes with a collection of systems that cater to a range of applications across various domains and complexities.
3. **Enhanced LLM Inference & Optimization**: The platform supports enhanced LLM inference APIs, improving performance and reducing costs.

### Explore Content
- [Quick Start: Start building your AG2 application.](https://docs.ag2.ai/docs/user-guide/quick-start)
- [Concepts: Learn about key concepts including ConversableAgent, GroupChat, Swarm, and tools.](https://docs.ag2.ai/docs/user-guide/basic-concepts/installing-ag2)
- [Advanced Concepts: Explore topics such as RAG, Code Execution, and complex GroupChats and Swarms.](https://docs.ag2.ai/docs/user-guide/advanced-concepts)
- [Use Cases: Experiment with workflows in areas like Customer Service, Travel Planning, and Game Design.](https://docs.ag2.ai/docs/use-cases/use-cases)
- [Notebook Examples: Access a collection of interactive notebooks covering all AG2 topics.](https://docs.ag2.ai/docs/use-cases/notebooks/Notebooks)
- [API Reference: Detailed exploration of the AG2 API reference.](https://docs.ag2.ai/docs/api-reference)
- [How to Contribute: Guidelines for getting involved with AG2 and enhancing the framework.](https://docs.ag2.ai/docs/contributor-guide/contributing)

### Popular Resources
...

This summary covers the main sections, key features, links to further exploration, and resources available in the documentation.

--------------------------------------------------------------------------------

Conclusion

In this post, we’ve shown how to boost your agents with web browsing abilities using the WebSurferAgent. By using tools like BrowserUseTool and Crawl4AITool, your agents can easily fetch real-time information from the web. This makes your agents more useful and flexible, whether you’re getting data from specific pages or gathering info across different topics. With these tools, AG2 helps you create smarter agents that can navigate the web and bring back the details you need with minimal effort.

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for WebSurferAgent! Or maybe you would like to see more features or improvements? In any case, do join our Discord server for discussion.

RealtimeAgent with Gemini API

AG2 Blogger — Tue, 04 Feb 2025 19:42:55 +0000

Authors: Stella Xiang, Mark Sze, Tvrtko Sternak, Davor Runje

TL;DR:

RealtimeAgent now supports Gemini Multimodal Live API

Why is this important?

We previously supported a Realtime Agent powered by OpenAI. In December 2024, Google rolled out Gemini 2.0, which includes the multi-modal live APIs. These APIs enable advanced capabilities such as real-time processing of audio inputs in live conversational settings. To ensure developers can fully leverage the capabilities of the latest LLMs, we now also support a RealtimeAgent powered by Gemini.

How to Use?

To ensure a seamless experience for developers, we aim to minimize the required changes. The key step is to properly configure your LLM settings, including credentials, LLM setup, and tags. Once this is done, switching between different LLMs becomes straightforward.

For a practical walkthrough, refer to this Example project, which demonstrates how to instantiate a Gemini client and configure it within AG2. This includes an overview of WebSocket integration, a key feature for live-streaming scenarios.

For the highlights and demos of Gemini 2.0, check out their official tech blog.

Key Feature of Gemini Integration

Optimized for Realtime Low-latency processing makes Gemini a strong choice for live applications. Combined with AG2’s orchestration capabilities, developers can build robust systems that respond in near real time.

Considerations

During the implementation of this agent, we observed that audio truncation is not currently natively supported by Gemini. For instance, if the server generates a 10-second audio clip, and only the first 5 seconds are played while the rest is truncated, the server may remain unaware that the remaining 5 seconds were not played.

This limitation highlights an important consideration for applications that rely on fine-grained control over audio playback, such as interactive storytelling or call center solutions.

However, the APIs and models are evolving rapidly, and things could change very fast. With AG2, switching between models is streamlined, making it easier to adapt to different use cases and overcome such challenges.

Future Potential

Our decision to support Gemini opens up exciting possibilities for developers:

Customizable LLM Pipelines AG2’s architecture allows developers to orchestrate workflows involving Gemini and other LLMs (e.g., OpenAI or Cohere), enabling the creation of highly tailored solutions.
Expanding Model Ecosystem By supporting both OpenAI and Gemini, AG2 ensures flexibility and scalability for developers. This reduces vendor lock-in and allows teams to experiment with the strengths of different models to meet their specific needs.

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for RealtimeAgent with Gemini API! Or maybe you would like to see more features or improvements? In any case, do join our Discord server for discussion.

Tools with ChatContext Dependency Injection

AG2 Blogger — Mon, 27 Jan 2025 14:34:48 +0000

Author: Robert Jambrecic

Introduction

In this post, we’ll build upon the concepts introduced in our previous blog on Tools with Dependency Injection. We’ll take a deeper look at how ChatContext can be used to manage the flow of conversations in a more structured and secure way.

By using ChatContext, we can track and control the sequence of function calls during a conversation. This is particularly useful in situations where one task must be completed before another — for example, ensuring that a user logs in before they can check their account balance. This approach helps to prevent errors and enhances the security of the system.

Benefits of Using ChatContext:

Flow Control: Ensures tasks are performed in the correct order, reducing the chance of mistakes.
Enhanced Security: Prevents unauthorized actions, such as accessing sensitive data before authentication.
Simplified Debugging: Logs the conversation history, making it easier to trace and resolve issues.

This blog builds on the concepts shared in the notebook.

Installation

To install AG2, simply run the following command:

pip install ag2

Imports

import os
from typing import Annotated, Literal

from pydantic import BaseModel

from autogen.agentchat import AssistantAgent, UserProxyAgent
from autogen.tools.dependency_injection import BaseContext, ChatContext, Depends

Account and Helper Functions

The following Account class and helper functions are adapted from the Tools with Dependency Injection post. They define the structure for securely handling account data and operations like login and balance retrieval.

Define Account Class

class Account(BaseContext, BaseModel):
    username: str
    password: str
    currency: Literal["USD", "EUR"] = "USD"


alice_account = Account(username="alice", password="password123")
bob_account = Account(username="bob", password="password456")

account_ballace_dict = {
    (alice_account.username, alice_account.password): 300,
    (bob_account.username, bob_account.password): 200,
}

Helper Functions

These functions validate account credentials and retrieve account balances.

def _verify_account(account: Account):
    if (account.username, account.password) not in account_ballace_dict:
        raise ValueError("Invalid username or password")


def _get_balance(account: Account):
    _verify_account(account)
    return f"Your balance is {account_ballace_dict[(account.username, account.password)]}{account.currency}"

Agent Configuration

Configure the agents for the interaction.

config_list defines the LLM configurations, including the model and API key.
UserProxyAgent simulates user inputs without requiring actual human interaction (set to NEVER).
AssistantAgent represents the AI agent, configured with the LLM settings.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]
agent = AssistantAgent(
    name="agent",
    llm_config={"config_list": config_list},
)
user_proxy = UserProxyAgent(
    name="user_proxy_1",
    human_input_mode="NEVER",
    llm_config=False,
)

Injecting a ChatContext Parameter

Now let’s upgrade the example from the previous post by introducing the ChatContext parameter. This enhancement allows us to enforce proper execution order in the workflow, ensuring that users log in before accessing sensitive data like account balances.

The following functions will be registered:

login: Verifies the user’s credentials and ensures they are logged in.
get_balance: Retrieves the account balance but only if the user has successfully logged in first.

@user_proxy.register_for_execution()
@agent.register_for_llm(description="Login")
def login(
    account: Annotated[Account, Depends(bob_account)],
) -> str:
    _verify_account(account)
    return "You are logged in"


@user_proxy.register_for_execution()
@agent.register_for_llm(description="Get balance")
def get_balance(
    account: Annotated[Account, Depends(bob_account)],
    chat_context: ChatContext,
) -> str:
    _verify_account(account)

    # Extract the list of messages exchanged with the first agent in the conversation.
    # The chat_context.chat_messages is a dictionary where keys are agents (objects)
    # and values are lists of message objects. We take the first value (messages of the first agent).
    messages_with_first_agent = list(chat_context.chat_messages.values())[0]

    login_function_called = False
    for message in messages_with_first_agent:
        if "tool_calls" in message and message["tool_calls"][0]["function"]["name"] == "login":
            login_function_called = True
            break

    if not login_function_called:
        raise ValueError("Please login first")

    balance = _get_balance(account)
    return balance

Finally, we initiate a chat to retrieve the balance.

user_proxy.initiate_chat(agent, message="Get users balance", max_turns=4)

When user_proxy.initiate_chat(agent, message="Get users balance", max_turns=4) is called, the following happens:

User Request: The user sends the message "Get users balance", prompting the agent to suggest calling get_balance.
First Function Call: The agent attempts to execute get_balance, but since the user isn’t logged in, it raises an error: "Please login first". The agent then suggests calling login to authenticate the user.
Login Execution: The login function is executed, confirming the user’s login with "You are logged in".
Second Function Call: With the user logged in, the agent successfully calls get_balance and retrieves the balance.
Final Response: The agent responds with "Your balance is 200 USD", confirming the correct execution of the workflow.

user_proxy_1 (to agent):

Get users balance

--------------------------------------------------------------------------------
agent (to user_proxy_1):

***** Suggested tool call (call_aeSi4R4Mo7f6JpSLTpll6lmH): get_balance *****
Arguments:
{}
****************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION get_balance...
Call ID: call_aeSi4R4Mo7f6JpSLTpll6lmH
Input arguments: {}
user_proxy_1 (to agent):

***** Response from calling tool (call_aeSi4R4Mo7f6JpSLTpll6lmH) *****
Error: Please login first
**********************************************************************

--------------------------------------------------------------------------------
agent (to user_proxy_1):

***** Suggested tool call (call_7hY8yZWjrD7Irh1mKvFJL40y): login *****
Arguments:
{}
**********************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION login...
Call ID: call_7hY8yZWjrD7Irh1mKvFJL40y
Input arguments: {}
user_proxy_1 (to agent):

***** Response from calling tool (call_7hY8yZWjrD7Irh1mKvFJL40y) *****
You are logged in
**********************************************************************

--------------------------------------------------------------------------------
agent (to user_proxy_1):

***** Suggested tool call (call_yTSqe4kxUbCH6Kxub4wiYlIY): get_balance *****
Arguments:
{}
****************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION get_balance...
Call ID: call_yTSqe4kxUbCH6Kxub4wiYlIY
Input arguments: {}
user_proxy_1 (to agent):

***** Response from calling tool (call_yTSqe4kxUbCH6Kxub4wiYlIY) *****
Your balance is 200USD
**********************************************************************

--------------------------------------------------------------------------------
agent (to user_proxy_1):

Your balance is 200 USD.

TERMINATE

--------------------------------------------------------------------------------

Conclusion

In this post, we introduced ChatContext as an extension of dependency injection, showing how it simplifies and secures workflows in conversational AI by ensuring tasks like login happen before accessing sensitive data, improving both functionality and security.

Highlights:

Flow Control: ChatContext can ensure that functions are executed in the correct sequence, reducing the risk of logical errors.
Enhanced Security: Sensitive actions like balance retrieval are protected by preconditions like login verification.
Simplified Debugging: With ChatContext, developers can trace conversations, making it easier to identify and resolve issues.

By leveraging tools like ChatContext and employing dependency injection principles, you can build robust, secure, and user-friendly conversational workflows.

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for ChatContext! Or maybe you would like to see more features or improvements? In any case, do join our Discord server for discussion.

Streaming input and output using WebSockets

AG2 Blogger — Wed, 15 Jan 2025 15:59:00 +0000

Authors: Mark Sze, Tvrtko Sternak, Davor Runje

TL;DR

Learn how to build an agent chat application using WebSockets and IOStream
Explore a hands-on example of connecting a web application to a responsive chat with agents over WebSockets.
Streamlined Real-Time Interactions: WebSockets offer a low-latency, persistent connection for sending and receiving data in real time.

Real-Time Applications: Why WebSockets?

WebSockets provide a powerful framework for real-time communication between a client and server. Unlike traditional HTTP requests, which require polling for updates, WebSockets establish a persistent, full-duplex connection that allows for continuous data exchange.

This capability is critical for applications that use AG2, where seamless interaction is essential.

Key Benefits of WebSockets

Low Latency: WebSockets reduce latency by maintaining a direct, open connection, avoiding the overhead of repeated HTTP handshakes.
Efficient Data Streaming: Continuous, two-way data streams enable smooth user experiences in real-time applications.
Event-Driven Communication: With WebSocket protocols, the server can “push” updates to the client as events occur.
Simplified Architecture: WebSockets eliminate the need for separate polling mechanisms, reducing server load and complexity.

Building a Chat System

This example demonstrates how to create a WebSocket-based chat system that streams real-time input and output from AG2 Agents.

How It Works

WebSocket Connection: The client establishes a persistent WebSocket connection to the server.
Real-Time Data Flow: Events in the conversation are streamed over WebSockets to the browser where they can be displayed.

Example: Creating a Weather chat app

Let’s walk through an example that integrates WebSockets with a weather-focused chat.

You can explore the full example code here.

1. Clone the Repository

git clone https://github.com/ag2ai/agentchat-over-websockets.git
cd agentchat-over-websockets

2. Set Up Environment Variables

Create a OAI_CONFIG_LIST file based on the provided OAI_CONFIG_LIST_sample:

cp OAI_CONFIG_LIST_sample OAI_CONFIG_LIST

In the OAI_CONFIG_LIST file, update the api_key to your OpenAI API key.

(Optional) Create and use a virtual environment

To reduce cluttering your global Python environment on your machine, you can create a virtual environment. On your command line, enter:

python3 -m venv env
source env/bin/activate

3. Install Dependencies

Install the required Python packages using pip:

pip install -r requirements.txt

4. Start the Server

Run the main.py file:

python agentchat-over-websockets/main.py

Test the App

With the server running, open the client application in your browser by navigating to http://localhost:8001/. And send a message to the chat and watch the conversation between agents roll out in your browser.

Code review

Backend Code: `main.py`

The backend is responsible for serving the frontend, managing WebSocket connections, and hosting the AI-powered conversational agent. Below is a step-by-step breakdown.

Setting Up the WebSocket Server

The IOWebsockets.run_server_in_thread utility is used to run a WebSocket server. The on_connect function handles new client connections and initializes the chatbot.

from autogen.io.websockets import IOWebsockets
from datetime import datetime

def on_connect(iostream: IOWebsockets) -> None:
    print(f"Connected to client: {iostream}")
    initial_msg = iostream.input()  # Receive the first message from the client.
    print(f"Initial message: {initial_msg}")

    # Define the agent
    agent = autogen.ConversableAgent(
        name="chatbot",
        system_message="Complete tasks and reply TERMINATE when done. Use the 'weather_forecast' tool for weather-related queries.",
        llm_config={"stream": False},
    )

    # Define the user proxy
    user_proxy = autogen.UserProxyAgent(
        name="user_proxy",
        system_message="A proxy for the user.",
        is_termination_msg=lambda msg: msg.get("content", "").endswith("TERMINATE"),
        human_input_mode="NEVER",
    )

    # Register tool functions
    def weather_forecast(city: str) -> str:
        return f"The weather forecast for {city} is sunny as of {datetime.now()}."

    autogen.register_function(
        weather_forecast,
        caller=agent,
        executor=user_proxy,
        description="Provides a mock weather forecast.",
    )

    # Initiate conversation
    user_proxy.initiate_chat(agent, message=initial_msg)

Explanation:

on_connect: Handles client connections and manages the interaction between the ConversableAgent and the client.
Tool Registration: The weather_forecast function provides a mock weather report and is linked to the agent for handling weather-related queries.Serving the Frontend

The SimpleHTTPRequestHandler is used to serve HTML files. A custom handler class overrides the behavior for the root path to serve chat.html.

class MyRequestHandler(SimpleHTTPRequestHandler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, directory=Path(__file__).parent / "website_files" / "templates", **kwargs)

    def do_GET(self):
        if self.path == "/":
            self.path = "/chat.html"
        return super().do_GET()

Explanation:

The MyRequestHandler class ensures that the default page served is chat.html.
Files are served from the website_files/templates directory.

Running the Servers

Finally, both the WebSocket and HTTP servers are started.

from http.server import HTTPServer

PORT = 8001

handler = MyRequestHandler

# Start WebSocket server
with IOWebsockets.run_server_in_thread(on_connect=on_connect, port=8080) as uri:
    print(f"WebSocket server started at {uri}")

    # Start HTTP server
    with HTTPServer(("", PORT), handler) as httpd:
        print(f"HTTP server started at http://localhost:{PORT}")
        try:
            httpd.serve_forever()
        except KeyboardInterrupt:
            print("HTTP server stopped.")

Explanation:

The WebSocket server listens on port 8080, while the HTTP server listens on port 8001.
The WebSocket server handles real-time communication, while the HTTP server serves static files.

Frontend Code: `chat.html`

The frontend provides a simple interface for users to interact with the chatbot.

HTML Structure

The HTML structure defines an input form for sending messages and a list for displaying them.

<!DOCTYPE html>
<html>
<head>
    <title>Chat Interface</title>
    <style>
        body { font-family: monospace; max-width: 800px; margin: 20px auto; }
        #messages { list-style: none; padding: 0; }
        #messages li { background: #f1f3f4; padding: 8px; border-radius: 4px; margin: 4px 0; }
    </style>
</head>
<body>
    <h1>AI Chat Interface</h1>
    <form onsubmit="sendMessage(event)">
        <input type="text" id="messageText" autocomplete="off" />
        <button>Send</button>
    </form>
    <ul id="messages"></ul>
</body>
</html>

JavaScript Logic

The JavaScript code establishes a WebSocket connection, handles incoming messages, and sends user input to the backend.

var ws = new WebSocket("ws://localhost:8080");

ws.onmessage = function(event) {
    var messages = document.getElementById('messages');
    var message = document.createElement('li');
    message.textContent = event.data;  // Display the message content.
    messages.appendChild(message);
};

function sendMessage(event) {
    var input = document.getElementById("messageText");
    ws.send(input.value);  // Send the input value to the backend.
    input.value = '';  // Clear the input field.
    event.preventDefault();  // Prevent form submission.
}

Explanation:

WebSocket Initialization: Connects to the WebSocket server at ws://localhost:8080.
Message Display: Appends incoming messages to the #messages list.
Sending Messages: Captures user input, sends it to the server, and clears the input field.

Conclusion

Building an AgentChat system with WebSockets unlocks the potential for real-time, interactive applications. By maintaining a persistent connection, WebSockets enable seamless communication, enhancing user experience with minimal latency.

The example of a weather chatbot demonstrates the ease of integrating AG2 with WebSockets to create dynamic conversational agents. Whether for customer support, virtual assistants, or personalized services, this architecture provides a robust foundation for building next-generation applications.

Ready to start building? Explore the full example code here.

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for AgentChat with WebSockets? Or maybe you would like to see more features or improvements? Do join our Discord server for discussion.

Real-Time Voice Interactions over WebRTC

AG2 Blogger — Tue, 14 Jan 2025 17:19:07 +0000

Authors: Mark Sze, Tvrtko Sternak, Davor Runje, Davorin Rusevljan

TL;DR:

Build a real-time voice application using WebRTC and connect it with the RealtimeAgent. Demo implementation.
Optimized for Real-Time Interactions: Experience seamless voice communication with minimal latency and enhanced reliability.

Realtime Voice Applications with WebRTC

In our previous blog post, we introduced the WebSocketAudioAdapter, a simple way to stream real-time audio using WebSockets. While effective, WebSockets can face challenges with quality and reliability in high-latency or network-variable scenarios. Enter WebRTC.

Today, we’re excited to showcase the integration with OpenAI Realtime API with WebRTC, leveraging WebRTC’s peer-to-peer communication capabilities to provide a robust, low-latency, high-quality audio streaming experience directly from the browser.

Why WebRTC?

WebRTC (Web Real-Time Communication) is a powerful technology for enabling direct peer-to-peer communication between browsers and servers. It was built with real-time audio, video, and data transfer in mind, making it an ideal choice for real-time voice applications. Here are some key benefits:

1. Low Latency

WebRTC’s peer-to-peer design minimizes latency, ensuring natural, fluid conversations.

2. Adaptive Quality

WebRTC dynamically adjusts audio quality based on network conditions, maintaining a seamless user experience even in suboptimal environments.

3. Secure by Design

With encryption (DTLS and SRTP) baked into its architecture, WebRTC ensures secure communication between peers.

4. Widely Supported

WebRTC is supported by all major modern browsers, making it highly accessible for end users.

How It Works

This example demonstrates using WebRTC to establish low-latency, real-time interactions with OpenAI Realtime API with WebRTC from a web browser. Here’s how it works:

1. Request an Ephemeral API Key

* The browser connects to your backend via [**WebSockets**](https://fastapi.tiangolo.com/advanced/websockets/) to exchange configuration details, such as the ephemeral key and model information.

* [**WebSockets**](https://fastapi.tiangolo.com/advanced/websockets/) handle signaling to bootstrap the [**WebRTC**](https://webrtc.org/) session.

* The browser requests a short-lived API key from your server.

2. Generate an Ephemeral API Key

* Your backend generates an ephemeral key via the OpenAI REST API and returns it. These keys expire after one minute to enhance security.

3. Initialize the WebRTC Connection

* **Audio Streaming**: The browser captures microphone input and streams it to OpenAI while playing audio responses via an `<audio>` element.

* **DataChannel**: A `DataChannel` is established to send and receive events (e.g., function calls).

* **Session Handshake**: The browser creates an SDP offer, sends it to OpenAI with the ephemeral key, and sets the remote SDP answer to finalize the connection.

* The audio stream and events flow in real time, enabling interactive, low-latency conversations.

Example: Build a Voice-Enabled Language Translator

Let’s walk through a practical example of using WebRTC to create a voice-enabled language translator.

You can find the full example here.

1. Clone the Repository

Start by cloning the example project from GitHub:

git clone https://github.com/ag2ai/realtime-agent-over-webrtc.git
cd realtime-agent-over-webrtc

2. Set Up Environment Variables

Create a OAI_CONFIG_LIST file based on the provided OAI_CONFIG_LIST_sample:

cp OAI_CONFIG_LIST_sample OAI_CONFIG_LIST

In the OAI_CONFIG_LIST file, update the api_key with your OpenAI API key.

Supported key format

Currently WebRTC can be used only by API keys the begin with:

sk-proj

Other keys may result internal server error (500) on OpenAI server. For more details see this issue

(Optional) Create and Use a Virtual Environment

To avoid cluttering your global Python environment:

python3 -m venv env
source env/bin/activate

3. Install Dependencies

Install the required Python packages:

pip install -r requirements.txt

4. Start the Server

Run the application with Uvicorn:

uvicorn realtime_over_webrtc.main:app --port 5050

When the server starts, you should see:

INFO:     Started server process [12345]
INFO:     Uvicorn running on http://0.0.0.0:5050 (Press CTRL+C to quit)

5. Open the Application

Navigate to localhost:5050/start-chat in your browser. The application will request microphone permissions to enable real-time voice interaction.

6. Start Speaking

To get started, simply speak into your microphone and ask a question. For example, you can say:

“What’s the weather like in Rome?”

This initial question will activate the agent, and it will respond, showcasing its ability to understand and interact with you in real time.

Code review

WebRTC connection

A lot of the WebRTC connection logic happens in the website_files/static /WebRTC.js, so lets take a look at the code there first.

WebSocket Initialization

The WebSocket is responsible for exchanging initialization data and signaling messages.

ws = new WebSocket(webSocketUrl);

ws.onmessage = async event => {
    const message = JSON.parse(event.data);
    console.info("Received Message from AG2 backend", message);
    if (message.type === "ag2.init") {
        await openRTC(message.config); // Starts the WebRTC connection
        return;
    }
    if (dc) {
        dc.send(JSON.stringify(message)); // Sends data via DataChannel
    } else {
        console.log("DC not ready yet", message);
    }
};

WebRTC Setup

This block configures the WebRTC connection, adds audio tracks, and initializes the DataChannel.

async function openRTC(data) {
    const EPHEMERAL_KEY = data.client_secret.value;

    // Set up to play remote audio
    const audioEl = document.createElement("audio");
    audioEl.autoplay = true;
    pc.ontrack = e => audioEl.srcObject = e.streams[0];

    // Add microphone input as local audio track
    const ms = await navigator.mediaDevices.getUserMedia({ audio: true });
    pc.addTrack(ms.getTracks()[0]);

    // Create a DataChannel
    dc = pc.createDataChannel("oai-events");
    dc.addEventListener("message", e => {
        const message = JSON.parse(e.data);
        if (message.type.includes("function")) {
            ws.send(e.data); // Forward function messages to WebSocket
        }
    });

    // Create and send an SDP offer
    const offer = await pc.createOffer();
    await pc.setLocalDescription(offer);

    // Send the offer to OpenAI
    const baseUrl = "https://api.openai.com/v1/realtime";
    const sdpResponse = await fetch(`${baseUrl}?model=${data.model}`, {
        method: "POST",
        body: offer.sdp,
        headers: {
            Authorization: `Bearer ${EPHEMERAL_KEY}`,
            "Content-Type": "application/sdp"
        },
    });

    // Set the remote SDP answer
    const answer = { type: "answer", sdp: await sdpResponse.text() };
    await pc.setRemoteDescription(answer);
    console.log("Connected to OpenAI WebRTC");
}

Server implementation

This server implementation uses FastAPI to set up a WebRTC and WebSockets interaction, allowing clients to communicate with a chatbot powered by OpenAI’s Realtime API. The server provides endpoints for a simple chat interface and real-time audio communication.

Create an app using FastAPI

First, initialize a FastAPI app instance to handle HTTP requests and WebSocket connections.

app = FastAPI()

This creates an app instance that will be used to manage both regular HTTP requests and real-time WebSocket interactions.

Define the root endpoint for status

Next, define a root endpoint to verify that the server is running.

@app.get("/", response_class=JSONResponse)
async def index_page():
    return {"message": "WebRTC AG2 Server is running!"}

When accessed, this endpoint responds with a simple status message indicating that the WebRTC server is up and running.

Set up static files and templates

Mount a directory for static files (e.g., CSS, JavaScript) and configure templates for rendering HTML.

website_files_path = Path(__file__).parent / "website_files"

app.mount(
    "/static", StaticFiles(directory=website_files_path / "static"), name="static"
)

templates = Jinja2Templates(directory=website_files_path / "templates")

This ensures that static assets (like styling or scripts) can be served and that HTML templates can be rendered for dynamic responses.

Serve the chat interface page

Create an endpoint to serve the HTML page for the chat interface.

@app.get("/start-chat/", response_class=HTMLResponse)
async def start_chat(request: Request):
    """Endpoint to return the HTML page for audio chat."""
    port = request.url.port
    return templates.TemplateResponse("chat.html", {"request": request, "port": port})

This endpoint serves the chat.html page and provides the port number in the template, which is used for WebSockets connections.

Handle WebSocket connections for media streaming

Set up a WebSocket endpoint to handle real-time interactions, including receiving audio streams and responding with OpenAI’s model output.

@app.websocket("/session")
async def handle_media_stream(websocket: WebSocket):
    """Handle WebSocket connections providing audio stream and OpenAI."""
    await websocket.accept()

    logger = getLogger("uvicorn.error")

    realtime_agent = RealtimeAgent(
        name="Weather Bot",
        system_message="Hello there! I am an AI voice assistant powered by Autogen and the OpenAI Realtime API. You can ask me about weather, jokes, or anything you can imagine. Start by saying 'How can I help you'?",
        llm_config=realtime_llm_config,
        websocket=websocket,
        logger=logger,
    )

This WebSocket endpoint establishes a connection and creates a RealtimeAgent that will manage interactions with OpenAI’s Realtime API. It also includes logging for monitoring the process.

Register and implement real-time functions

Define custom real-time functions that can be called from the client side, such as fetching weather data.

    @realtime_agent.register_realtime_function(
        name="get_weather", description="Get the current weather"
    )
    def get_weather(location: Annotated[str, "city"]) -> str:
        logger.info(f"Checking the weather: {location}")
        return (
            "The weather is cloudy." if location == "Rome" else "The weather is sunny."
        )

Here, a weather-related function is registered with the RealtimeAgent. It responds with a simple weather message based on the input city.

Run the RealtimeAgent

Finally, run the RealtimeAgent to start handling the WebSocket interactions.

    await realtime_agent.run()

This starts the agent’s event loop, which listens for incoming messages and responds accordingly.

Conclusion

New integration of OpenAI Realtime API with WebRTC unlocks the full potential of WebRTC for real-time voice applications. With its low latency, adaptive quality, and secure communication, it’s the perfect tool for building interactive, voice-enabled applications.

Try it today and take your voice applications to the next level!

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for RealtimeAgent? Or maybe you would like to see more features or improvements? Do join our Discord server for discussion.

Real-Time Voice Interactions with the WebSocket Audio Adapter

AG2 Blogger — Tue, 14 Jan 2025 17:07:24 +0000

Authors: Mark Sze, Tvrtko Sternak, Davor Runje, Davorin Rusevljan

TL;DR:

Demo implementation: Implement a website using websockets and communicate using voice with the RealtimeAgent
Introducing WebSocketAudioAdapter: Stream audio directly from your browser using WebSockets.
Simplified Development: Connect to real-time agents quickly and effortlessly with minimal setup.

Realtime over WebSockets

In our previous blog post, we introduced a way to interact with the RealtimeAgent using TwilioAudioAdapter. While effective, this approach required a setup-intensive process involving Twilio integration, account configuration, number forwarding, and other complexities. Today, we’re excited to introduce theWebSocketAudioAdapter, a streamlined approach to real-time audio streaming directly via a web browser.

This post explores the features, benefits, and implementation of the WebSocketAudioAdapter, showing how it transforms the way we connect with real-time agents.

Why We Built the `WebSocketAudioAdapter`

Challenges with Existing Solutions

Previously introduced TwilioAudioAdapter provides a robust way to cennect to your RealtimeAgent, it comes with challenges:

Browser Limitations: For teams building web-first applications, integrating with a telephony platform can feel redundant.
Complex Setup: Configuring Twilio accounts, verifying numbers, and setting up forwarding can be time-consuming.
Platform Dependency: This solution requires developers to rely on external API, which adds latency and costs.

Our Solution

The WebSocketAudioAdapter eliminates these challenges by allowing direct audio streaming over WebSockets. It integrates seamlessly with modern web technologies, enabling real-time voice interactions without external telephony platforms.

How It Works

At its core, the WebSocketAudioAdapter leverages WebSockets to handle real-time audio streaming. This means your browser becomes the communication bridge, sending audio packets to a server where a RealtimeAgent agent processes them.

Here’s a quick overview of its components and how they fit together:

WebSocket Connection:

* The adapter establishes a [**WebSockets**](https://fastapi.tiangolo.com/advanced/websockets/) connection between the client (browser) and the server.

* Audio packets are streamed in real time through this connection.

Integration with FastAPI:

* Using Python’s [**FastAPI**](https://fastapi.tiangolo.com/) framework, developers can easily set up endpoints for handling [**WebSockets**](https://fastapi.tiangolo.com/advanced/websockets/) traffic.

Powered by Realtime Agents:

* The audio adapter integrates with an AI-powered [`RealtimeAgent`](https://docs.ag2.ai/docs/reference/agentchat/realtime_agent/realtime_agent), allowing the agent to process audio inputs and respond intelligently.

Key Features

1. Simplified Setup

Unlike TwilioAudioAdapter, the WebSocketAudioAdapter requires no phone numbers, no telephony configuration, and no external accounts. It’s a plug-and-play solution.

2. Real-Time Performance

By streaming audio over WebSockets, the adapter ensures low latency, making conversations feel natural and seamless.

3. Browser-Based

Everything happens within the user’s browser, meaning no additional software is required. This makes it ideal for web applications.

4. Flexible Integration

Whether you’re building a chatbot, a voice assistant, or an interactive application, the adapter can integrate easily with existing frameworks and AI systems.

Example: Build a Voice-Enabled Weather Bot

Let’s walk through a practical example where we use the WebSocketAudioAdapter to create a voice-enabled weather bot. You can find the full example here.

To run the demo example, follow these steps:

1. Clone the Repository

git clone https://github.com/ag2ai/realtime-agent-over-websockets.git
cd realtime-agent-over-websockets

2. Set Up Environment Variables

Create a OAI_CONFIG_LIST file based on the provided OAI_CONFIG_LIST_sample:

cp OAI_CONFIG_LIST_sample OAI_CONFIG_LIST

In the OAI_CONFIG_LIST file, update the api_key to your OpenAI API key.

(Optional) Create and use a virtual environment

To reduce cluttering your global Python environment on your machine, you can create a virtual environment. On your command line, enter:

python3 -m venv env
source env/bin/activate

3. Install Dependencies

Install the required Python packages using pip:

pip install -r requirements.txt

4. Start the Server

Run the application with Uvicorn:

uvicorn realtime_over_websockets.main:app --port 5050

After you start the server you should see your application running in the logs:

INFO:     Started server process [64425]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5050 (Press CTRL+C to quit)

Ready to Chat? 🚀

Now you can simply open localhost:5050/start-chat in your browser, and dive into an interactive conversation with the RealtimeAgent! 🎤✨

To get started, simply speak into your microphone and ask a question. For example, you can say:

“What’s the weather like in Seattle?”

This initial question will activate the agent, and it will respond, showcasing its ability to understand and interact with you in real time.

Code review

Let’s dive in and break down how this example works—from setting up the server to handling real-time audio streaming with WebSockets.

Set Up the FastAPI app

We use FastAPI to serve the chat interface and handle WebSocket connections. A key part is configuring the server to load and render HTML templates dynamically for the user interface.

Template Loading: Use Jinja2Templates to load chat.html from the templates directory. The template is dynamically rendered with variables like the server’s port.
Static Files: Serve assets (e.g., JavaScript, CSS) from the static directory.

app = FastAPI()


@app.get("/", response_class=JSONResponse)
async def index_page() -> dict[str, str]:
    return {"message": "WebSocket Audio Stream Server is running!"}


website_files_path = Path(__file__).parent / "website_files"

app.mount(
    "/static", StaticFiles(directory=website_files_path / "static"), name="static"
)

templates = Jinja2Templates(directory=website_files_path / "templates")


@app.get("/start-chat/", response_class=HTMLResponse)
async def start_chat(request: Request) -> HTMLResponse:
    """Endpoint to return the HTML page for audio chat."""
    port = request.url.port
    return templates.TemplateResponse("chat.html", {"request": request, "port": port})

Defining the WebSocket Endpoint

The /media-stream WebSocket route is where real-time audio interaction is processed and streamed to the AI assistant. Let’s break it down step-by-step:

1. Accept the WebSocket Connection The WebSocket connection is established when a client connects to /media-stream. Using await websocket.accept(), we ensure the connection is live and ready for communication.

@app.websocket("/media-stream")
async def handle_media_stream(websocket: WebSocket) -> None:
    """Handle WebSocket connections providing audio stream and OpenAI."""
    await websocket.accept()

2. Initialize Logging A logger instance (getLogger("uvicorn.error")) is set up to monitor and debug the server’s activities, helping track events during the connection and interaction process.

    logger = getLogger("uvicorn.error")

3. Set Up the WebSocketAudioAdapter The WebSocketAudioAdapter bridges the client’s audio stream with the RealtimeAgent. It streams audio data over WebSockets in real time, ensuring seamless communication between the browser and the agent.

    audio_adapter = WebSocketAudioAdapter(websocket, logger=logger)

4. Configure the Realtime Agent The RealtimeAgent is the AI assistant driving the interaction. Key parameters include:

* **Name**: The agent identity, here called `"Weather Bot"`.

* **System Message**: System message for the agent.

* **Language Model Configuration**: Defined by `realtime_llm_config` for LLM settings.

* **Audio Adapter**: Connects the [`WebSocketAudioAdapter`](https://docs.ag2.ai/docs/reference/agentchat/realtime_agent/websocket_audio_adapter#websocketaudioadapter) for handling audio.

* **Logger**: Logs the agent’s activities for better observability.

    realtime_agent = RealtimeAgent(
        name="Weather Bot",
        system_message="Hello there! I am an AI voice assistant powered by Autogen and the OpenAI Realtime API. You can ask me about weather, jokes, or anything you can imagine. Start by saying 'How can I help you'?",
        llm_config=realtime_llm_config,
        audio_adapter=audio_adapter,
        logger=logger,
    )

5. Define a Custom Realtime Function The get_weather function is registered as a realtime callable function. When the user asks about the weather, the agent can call the function to get an accurate weather report and respond based on the provided information:

* Returns `"The weather is cloudy."` for `"Seattle"`.

* Returns `"The weather is sunny."` for other locations.

    @realtime_agent.register_realtime_function(  # type: ignore [misc]
        name="get_weather", description="Get the current weather"
    )
    def get_weather(location: Annotated[str, "city"]) -> str:
        return (
            "The weather is cloudy."
            if location == "Seattle"
            else "The weather is sunny."
        )

6. Run the Realtime Agent The await realtime_agent.run() method starts the agent, handling incoming audio streams, processing user queries, and responding in real time.

Here is the full code for the /media-stream endpoint:

@app.websocket("/media-stream")
async def handle_media_stream(websocket: WebSocket) -> None:
    """Handle WebSocket connections providing audio stream and OpenAI."""
    await websocket.accept()

    logger = getLogger("uvicorn.error")

    audio_adapter = WebSocketAudioAdapter(websocket, logger=logger)

    realtime_agent = RealtimeAgent(
        name="Weather Bot",
        system_message="Hello there! I am an AI voice assistant powered by Autogen and the OpenAI Realtime API. You can ask me about weather, jokes, or anything you can imagine. Start by saying 'How can I help you'?",
        llm_config=realtime_llm_config,
        audio_adapter=audio_adapter,
        logger=logger,
    )

    @realtime_agent.register_realtime_function(  # type: ignore [misc]
        name="get_weather", description="Get the current weather"
    )
    def get_weather(location: Annotated[str, "city"]) -> str:
        return (
            "The weather is cloudy."
            if location == "Seattle"
            else "The weather is sunny."
        )

    await realtime_agent.run()

Benefits in Action

Quick Prototyping: Spin up a real-time voice application in minutes.
Cost Efficiency: Eliminate third-party telephony costs.
User-Friendly: Runs in the browser, making it accessible to anyone with a microphone.

Conclusion

The WebSocketAudioAdapter marks a shift toward simpler, more accessible real-time audio solutions. It empowers developers to build and deploy voice applications faster and more efficiently. Whether you’re creating an AI assistant, a voice-enabled app, or an experimental project, this adapter is your go-to tool for real-time audio streaming.

Try it out and bring your voice-enabled ideas to life!

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for RealtimeAgent? Or maybe you would like to see more features or improvements? Do join our Discord server for discussion.

Tools Dependency Injection

AG2 Blogger — Mon, 13 Jan 2025 18:58:24 +0000

Author: Robert Jambrecic

Dependency Injection is a secure way to connect external functions to agents without exposing sensitive data such as passwords, tokens, or personal information. This approach ensures that sensitive information remains protected while still allowing agents to perform their tasks effectively, even when working with large language models (LLMs).

In this guide, we’ll explore how to build secure workflows that handle sensitive data safely.

As an example, we’ll create an agent that retrieves user’s account balance. The best part is that sensitive data like username and password are never shared with the LLM. Instead, it’s securely injected directly into the function at runtime, keeping it safe while maintaining seamless functionality.

Why Dependency Injection Is Essential

Here’s why dependency injection is a game-changer for secure LLM workflows:

Enhanced Security: Your sensitive data is never directly exposed to the LLM.
Simplified Development: Secure data can be seamlessly accessed by functions without requiring complex configurations.
Unmatched Flexibility: It supports safe integration of diverse workflows, allowing you to scale and adapt with ease.

In this guide, we’ll explore how to set up dependency injection and build secure workflows. Let’s dive in!

Note: This blog builds upon the concepts covered in the following notebook.

Installation

To install AG2, simply run the following command:

pip install ag2

Imports

The functionality demonstrated in this guide is located in the autogen.tools.dependency_injection module. This module provides key components for dependency injection:

BaseContext: abstract base class used to define and encapsulate data contexts, such as user account information, which can then be injected into functions or agents securely.
Depends: a function used to declare and inject dependencies, either from a context (like BaseContext) or a function, ensuring sensitive data is provided securely without direct exposure.

import os
from typing import Annotated, Literal

from pydantic import BaseModel

from autogen import GroupChat, GroupChatManager
from autogen.agentchat import ConversableAgent, UserProxyAgent
from autogen.tools.dependency_injection import BaseContext, Depends

Define a BaseContext Class

We start by defining a BaseContext class for accounts. This will act as the base structure for dependency injection. By using this approach, sensitive information like usernames and passwords is never exposed to the LLM.

class Account(BaseContext, BaseModel):
    username: str
    password: str
    currency: Literal["USD", "EUR"] = "USD"


alice_account = Account(username="alice", password="password123")
bob_account = Account(username="bob", password="password456")

account_ballace_dict = {
    (alice_account.username, alice_account.password): 300,
    (bob_account.username, bob_account.password): 200,
}

Helper Functions

To ensure that the provided account is valid and retrieve its balance, we create two helper functions.

def _verify_account(account: Account):
    if (account.username, account.password) not in account_ballace_dict:
        raise ValueError("Invalid username or password")


def _get_balance(account: Account):
    _verify_account(account)
    return f"Your balance is {account_ballace_dict[(account.username, account.password)]}{account.currency}"

Injecting BaseContext Parameter

Dependency injection simplifies passing data to a function. Here, we’ll inject an Account instance into a function automatically.

Agent Configuration

Configure the agents for the interaction.

config_list defines the LLM configurations, including the model and API key.
UserProxyAgent simulates user inputs without requiring actual human interaction (set to NEVER).
AssistantAgent represents the AI agent, configured with the LLM settings.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]
assistant = ConversableAgent(
    name="assistant",
    llm_config={"config_list": config_list},
)
user_proxy = UserProxyAgent(
    name="user_proxy_1",
    human_input_mode="NEVER",
    llm_config=False,
)

Register the Function with Dependency Injection

We register a function where the account information for bob is injected as a dependency.

Note: You can also use account: Account = Depends(bob_account) as an alternative syntax.

@user_proxy.register_for_execution()
@assistant.register_for_llm(description="Get the balance of the account")
def get_balance_1(
    # Account which will be injected to the function
    account: Annotated[Account, Depends(bob_account)],
    # It is also possible to use the following syntax to define the dependency
    # account: Account = Depends(bob_account),
) -> str:
    return _get_balance(account)

Initiate the Chat

Finally, we initiate a chat to retrieve the balance.

user_proxy.initiate_chat(assistant, message="Get users balance", max_turns=2)

user_proxy_1 (to assistant):

Get users balance

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
assistant (to user_proxy_1):

***** Suggested tool call (call_ognvIidhVCUdxvH0vnJEPxzk): get_balance_1 *****
Arguments:
{}
******************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION get_balance_1...
user_proxy_1 (to assistant):

***** Response from calling tool (call_ognvIidhVCUdxvH0vnJEPxzk) *****
Your balance is 200USD
**********************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
assistant (to user_proxy_1):

Your balance is 200 USD.

--------------------------------------------------------------------------------

Injecting Parameters Without BaseContext

Sometimes, you might not want to use BaseContext. Here’s how to inject simple parameters directly.

Agent Configuration

Configure the agents for the interaction.

config_list defines the LLM configurations, including the model and API key.
UserProxyAgent simulates user inputs without requiring actual human interaction (set to NEVER).
AssistantAgent represents the AI agent, configured with the LLM settings.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]
assistant = ConversableAgent(
    name="assistant",
    llm_config={"config_list": config_list},
)
user_proxy = UserProxyAgent(
    name="user_proxy_1",
    human_input_mode="NEVER",
    llm_config=False,
)

Register the Function with Direct Parameter Injection

Instead of injecting a full context like Account, you can directly inject individual parameters, such as the username and password, into a function. This allows for more granular control over the data injected into the function, and still ensures that sensitive information is managed securely.

Here’s how you can set it up:

def get_username() -> str:
    return "bob"

def get_password() -> str:
    return "password456"

@user_proxy.register_for_execution()
@assistant.register_for_llm(description="Get the balance of the account")
def get_balance_2(
    username: Annotated[str, Depends(get_username)],
    password: Annotated[str, Depends(get_password)],
    # or use lambdas
    # username: Annotated[str, Depends(lambda: "bob")],
    # password: Annotated[str, Depends(lambda: "password456")],
) -> str:
    account = Account(username=username, password=password)
    return _get_balance(account)

Initiate the Chat

As before, initiate a chat to test the function.

user_proxy.initiate_chat(assistant, message="Get users balance", max_turns=2)

user_proxy_1 (to assistant):

Get users balance

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
assistant (to user_proxy_1):

***** Suggested tool call (call_REyBiQkznsd2JzNr4i7Z2N7q): get_balance_2 *****
Arguments:
{}
******************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION get_balance_2...
user_proxy_1 (to assistant):

***** Response from calling tool (call_REyBiQkznsd2JzNr4i7Z2N7q) *****
Your balance is 200USD
**********************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
assistant (to user_proxy_1):

Your balance is 200 USD.

--------------------------------------------------------------------------------

Assigning Different Contexts to Multiple Agents

You can assign different contexts, such as distinct account data, to different agents within the same group chat. This ensures that each assistant works with its own unique set of data—e.g., one assistant can use alice_account, while another can use bob_account.

GroupChat Configuration

Let’s configure a GroupChat with two assistant agents.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]
llm_config = {"config_list": config_list}
assistant_1 = ConversableAgent(
    name="assistant_1",
    llm_config={"config_list": config_list},
)
assistant_2 = ConversableAgent(
    name="assistant_2",
    llm_config={"config_list": config_list},
)
user_proxy = UserProxyAgent(
    name="user_proxy_1",
    human_input_mode="NEVER",
    llm_config=False,
)

groupchat = GroupChat(agents=[user_proxy, assistant_1, assistant_2], messages=[], max_round=5)
manager = GroupChatManager(groupchat=groupchat, llm_config=llm_config)

Register Functions for Each Assistant

For assistant_1, we inject the alice_account context using Depends(alice_account), ensuring that it retrieves the balance for Alice’s account.
For assistant_2, we inject the bob_account context using Depends(bob_account), ensuring that it retrieves the balance for Bob’s account.

@user_proxy.register_for_execution()
@assistant_1.register_for_llm(description="Get the balance of the account")
def get_balance_for_assistant_1(
    account: Annotated[Account, Depends(alice_account)],
) -> str:
    return _get_balance(account)


@user_proxy.register_for_execution()
@assistant_2.register_for_llm(description="Get the balance of the account")
def get_balance_for_assistant_2(
    account: Annotated[Account, Depends(bob_account)],
) -> str:
    return _get_balance(account)

Initiate the Chat

Finally, initiate the group chat where both assistants respond by using their respective contexts. Each assistant will handle its own task — one will retrieve Alice’s balance, and the other will retrieve Bob’s balance

message = "Both assistants, please get the balance of the account"
user_proxy.initiate_chat(manager, message=message, max_turns=1)

user_proxy_1 (to chat_manager):

Both assistants, please get the balance of the account

--------------------------------------------------------------------------------

Next speaker: assistant_1


>>>>>>>> USING AUTO REPLY...
assistant_1 (to chat_manager):

***** Suggested tool call (call_wfTGOY4O9mEDBuIOrajJDSNj): get_balance_for_assistant_1 *****
Arguments:
{}
********************************************************************************************

--------------------------------------------------------------------------------

Next speaker: user_proxy_1


>>>>>>>> EXECUTING FUNCTION get_balance_for_assistant_1...
user_proxy_1 (to chat_manager):

***** Response from calling tool (call_wfTGOY4O9mEDBuIOrajJDSNj) *****
Your balance is 300USD
**********************************************************************

--------------------------------------------------------------------------------

Next speaker: assistant_2


>>>>>>>> USING AUTO REPLY...
assistant_2 (to chat_manager):

***** Suggested tool call (call_QNO5v9vGRUfRsmUAjL9yV318): get_balance_for_assistant_2 *****
Arguments:
{}
********************************************************************************************

--------------------------------------------------------------------------------

Next speaker: user_proxy_1


>>>>>>>> EXECUTING FUNCTION get_balance_for_assistant_2...
user_proxy_1 (to chat_manager):

***** Response from calling tool (call_QNO5v9vGRUfRsmUAjL9yV318) *****
Your balance is 200USD
**********************************************************************

--------------------------------------------------------------------------------

Conclusion

In this blog post, we explore Dependency Injection (DI) as a secure and effective way to manage sensitive data in workflows involving LLMs. Dependency Injection ensures that sensitive data, such as passwords or personal information, remains protected by injecting necessary details at runtime instead of exposing them directly to the LLM.

The post provides a comprehensive guide on setting up and using DI with agents, illustrating how to securely retrieve account balances without sharing sensitive data. It includes step-by-step instructions on configuring agents, defining contexts, and using the Depends function to inject account details directly into the functions. Various methods are demonstrated, such as injecting contexts, passing simple parameters, and even managing multiple contexts for different agents in group chats.

By following this guide, developers can create secure and flexible workflows that prevent unauthorized access to sensitive data while leveraging LLMs’ full potential.

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know what you think of our tools with Dependency injection! Or if you’d maybe like to see more features or improvements. And do join our Discord server for discussion!

ReasoningAgent Update - Beam Search, MCTS, and LATS for LLM Reasoning

AG2 Blogger — Tue, 07 Jan 2025 18:49:21 +0000

Authors BabyCNM, Hrushikesh Dokala, Chi Wang, Qinyung Wu

Key Updates in this Release:

1. Configuration Changes

All reasoning parameters are now configured through a single reason_config dictionary
Breaking Change: Parameters like max_depth, beam_size, and answer_approach have moved from constructor arguments into reason_config

2. New Search Strategies

Added Monte Carlo Tree Search (MCTS) as an alternative to Beam Search
Introduced Language Agent Tree Search (LATS) - an enhancement to MCTS that incorporates reflection prior to the next round of simulation.

3. Enhanced Features

New forest_size parameter enables maintaining multiple independent reasoning trees
Support for ground truth answers in prompts to generate training data for LLM fine-tuning

Introduction

In our previous post, we introduced the ReasoningAgent, which utilized Beam Search for systematic reasoning. Today, we include MCTS (Monte Carlo Tree Search) and Language Agent Tree Search (LATS) as alternative search strategies, which present advantages in different scenarios.

Our previous ReasoningAgent draws inspiration from OpenAI’s 2023 paper, Let’s Verify Step by Step, as well as the 2024 O1 feature. The landscape of contemporary research is rich, with notable works such as DeepSeek-R1, Macro-O1, and OpenR.

Quick Start Guide

Let’s start with a simple example using MCTS:

import os
from autogen import UserProxyAgent, ReasoningAgent

# Configure the model
config_list = [{"model": "gpt-4o-mini", "api_key": os.environ.get("OPENAI_API_KEY")}]

# Create a reasoning agent with MCTS
mcts_agent = ReasoningAgent(
    name="mcts_agent",
    llm_config={"config_list": config_list},
    reason_config={
        "method": "mcts",  # Use MCTS instead of beam search
        "nsim": 5,  # Number of MCTS simulations
        "exploration_constant": 1.41  # UCT exploration parameter
    }
)

# Create a user proxy agent
user_proxy = UserProxyAgent(
    name="user_proxy",
    human_input_mode="NEVER",
    code_execution_config={"use_docker": False}
)

prompt = "What is the expected maximum dice value if you can roll a 6-sided dice three times?"
response = user_proxy.initiate_chat(mcts_agent, message=prompt)

Configuring a Separate Grader Model

In addition to the main reasoning model, you can now specify a different model for the grader by using the grader_llm_config parameter. This allows for more flexibility in evaluating the reasoning paths generated by the agent. If this parameter is not provided, the grader will use the same model as the reasoning agent. Here’s how you can set it up:

# Configure the model
config_list = [{"model": "gpt-4o-mini", "api_key": os.environ.get("OPENAI_API_KEY")}]
config_list_larger = [{"model": "gpt-4o", "api_key": os.environ.get("OPENAI_API_KEY")}]

# Create a reasoning agent with MCTS
mcts_agent = ReasoningAgent(
    name="mcts_agent",
    llm_config={"config_list": mini_config_list},
    grader_llm_config={"config_list": config_list_larger},
    reason_config={
        "method": "mcts",
        "nsim": 5
    }
)

Key Features in the New Version

1. Multiple Search Methods

ReasoningAgent now supports three search strategies:

As the previous blog, the default method is beam search.

# Beam Search (default)
beam_agent = ReasoningAgent(
    name="beam_agent",
    llm_config={"config_list": config_list},
    reason_config={
        "method": "beam_search",
        "beam_size": 3,
        "answer_approach": "pool"  # or "best"
    }
)

MCTS is also included as a common approach.

# Monte Carlo Tree Search
mcts_agent = ReasoningAgent(
    name="mcts_agent",
    llm_config={"config_list": config_list},
    reason_config={
        "method": "mcts",
        "nsim": 5 # number of simulations
    }
)

It is important to note that our reasoning agent operates based on “process” and lacks direct access to the environment. In contrast, the LATS approach relies on feedback from the environment. To address this, we utilize our existing grader agent to generate pseudo-rewards and provide feedback. The major difference between our LATS implementation and our MCTS implementation is that the LATS approach incorporate the reflection into prompt context before next round of simulation. You can define the agent using the LATS approach as follows.

# Language Agent Tree Search
lats_agent = ReasoningAgent(
    name="lats_agent",
    llm_config={"config_list": config_list},
    reason_config={
        "method": "lats",
        "nsim": 5
    }
)

2. Incorporating Ground Truth for Enhanced Training Data Synthesis

You can now include ground truth in your prompts to achieve more precise evaluations (grading). This allows you to leverage the reasoning agent to generate diverse thinking trajectories, further finetuning the base LLM.

prompt = """Solve this calculus problem: ∫x²dx

GROUND_TRUTH:
The integral of x² is (x³/3) + C
Steps:
1. Use power rule: increase power by 1
2. Divide by new power
3. Add constant of integration
"""

response = user_proxy.initiate_chat(mcts_agent, message=prompt)

# After running queries...
sft_data = extract_sft_dataset(mcts_agent._root)
rlhf_data = extract_rlhf_preference_dataset(mcts_agent._root)

3. Forest of Trees

Enable ensemble reasoning with multiple independent trees:

forest_agent = ReasoningAgent(
    name="forest_agent",
    llm_config={"config_list": config_list},
    reason_config={
        "method": "mcts",
        "forest_size": 5  # Run 5 independent trees
    }
)

When to Use Each Method

Use Beam Search when:

You want a deterministic search process
You can reliably evaluate intermediate steps
You need fast, memory-efficient search
The solution space is relatively small and structured
Early decisions strongly influence final outcomes

Use MCTS when:

You need stochastic exploration of solution paths
Final outcome evaluation is more reliable than intermediate steps
The solution space is large or complex
You want to balance exploration vs exploitation
You have computational budget for multiple simulations

Use LATS when:

Provides immediate reflection feedback before the next simulation
Helps identify poor reasoning paths early for future improvement
Especially useful for complex multi-step reasoning

Advanced Features

1. Visualization

Visualize the reasoning tree using graphviz:

from autogen.agentchat.contrib.reasoning_agent import visualize_tree

# After running queries...
visualize_tree(mcts_agent._root)

2. Custom Evaluation

Modify the rating scale and evaluation criteria:

custom_agent = ReasoningAgent(
    name="custom_agent",
    llm_config={"config_list": config_list},
    reason_config={
        "rating_scale": 100,  # Use 1-100 scale instead of default 1-10 for grading
    }
)

3. Save and Load Trees

Save reasoning trees for later analysis:

import json

# Save tree
data = mcts_agent._root.to_dict()
with open("reasoning_tree.json", "w") as f:
    json.dump(data, f)

# Load tree
from autogen.agentchat.contrib.reasoning_agent import ThinkNode
loaded_tree = ThinkNode.from_dict(json.load(open("reasoning_tree.json")))

Performance Comparison

Variables

d: Maximum depth of the reasoning tree
b: Beam size (number of parallel paths maintained)
w: Branching factor (number of child nodes per parent)
n: Number of MCTS simulations

Time Complexity

Each algorithm has different computational costs:

Beam Search: O(d × b × (w + 1))
- At each depth level d, evaluates w options for each of b beams
- Plus 1 for generating the options
MCTS and LATS: O(n × d)
- Each simulation traverses down to depth d
- Performs n total simulations

Memory Usage

Storage requirements vary by approach:

Beam Search: O(b × d)
- Fixed memory proportional to beam size and depth
- Only stores active beams
MCTS and LATS: O(w^d)
- Worst case stores complete tree
- In practice much smaller due to selective expansion

Conclusion

The new ReasoningAgent offers a flexible toolkit for systematic reasoning with LLMs. Choose between Beam Search, MCTS, and LATS based on your specific needs regarding:

Evaluation cost and availability
Time and resource constraints
Desired exploration vs exploitation balance
Training data generation requirements

Next Steps

Async Client Call: parallelize LLM calling to speed up searching
Swarm Agent implementation
Efficient Mode: merging thinker and grader
Batch Norm: normalizing scores for MCTS

For Further Reading

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for ReasoningAgent? Or maybe you would like to see more features or improvements? Do join our Discord server for discussion.

Cross-Framework LLM Tool Integration with AG2

AG2 Blogger — Mon, 30 Dec 2024 15:58:00 +0000

Author: Robert Jambrecic

TL;DR

AG2 lets you bring in Tools from different frameworks like LangChain, CrewAI, and PydanticAI.

LangChain Tools: Useful for tasks like API querying and web scraping.
CrewAI Tools: Offers a variety of tools for web scraping, search, and more.
PydanticAI Tools: Adds context-driven tools and structured data processing.

With AG2, you can combine these tools and enhance your agents’ capabilities.

In this post, we’ll walk through how to integrate tools from various frameworks—like LangChain Tools, CrewAI Tools, and PydanticAI Tools—into AG2.

Because, really, the magic happens when you combine them all. This allows you to use tools from different frameworks within AG2, giving your agents more power and flexibility. This blog builds upon the concepts covered in the Tool Integration notebook.

In this post, you will understand how to configure agents, adapt these tools for use in AG2, and validate the integration through practical examples.

LangChain Tools Integration

LangChain is a popular framework with lots of tools for working with LLMs. It’s got a range of tools that can be easily integrated into AG2. If you want to see the full list, check out the LangChain Community Tools. You can quickly add things like API queries, web scraping, and text generation to your AG2 setup.

Installation

To get LangChain tools working with AG2, you’ll need to install a couple of dependencies:

pip install ag2[interop-langchain]

Also, we’ll use LangChain’s Wikipedia Tool, which needs the wikipedia package. Install it like this:

pip install wikipedia

Imports

Now, let’s import the necessary modules and tools.

WikipediaQueryRun and WikipediaAPIWrapper are the tools for querying Wikipedia.
AssistantAgent and UserProxyAgent are the agents for interaction within AG2.
Interoperability is what helps connect LangChain tools with AG2.

import os

from langchain_community.tools import WikipediaQueryRun
from langchain_community.utilities import WikipediaAPIWrapper

from autogen import AssistantAgent, UserProxyAgent
from autogen.interop import Interoperability

Agent Configuration

Let’s set up the agents for interaction.

config_list is where you define the LLM configuration, like the model and API key.
UserProxyAgent simulates user inputs without requiring actual human interaction (set to NEVER).
AssistantAgent represents the AI agent, configured with the LLM settings.

config_list = [{"model": "gpt-4o", "api_key": os.environ["OPENAI_API_KEY"]}]
user_proxy = UserProxyAgent(
    name="User",
    human_input_mode="NEVER",
)

chatbot = AssistantAgent(
    name="chatbot",
    llm_config={"config_list": config_list},
)

Tool Integration

Here’s where we connect everything.

First, we set up WikipediaAPIWrapper, which fetches the top Wikipedia result (with a character limit).
Then, we use WikipediaQueryRun to perform Wikipedia queries.
Interoperability helps convert the LangChain tool to AG2’s format.
Finally, we register the tool for use with both the user_proxy and chatbot.

api_wrapper = WikipediaAPIWrapper(top_k_results=1, doc_content_chars_max=1000)
langchain_tool = WikipediaQueryRun(api_wrapper=api_wrapper)

interop = Interoperability()
ag2_tool = interop.convert_tool(tool=langchain_tool, type="langchain")

ag2_tool.register_for_execution(user_proxy)
ag2_tool.register_for_llm(chatbot)

Initiating the Chat

Once everything’s set up, we can send a message to the chatbot, and it’ll use the Wikipedia tool to fetch the relevant information.

message = "Tell me about the history of the United States"
user_proxy.initiate_chat(recipient=chatbot, message=message, max_turns=2)

Output

When the chat is initiated, here’s the output you’ll see:

User (to chatbot):

Tell me about the history of the United States

--------------------------------------------------------------------------------
chatbot (to User):

***** Suggested tool call (call_hhy2G43ymytUFmJlDsK9J0tk): wikipedia *****
Arguments:
{"tool_input":{"query":"history of the United States"}}
**************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION wikipedia...
User (to chatbot):

***** Response from calling tool (call_hhy2G43ymytUFmJlDsK9J0tk) *****
Page: History of the United States
Summary: The history of the lands that became the United States began with the arrival of the first people in the Americas around 15,000 BC. After European colonization of North America began in the late 15th century, wars and epidemics decimated Indigenous societies. By the 1760s, the thirteen British colonies were established. The Southern Colonies built an agricultural system on slave labor, enslaving millions from Africa. After defeating France, the British Parliament imposed a series of taxes; resistance to these taxes, especially the Boston Tea Party in 1773, led to Parliament issuing the Intolerable Acts designed to end self-government.
In 1776, the United States declared its independence. Led by General George Washington, it won the Revolutionary War in 1783. The Constitution was adopted in 1789, and a Bill of Rights was added in 1791 to guarantee inalienable rights. Washington, the first president, and his adviser Alexander Hamilton created a
**********************************************************************

--------------------------------------------------------------------------------
chatbot (to User):

The history of the United States begins with the arrival of the first peoples in the Americas around 15,000 BC. This pre-Columbian era was followed by European colonization, beginning in the late 15th century, which dramatically altered the indigenous societies through wars and epidemics.

By the 1760s, thirteen British colonies were established along the Atlantic seaboard. In the Southern Colonies, an agricultural economy heavily reliant on enslaved labor from Africa was developed. The British victory over France in the Seven Years' War led Parliament to impose various taxes on the colonies. Resistance to these taxes, exemplified by the Boston Tea Party in 1773, prompted the Parliament to enact the Intolerable Acts, seeking to curtail colonial self-governance.

The United States declared independence in 1776. Under the leadership of General George Washington, the American Revolutionary War concluded successfully in 1783. Subsequently, the U.S. Constitution was adopted in 1789, with the Bill of Rights added in 1791 to ensure inalienable rights. During this early period, President George Washington and his advisor Alexander Hamilton played significant roles in forming the young nation's governmental and economic foundations.

This overview covers the early formation and foundational moments of what became the United States, setting the stage for the country's subsequent expansion and development. TERMINATE

--------------------------------------------------------------------------------

CrewAI Tools Integration

CrewAI provides a variety of powerful tools designed for tasks such as web scraping, search, code interpretation, and more. These tools are easy to integrate into the AG2 framework, allowing you to enhance your agents with advanced capabilities. You can explore the full list of available tools in the CrewAI Tools repository.

Installation

Install the required packages for integrating CrewAI tools into the AG2 framework. This ensures all dependencies for both frameworks are installed.

pip install ag2[interop-crewai]

Imports

Import necessary modules and tools.

ScrapeWebsiteTool: A CrewAI tool for web scraping.
AssistantAgent and UserProxyAgent: Core AG2 classes.
Interoperability: This module acts as a bridge, making it easier to integrate CrewAI tools with AG2’s architecture.

import os

from crewai_tools import ScrapeWebsiteTool

from autogen import AssistantAgent, UserProxyAgent
from autogen.interop import Interoperability

Agent Configuration

Configure the agents for the interaction.

config_list defines the LLM configurations, including the model and API key.
UserProxyAgent simulates user inputs without requiring actual human interaction (set to NEVER).
AssistantAgent represents the AI agent, configured with the LLM settings.

config_list = [{"model": "gpt-4o", "api_key": os.environ["OPENAI_API_KEY"]}]
user_proxy = UserProxyAgent(
    name="User",
    human_input_mode="NEVER",
)

chatbot = AssistantAgent(
    name="chatbot",
    llm_config={"config_list": config_list},
)

Tool Integration

Integrate the CrewAI tool with AG2.

Interoperability converts the CrewAI tool to a format compatible with AG2.
ScrapeWebsiteTool is used for web scraping tasks.
Register the tool for both execution and interaction with LLMs.

interop = Interoperability()
crewai_tool = ScrapeWebsiteTool()
ag2_tool = interop.convert_tool(tool=crewai_tool, type="crewai")

ag2_tool.register_for_execution(user_proxy)
ag2_tool.register_for_llm(chatbot)

Initiating the chat

Initiate the conversation between the UserProxyAgent and the AssistantAgent to utilize the CrewAI tool.

message = "Scrape the website https://ag2.ai/"
chat_result = user_proxy.initiate_chat(recipient=chatbot, message=message, max_turns=2)

Output

The chatbot provides results based on the web scraping operation:

User (to chatbot):

Scrape the website https://ag2.ai/

--------------------------------------------------------------------------------
chatbot (to User):

***** Suggested tool call (call_ZStuwmexfN7j56uJKOi6BCid): Read_website_content *****
Arguments:
{"args":{"website_url":"https://ag2.ai/"}}
*************************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION Read_website_content...
Using Tool: Read website content
User (to chatbot):

***** Response from calling tool (call_ZStuwmexfN7j56uJKOi6BCid) *****

AgentOS
Join our growing community of over 20,000 agent builders Join our growing community of over 20,000 agent builders The Open-Source AgentOS Build production-ready multi-agent systems in minutes, not months. Github Discord The End-to-End Platform for Multi-Agent Automation The End-to-End Platform for Multi-Agent Automation Flexible Agent Construction and Orchestration Create specialized agents that work together seamlessly. AG2 makes it easy to define roles, configure behaviors, and orchestrate collaboration - all through simple, intuitive code. → Assistant agents for problem-solving → Executor agents for taking action → Critic agents for validation → Group chat managers for coordination Built-in Conversation Patterns Built-in Conversation Patterns Stop wrestling with agent coordination. AG2 handles message routing, state management, and conversation flow automatically. → Two-agent conversations → Group chats with dynamic speaker selection → Sequential chats with context carryover → Nested conversations for modularity Seamless Human-AI collaboration Seamless Human-AI collaboration Seamlessly integrate human oversight and input into your agent workflows. → Configurable human input modes → Flexible intervention points → Optional human approval workflows → Interactive conversation interfaces → Context-aware human handoff Roadmap AG2 STUDIO → Visual agent system design → Real-time testing and debugging → One-click deployment to production → Perfect for prototyping and MVPs AG2 STUDIO → Visual agent system design → Real-time testing and debugging → One-click deployment to production → Perfect for prototyping and MVPs AG2 STUDIO → Visual agent system design → Real-time testing and debugging → One-click deployment to production → Perfect for prototyping and MVPs AG2 MARKETPLACE → Share and monetize your agents → Discover pre-built solution templates → Quick-start your agent development → Connect with other builders AG2 MARKETPLACE → Share and monetize your agents → Discover pre-built solution templates → Quick-start your agent development → Connect with other builders AG2 MARKETPLACE → Share and monetize your agents → Discover pre-built solution templates → Quick-start your agent development → Connect with other builders SCALING TOOLS → Zero to production deployment guides → Usage analytics and cost optimization → Team collaboration features → Enterprise-ready security controls SCALING TOOLS → Zero to production deployment guides → Usage analytics and cost optimization → Team collaboration features → Enterprise-ready security controls SCALING TOOLS → Zero to production deployment guides → Usage analytics and cost optimization → Team collaboration features → Enterprise-ready security controls AG2 STUDIO → Visual agent system design → Real-time testing and debugging → One-click deployment to production → Perfect for prototyping and MVPs AG2 STUDIO → Visual agent system design → Real-time testing and debugging → One-click deployment to production → Perfect for prototyping and MVPs AG2 MARKETPLACE → Share and monetize your agents → Discover pre-built solution templates → Quick-start your agent development → Connect with other builders AG2 MARKETPLACE → Share and monetize your agents → Discover pre-built solution templates → Quick-start your agent development → Connect with other builders SCALING TOOLS → Zero to production deployment guides → Usage analytics and cost optimization → Team collaboration features → Enterprise-ready security controls SCALING TOOLS → Zero to production deployment guides → Usage analytics and cost optimization → Team collaboration features → Enterprise-ready security controls Whether you're a solo founder prototyping the next big AI product, or an enterprise team deploying at scale we're building AG2 for you. This is AgentOS - making multi-agent development accessible to everyone. Github Join Our Growing Community Join Our Growing Community → 20,000+ active agent builders → Daily technical discussions → Weekly community calls → Open RFC process → Regular contributor events (Coming soon) Discord Problem Features Roadmap Community Documentation Problem Features Roadmap Community Documentation Problem Features Roadmap Community Documentation

**********************************************************************

--------------------------------------------------------------------------------
chatbot (to User):

The website "https://ag2.ai/" promotes a platform named AgentOS, which is designed for building multi-agent systems efficiently. Key highlights from the website are:

- **Community**: They have a growing community of over 20,000 agent builders.

- **End-to-End Platform**: AG2 is described as an end-to-end platform for multi-agent automation. It supports flexible agent construction and orchestration, helping to define roles, configure behaviors, and orchestrate collaboration.

- **Agent Types**: It includes assistant agents for problem-solving, executor agents for taking action, critic agents for validation, and group chat managers for coordination.

- **Built-in Conversation Patterns**: AG2 offers capabilities for message routing, state management, and conversation flow management, supporting various conversation types like two-agent conversations, group chats, and nested conversations.

- **Human-AI Collaboration**: The platform facilitates seamless integration of human oversight and input, with options for human intervention and approval workflows.

- **AG2 Studio**: This feature provides visual agent system design, real-time testing, debugging, and one-click deployment, suited for prototyping and MVPs.

- **AG2 Marketplace**: Provides a place to share, monetize agents, discover pre-built solution templates, and connect with other builders.

- **Scaling Tools**: Includes guides for deployment, analytics, cost optimization, team collaboration features, and enterprise-ready security controls.

- **Community and Documentation**: They encourage connecting through GitHub and Discord and have regular community calls and events planned.

This comprehensive platform seems to aim at both individual developers and enterprise teams looking to deploy multi-agent systems effectively and collaboratively. TERMINATE

--------------------------------------------------------------------------------

You can also access a detailed summary of the interaction:

print(chat_result.summary)

The website "https://ag2.ai/" promotes a platform named AgentOS, which is designed for building multi-agent systems efficiently. Key highlights from the website are:

- **Community**: They have a growing community of over 20,000 agent builders.

- **End-to-End Platform**: AG2 is described as an end-to-end platform for multi-agent automation. It supports flexible agent construction and orchestration, helping to define roles, configure behaviors, and orchestrate collaboration.

- **Agent Types**: It includes assistant agents for problem-solving, executor agents for taking action, critic agents for validation, and group chat managers for coordination.

- **Built-in Conversation Patterns**: AG2 offers capabilities for message routing, state management, and conversation flow management, supporting various conversation types like two-agent conversations, group chats, and nested conversations.

- **Human-AI Collaboration**: The platform facilitates seamless integration of human oversight and input, with options for human intervention and approval workflows.

- **AG2 Studio**: This feature provides visual agent system design, real-time testing, debugging, and one-click deployment, suited for prototyping and MVPs.

- **AG2 Marketplace**: Provides a place to share, monetize agents, discover pre-built solution templates, and connect with other builders.

- **Scaling Tools**: Includes guides for deployment, analytics, cost optimization, team collaboration features, and enterprise-ready security controls.

- **Community and Documentation**: They encourage connecting through GitHub and Discord and have regular community calls and events planned.

This comprehensive platform seems to aim at both individual developers and enterprise teams looking to deploy multi-agent systems effectively and collaboratively.

PydanticAI Tools Integration

PydanticAI is a newer framework that brings powerful features for working with LLMs. Although it doesn’t yet have a collection of pre-built tools like other frameworks, it offers useful capabilities such as dependency injection. This feature allows you to inject a “Context” into tools, which can help pass parameters or manage state without relying on LLMs. Though it’s still evolving, you can easily integrate PydanticAI tools into AG2 to boost agent capabilities, particularly for tasks that involve structured data and context-driven logic.

Installation

To get PydanticAI tools working with AG2, install the necessary dependencies:

pip install ag2[interop-pydantic-ai]

Imports

Import necessary modules and tools.

BaseModel: Used to define data structures for tool inputs and outputs.
RunContext: Provides context during the execution of tools.
PydanticAITool: Represents a tool in the PydanticAI framework.
AssistantAgent and UserProxyAgent: Agents that facilitate communication in the AG2 framework.
Interoperability: This module acts as a bridge, making it easier to integrate PydanticAI tools with AG2’s architecture.

import os
from typing import Optional

from pydantic import BaseModel
from pydantic_ai import RunContext
from pydantic_ai.tools import Tool as PydanticAITool

from autogen import AssistantAgent, UserProxyAgent
from autogen.interop import Interoperability

Agent Configuration

Configure the agents for the interaction.

config_list defines the LLM configurations, including the model and API key.
UserProxyAgent simulates user inputs without requiring actual human interaction (set to NEVER).
AssistantAgent represents the AI agent, configured with the LLM settings.

config_list = [{"model": "gpt-4o", "api_key": os.environ["OPENAI_API_KEY"]}]
user_proxy = UserProxyAgent(
    name="User",
    human_input_mode="NEVER",
)

chatbot = AssistantAgent(
    name="chatbot",
    llm_config={"config_list": config_list},
)

Tool Integration

To integrate a PydanticAI tool into AG2:

First, define a Player model using BaseModel to structure the input data.
Use RunContext to inject dependencies (like the Player instance) securely into the tool.
The get_player function defines the tool’s functionality, retrieving injected data through ctx.deps.
Then, convert the tool into an AG2-compatible format with Interoperability.
Register the tool for execution and interaction with both the user_proxy and chatbot.

class Player(BaseModel):
    name: str
    age: int


def get_player(ctx: RunContext[Player], additional_info: Optional[str] = None) -> str:  # type: ignore[valid-type]
    """Get the player's name.

    Args:
        additional_info: Additional information which can be used.
    """
    return f"Name: {ctx.deps.name}, Age: {ctx.deps.age}, Additional info: {additional_info}"  # type: ignore[attr-defined]


interop = Interoperability()
pydantic_ai_tool = PydanticAITool(get_player, takes_ctx=True)

# player will be injected as a dependency
player = Player(name="Luka", age=25)
ag2_tool = interop.convert_tool(tool=pydantic_ai_tool, type="pydanticai", deps=player)

ag2_tool.register_for_execution(user_proxy)
ag2_tool.register_for_llm(chatbot)

Initiating the chat

Now that everything is set up, you can initiate a chat between the UserProxyAgent and the AssistantAgent:

The user_proxy sends a message to the chatbot.
The user requests player information, and includes “goal keeper” as additional context.
The Player data is securely injected into the tool, and the chatbot can access and use it during the chat.

user_proxy.initiate_chat(
    recipient=chatbot, message="Get player, for additional information use 'goal keeper'", max_turns=3
)

Output

User (to chatbot):

Get player, for additional information use 'goal keeper'

--------------------------------------------------------------------------------
chatbot (to User):

***** Suggested tool call (call_lPXIohFiJfnjmgwDnNFPQCzc): get_player *****
Arguments:
{"additional_info":"goal keeper"}
***************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION get_player...
User (to chatbot):

***** Response from calling tool (call_lPXIohFiJfnjmgwDnNFPQCzc) *****
Name: Luka, Age: 25, Additional info: goal keeper
**********************************************************************

--------------------------------------------------------------------------------
chatbot (to User):

The player's name is Luka, who is a 25-year-old goalkeeper. TERMINATE

Summary

In this post, we’ve explored how to integrate tools from multiple frameworks (LangChain, CrewAI, and PydanticAI) into the AG2 framework, enabling cross-framework interoperability. By integrating these tools, you can enhance your agents with a variety of capabilities, such as API querying, web scraping, and structured data processing.

LangChain offers a wide range of pre-built tools for working with APIs and web scraping, making it easy to extend AG2’s functionality.
CrewAI brings diverse tools for search, web scraping, and more, allowing for robust agent interactions.
PydanticAI introduces dependency injection and context-driven logic, enabling efficient data handling without relying on LLMs.

With AG2’s flexible architecture and the power of these frameworks, developers can create agents that are more capable and adaptable. By following the integration steps for each framework, you can enhance your agents’ performance, expand their capabilities, and create more dynamic interactions.

Now you should have a better understanding of how to integrate tools from different frameworks into AG2, and how to use these tools effectively within your own projects.

Finding this useful?

The AG2 team is working hard to create content like this, not to mention building a powerful, open-source, end-to-end platform for multi-agent automation.

The easiest way to show your support is just to star AG2 repo, but also take a look at it for contributions or simply to give it a try.

Also, let us know if you have any interesting use cases for tool interoperability? Or maybe you would like to see more features or improvements? Do join our Discord server for discussion.

DEV Community: AG2

Get Communicating with Discord, Slack, and Telegram

Welcome DiscordAgent, SlackAgent, and TelegramAgent

​New agents need new tools

​Capabilities

​Configuration

​Agent vs Tools

​Installation

​Demo

​Feedback

What’s next?

Finding this useful?

DeepResearchAgent - Your Shortcut for Faster Research

Introduction

​Installation

​Imports

​Starting the Conversation

​Conclusion

Finding this useful?

Adding Browsing Capabilities to AG2

Introduction

​Browser Use Integration

​Installation

​Imports

​Agent Configuration

​Web Browsing with Browser Use

​Initiate Chat

​Crawl4AI Integration

​Installation

​Imports

​Agent Configuration

​Web Browsing with Crawl4AI

​Initiate Chat

​Conclusion

Finding this useful?

Riding the Web with WebSurferAgent

Introduction

​WebSurferAgent with BrowserUseTool

​Installation

​Imports

​Configure WebSurferAgent with Browser Use

​Recommended: Using the run Method

​Manual Setup: Using initiate_chat Method

​Output

​WebSurferAgent with Crawl4AITool

​Installation

​Imports

​Configure WebSurferAgent with Crawl4AI

​Recommended: Using the run Method

​Manual Setup: Using initiate_chat Method

​Output

​Conclusion

Finding this useful?

RealtimeAgent with Gemini API

Why is this important?

Key Feature of Gemini Integration

Considerations

Future Potential

Finding this useful?

Tools with ChatContext Dependency Injection

Introduction

​Installation

​Imports

​Account and Helper Functions

​Define Account Class

​Helper Functions

​Agent Configuration

​Injecting a ChatContext Parameter

​Conclusion

Finding this useful?

Streaming input and output using WebSockets

​Real-Time Applications: Why WebSockets?

​Key Benefits of WebSockets

​Building a Chat System

​How It Works

​Example: Creating a Weather chat app

​1. Clone the Repository

​2. Set Up Environment Variables

​(Optional) Create and use a virtual environment

​3. Install Dependencies

New agents need new tools

Capabilities

Configuration

Agent vs Tools

Installation

Demo

Feedback

Installation

Imports

Starting the Conversation

Conclusion

`Browser Use` Integration

Installation

Imports

Agent Configuration

Web Browsing with Browser Use

Initiate Chat

`Crawl4AI` Integration

Installation

Imports

Agent Configuration

Web Browsing with `Crawl4AI`

Initiate Chat

Conclusion

WebSurferAgent with BrowserUseTool

Installation

Imports

Configure WebSurferAgent with Browser Use

Recommended: Using the `run` Method

Manual Setup: Using `initiate_chat` Method

Output

WebSurferAgent with Crawl4AITool

Installation

Imports

Configure WebSurferAgent with Crawl4AI

Recommended: Using the `run` Method

Manual Setup: Using `initiate_chat` Method

Output

Conclusion

Installation

Imports

Account and Helper Functions

Define Account Class

Helper Functions

Agent Configuration

Injecting a ChatContext Parameter

Conclusion

Real-Time Applications: Why WebSockets?

Key Benefits of WebSockets

Building a Chat System

How It Works

Example: Creating a Weather chat app

1. Clone the Repository

2. Set Up Environment Variables

(Optional) Create and use a virtual environment

3. Install Dependencies

4. Start the Server

Test the App

Code review

Backend Code: `main.py`

Setting Up the WebSocket Server

Running the Servers

Frontend Code: `chat.html`

JavaScript Logic

Conclusion

Realtime Voice Applications with WebRTC

Why WebRTC?

1. Low Latency

2. Adaptive Quality

3. Secure by Design

4. Widely Supported

How It Works

Example: Build a Voice-Enabled Language Translator

1. Clone the Repository

2. Set Up Environment Variables

(Optional) Create and Use a Virtual Environment

3. Install Dependencies

4. Start the Server

5. Open the Application

6. Start Speaking