LLM Agents: Introduction to Autogen

In my ongoing quest to design a question-answer system, agents are the final available design. An LLM agent is an instance of an LLM with a specifically crafted prompt so that it incorporates a defined behavior and mode of talking. A further enhancement are tools, essentially functions that provide access to additional source of information’s or enable the application and execution of programming code.

Autogen is an especially compelling framework. This project enables the definition and interaction of multi-agent chats, with or without involving a user. And it provides abstractions to access vector databases for RAG, defining custom Python functions that represent additional tools, and it can even execute generated Python code automatically.

This article shows how to create a custom agent and start an interactive, history-aware chat with only a few lines of code.

The technical context of this article is Python v3.11 and autogen v0.2.27. All code examples should work with newer library versions too, but may require code updates when the API of the library changes.

This article originally appeared at my blog admantium.com.

Required Libraries

The autogen library is a strict requirement. Install it via pip:

pip install autogen==0.2.27

To invoke an LLM, autogen relies on an OpenAI API compliant service. This means you can use the OpenAI APP directly, or use any other compatible solution, including local LLMs like ollama or vllm. See the compatible LLM documentation for further information.

Since ollama is my tool of choice for local LLMs, this article shows how to combine it with Autogen too.

Development Steps

To define and use an autogen agent, only four steps are necessarily:

• Define the LLM configuration
• Define the agents and its system prompt
• Add a user proxy
• Initiate the chat

Each step is detailed in the following sections.

Step 1: LLM Configuration

The config_list variable defines a list of LLMs that autogen uses to create agents. You can either define a global list with several entries, and instruct which LLM to choose for which agent, or define several lists with just one entry that is then passed to each specific agent.

Essentially you need to define only this:

config_list = [
  {
    "model": "llama3:instruct",
    "base_url": "http://localhost:11434/v1",
    "api_key": "ollama",
  }
]

Autogen is in active development. If you have trouble to get an ollama model to be loaded, use one of the following hints:

• Set the environment variable OPENAI_API_KEY:

os.environ["OPENAI_API_KEY"] ="EMPTY"

• Define additional autogen-specific environment variables as required:

os.environ["MODEL_NAME"] ="llama3"
os.environ['OAI_CONFIG_LIST'] ='[{"model": "EMPTY","api_key": "EMPTY", "max_tokens":1000}]'
os.environ["AUTOGEN_USE_DOCKER"] = "false"

Step 2: Agent Definition

Autogen provides two agent types. The ConversableAgent is a generic agent which can be customized with a system prompt and behavior regarding Python code execution and function calling. The AssistantAgent is a specialized agent with a default system prompt and the capability to execute generated Python code.

We will use the former, and define the prompt as shown

SYSTEM_PROMPT = """
You are a knowledgeable librarian that answers questions from your supervisor.

Constraints:
- Think step by step.
- Be accurate and precise.
- Answer briefly, in few words.
- Reflect on your answer, and if you think you are hallucinating, repeat this answer.
"""

system_message = {
  'role': 'system',
'content': SYSTEM_PROMPT
}

agent = ConversableAgent(
  name="librarian",
  system_message=SYSTEM_PROMPT,
  human_input_mode="NEVER",
  llm_config={
    "config_list": config_list,
    "timeout": 180,
    "temperature": 0.2},
)

Essentially, you can start a conversion directly with the agent. But the printed interactions might be confusing because the user input is not differentiated:

librarian (to librarian):

Please list 5 books about the history of human kind.

For this, and for better customization and multi-agent chats, you should define a user proxy too.

Step 3: User Proxy Definition

The user proxy is an agent type that allows the human user to participate in agent conversations. Interestingly, it is implemented as a subclass of ConversableAgent to explicitly wait for human input, which is especially helpfull when an agent creates Python code that should be executed.

Define the user proxy as shown:

user = UserProxyAgent(
  name="supervisor",
  human_input_mode="NEVER",
  max_consecutive_auto_reply=1,
  is_termination_msg=lambda x: x.get("content", "").strip().endswith("TERMINATE"),
)

Step 4: Implement Interactive Usage

Interactive usage is the default mode of autogen: it spawns a continuous loop, waits for user input, invokes the LLM, shows its output, and gives control to the user again. This mode also creates a complete chat history, making the agent aware of the complete conversation.

Use this snippet:

user.initiate_chat(agent)

For using such a chat in another application, for example invocation in a GUI or inside a Jupyter notebook, the initiate_chat function returns a complete chat history object for displaying. But this topic will be explored in future articles.

Example Conversion

Let’s invoke the agent in a discussion about books.

supervisor (to librarian):

Please list 5 books about the history of human kind.

--------------------------------------------------------------------------------
librarian (to supervisor):

Here are five book recommendations that explore the history of humanity:

1. "A History of Humanity" by Jacques Barzun - A comprehensive and engaging narrative that spans from prehistory to modern times.
2. "The Story of Humanity" by J.M. Roberts - A sweeping account of human civilization, covering 3 million years of history.
3. "The Human History of the World" by Robert C. Solomon - A concise and accessible overview of human history, from ancient civilizations to modern times.
4. "A Short History of Nearly Everything" by Bill Bryson - A delightful and informative book that covers a wide range of scientific and historical topics, including the history of humanity.
5. "The History of Humanity: From the Origins of Homo Sapiens to the Present Day" by Colin McEvedy - A thorough and well-researched account of human history, covering prehistory, ancient civilizations, and modern times.

Please note that these are just a few examples, and there are many more books on this topic. If you have any specific preferences or interests, I can try to suggest more targeted recommendations!

--------------------------------------------------------------------------------
supervisor (to librarian):

Tell me more about "A Short History of Nearly Everything" by Bill Bryson.

--------------------------------------------------------------------------------
librarian (to supervisor):

"A Short History of Nearly Everything" is a fantastic book!

Published in 2003, this book is a comprehensive and engaging overview of the history of science, from the earliest discoveries to modern times. Bill Bryson, an American author and journalist, takes readers on a journey through the development of human understanding, covering topics such as:

1. The origins of life on Earth
2. The rise of ancient civilizations
3. The scientific revolution of the 16th century
4. The discovery of the universe's vastness
5. The evolution of humans and our place in the natural world

Bryson's writing style is approachable, witty, and entertaining, making this book an excellent choice for readers who may not be experts in science or history. He weaves together fascinating stories, anecdotes, and insights from various fields, including biology, physics, chemistry, astronomy, and more.

Some notable aspects of the book include:

* Bryson's ability to explain complex scientific concepts in simple terms
* His use of engaging examples and historical context to illustrate key ideas
* The inclusion of lesser-known scientists and discoveries, which adds depth and variety to the narrative
* The author's own curiosity and enthusiasm for learning, which is contagious!

Overall, "A Short History of Nearly Everything" is a delightful and informative read that will appeal to anyone interested in the history of science and human understanding.

--------------------------------------------------------------------------------
supervisor (to librarian):

Can you contrast this with the book "Sapiens: A Brief History of Humankind" from Yuval Harari?

--------------------------------------------------------------------------------
librarian (to supervisor):

What a great comparison!
Both "A Short History of Nearly Everything" by Bill Bryson and "Sapiens: A Brief History of Humankind" by Yuval Noah Harari are popular and engaging books that explore the history of humanity. However, they have distinct approaches, focuses, and styles:
Bryson's book:

* Is a more general overview of science and its impact on human understanding
* Covers a wide range of scientific topics and discoveries from ancient times to modern era
* Has a more narrative style, with stories and anecdotes woven throughout the text
* Focuses on the development of scientific knowledge as a driving force behind human progress

Harari's book:

* Is a comprehensive history of humankind, covering 300,000 years of human existence
* Explores the evolution of Homo sapiens from a biological perspective, highlighting key milestones and developments
* Analyzes how humans have shaped their own destiny through culture, language, and technology
* Focuses on the impact of agriculture, cities, empires, and globalization on human history

Key differences:

* Bryson's book is more focused on scientific discoveries and their implications, while Harari's book delves deeper into human societies and cultures.
* "A Short History" has a broader scope, covering many scientific fields, whereas "Sapiens" concentrates on the evolution of Homo sapiens as a species.
* Bryson's writing style is more conversational and humorous, while Harari's prose is often more formal and analytical.

Both books are excellent choices for readers interested in history, science, or human culture. If you prefer a more general overview of scientific progress, Bryson's book might be the better fit. However, if you're looking for a comprehensive and thought-provoking exploration of human history, Harari's "Sapiens" is an outstanding choice.

Complete Code

Here is the complete code:

from autogen import ConversableAgent, UserProxyAgent

config_list = [
  {
    "model": "llama3:instruct",
    "base_url": "http://localhost:11434/v1",
    "api_key": "ollama",
  }
]

SYSTEM_PROMPT = """
You are a knowledgeable librarian that answers questions from your supervisor.

Constraints:
- Think step by step.
- Be accurate and precise.
- Answer briefly, in few words.
- Reflect on your answer, and if you think you are hallucinating, repeat this answer.
"""

system_message = {'role': 'system',
      'content': SYSTEM_PROMPT}

agent = ConversableAgent(
  name="librarian",
  system_message=SYSTEM_PROMPT,
  human_input_mode="NEVER",
  llm_config={
    "config_list": config_list,
    "timeout": 180,
    "temperature": 0.2},
)

user = UserProxyAgent(
  name="supervisor",
  human_input_mode="ALWAYS",
  is_termination_msg=lambda x: x.get("content", "").strip().endswith("TERMINATE"),
)

user.initiate_chat(agent)

Conclusion

This article showed how to define an agent and history-aware interactive chat with the autogen framework. Only four parts are required: a) the LLM configuration, a list of OpenAI API compatible local or external endpoints, b) the agent definition and its system prompt, c) a user proxy object to better steer the communication, and d) starting an interactive session. The complete source code are only 40 lines. And to this foundation, you can add additional agents that support each other in task solving, add vector databases or tools, and much more. These aspects will be explored in future articles.