Maia - Multi-AI Agent Test Framework

#ai #python #opensource #testing

Hey Dev community!
I want to share with you my recent open-source project which I am working on - Test Framework for testing Multi-AI Agent systems.

Website: Maia

Framework is written in Python and uses standard pytest approach.

The main features are:

Multi-Agent Simulation - Simulate conversations and interactions between multiple AI agents
Extensible Provider Model - Easily integrate with various AI model providers (e.g., LiteLLM, LangChain, CrewAI)
Built-in Assertions - A suite of assertions to verify agent behavior, including content analysis and participation checks
Dashboard for visualization - NextJS application to show test results for checking and debugging purpose.

You can use the framework for testing such scenarios like:

asking various models for the same thing and check the results
broadcasting a prompt and wait for the completion without user intervention (using not only CrewAI but also other providers!)
simulate tool calling, so checking if your AI Agent uses your tool in a proper way
much much more

As an example, please see how easy is to write a test:

class TestConversationSessions(MaiaTest):
    def setup_agents(self):
        self.create_agent(
            name="Alice",
            provider=GenericLiteLLMProvider(config={
                "model": "ollama/mistral",
                "api_base": "http://localhost:11434"
            }),
            system_message="You are a weather assistant. Only describe the weather.",
        )

        self.create_agent(
            name="Bob",
            provider=GenericLiteLLMProvider(config={
                "model": "ollama/mistral",
                "api_base": "http://localhost:11434"
            }),
            system_message="You are an assistant who only suggests clothing.",
        )

@pytest.mark.asyncio
  async def test_agent_to_agent_conversation(self):
      session = self.create_session(["Alice", "Bob"])

      # Alice initiates conversation with Bob
      await session.agent_says("Alice", "Bob", "Given the weather: rainy and 20 degrees Celsius, what clothes should I wear?")
      response = await session.agent_responds("Bob")
      assert_agent_participated(session, "Bob")

      # Bob responds back to Alice
      await session.agent_says("Bob", "Alice", f"Based on my info: {response.content}")
      response = await session.agent_responds("Alice")
      assert_agent_participated(session, "Alice")