Setting Up Your First Multi-Agent Test with Maia

#ai #python #testing #tutorial

In one of the previous articles, I introduced the MAIA Framework — an open-source toolkit for testing multi-agent AI systems. We discussed what it does, why it exists, and some of its key features such as assertions and validators.

Now it’s time to get practical.

In this post, we’ll set up our first test with MAIA, using both assertions and validators.

Note: I am assuming you have your Python project set up.

Installing the framework.

pip install maia-test-framework

Create a simple test file

import pytest
from maia_test_framework.testing.base import MaiaTest
from maia_test_framework.providers.generic_lite_llm import GenericLiteLLMProvider

class TestContentAssertions(MaiaTest):
    def setup_agents(self):
        self.create_agent(
            name="Alice",
            provider=GenericLiteLLMProvider(config={
                "model": "ollama/mistral",
                "api_base": "http://localhost:11434"
            }),
            system_message="You are a helpful AI assistant. You will follow user instructions precisely."
        )
    @pytest.mark.asyncio
    async def test_basic(self):
        session = self.create_session(["Alice"])

        await session.user_says("Please describe the usual weather in London in July, including temperature and conditions.")

        response = await session.agent_responds("Alice")

        assert "sunny" in response.content

Breaking down the test

Creating an agent

First, we define the agent that will be under test. The function takes a few key parameters:

name- a unique identifier for the agent in the test.
provider- specifies which model the agent should use. MAIA provides many integrations (e.g., LiteLLM, CrewAI), and you can also create your own.
system_message- the system prompt describing what the agent is.

Creating a session

To use the agent, you first need to create a session. In MAIA, a Session groups agents into communication channels.

For simple agentic systems, you’ll likely use just one session, since all agents can talk to each other freely.

Simulating a simple conversation.

Next, we simulate a short conversation between the user and the agent. The user asks for the weather, and the agent responds.

Finally, we check the content of the agent’s reply for specific patterns (in this case, whether it mentions “sunny”).

This is a very simple example — now let’s extend it with assertions and validators.

Adding Maia assertion

Assertions can be attached to a session, so every message is automatically checked.

For example, the built-in assert_professional_tone ensures that responses don’t contain unprofessional language (such as lol _ or _u r).

Here is the example of using such assertion:

from maia_test_framework.testing.assertions.content_patterns import assert_professional_tone

...

session = self.create_session(["Alice"], assertions=[assert_professional_tone])

You can also define your own custom assertions (we’ll cover this in a future article).

Adding Maia validator

Validators check the overall session instead of individual messages. They’re automatically run at the end of a test.

For example, this validator ensures that Alice sends at most one message in the session:

from maia_test_framework.testing.validators.performance import performance_validator

...

session = self.create_session(["Alice"], validators=[agent_message_count_validator(agent_name="Alice", max_messages=1)])