Introducing MATE: A Modular Testing Environment for AI Agents

Summary Lede

As AI agents become integral to business processes, reliable and repeatable testing is essential for confidence in deployment. This article introduces the Multi-Agent Test Environment (MATE) – an enterprise-grade framework for automated testing of AI agents across platforms and frameworks – and explains how its modular design addresses key challenges in agent testing. We explore why testing AI agents is critical, delve into MATE's architecture and features, compare MATE with alternative testing approaches, and outline MATE's roadmap including red-team testing, enhanced cloud deployment, and support for emerging agent frameworks.

Importance of Testing AI Agents

AI agents built with Microsoft Copilot Studio are powerful but complex systems. They combine natural language understanding, generative AI, and business logic, often operating in critical scenarios (customer support, data retrieval, workflow automation, etc.). Ensuring these agents work correctly and safely under diverse conditions is as important as testing traditional software – if not more so. Key reasons why rigorous agent testing is essential include:

• Reliability and Consistency: Unlike deterministic software, AI agents can produce different answers to the same question due to their probabilistic nature. Without structured tests, one might only catch issues by manually typing questions and hoping for the right answer, a fragile approach. Automated testing provides consistency – the same test can be run repeatedly to ensure the agent’s behavior remains reliable after updates.
• Enterprise-Grade Quality: In enterprise deployments, an untested agent can lead to incorrect or even unsafe outputs, damaging user trust or violating compliance. Ad-hoc testing that “relies on intuition instead of structured testing” doesn’t scale for enterprise needs. Organizations require repeatable, at-scale test processes to validate that agents meet quality standards (accuracy, relevance, safety) consistently before and after release.
• Complex Multi-turn Interactions: Copilot Studio agents often handle multi-turn conversations, maintaining context across multiple user and agent turns. Testing these multi-step dialogues manually is time-consuming and error-prone. Automated test suites allow developers to simulate complex conversation flows (with varying user inputs, branching dialogs, tool invocations, etc.) and verify the end-to-end behavior in one run. This ensures that the agent can handle scenario-based conversations robustly, from greeting to task completion.
• Nondeterministic and Generative Responses: When agents use generative AI capabilities, they might produce creative or unexpected phrasing. Verifying such responses is not as simple as exact string matching. Effective testing must evaluate responses on semantic correctness, completeness, and compliance, even if wording varies. This introduces a challenge: how do you automatically judge an AI-generated answer’s quality? We’ll see how MATE tackles this with an AI-based “judge” component.
• Frequent Updates and Continuous Integration: Agents are rarely static – their underlying prompts, skills, and knowledge sources evolve. Without automation, re-testing the agent after each change or on a schedule (for example, to catch drift or regressions) would be prohibitively labor-intensive. A good agent testing framework enables continuous integration (CI) pipelines and nightly runs, so that any breaking change or quality degradation is caught early. This is crucial for scaling up the number of agents in production while keeping maintenance overhead low.
• Transparency and Debugging: When a test fails, developers need insights into why. For example, did the agent retrieve the wrong data because of an intent misclassification? Or did it produce a partially correct answer that was marked as a failure due to a strict check? Good testing tools provide detailed reporting – conversation transcripts, logs, and metrics – to help pinpoint the root cause of failures. This accelerates debugging and continuous improvement of the agent.

In summary, robust testing of agents is the linchpin for trustworthy AI deployments. It allows teams to validate functionality, accuracy, robustness, and safety in a systematic way. This need has driven Microsoft to introduce solutions like the Power CAT Copilot Studio Kit (a Power Platform solution for agent testing) and, more recently, the built-in Agent Evaluation feature in Copilot Studio (now in preview). However, each solution comes with certain limitations or prerequisites, which the new MATE aims to overcome. Before comparing approaches, let’s first introduce MATE and how it works.

Introducing MATE: A Modular Testing Framework for AI Agents

Link to MATE GitHub Repository

Link to MATE Wiki

Multi-Agent Test Environment (MATE) is an internal project and framework designed to provide automated, comprehensive testing for AI agents, initially focusing on Microsoft Copilot Studio agents. MATE was created to address the challenges above by combining enterprise-grade tooling with a modular, extensible architecture. In essence, MATE allows developers and testers to connect to a running Copilot Studio agent, simulate conversations, evaluate the agent’s responses against expected outcomes using AI, and produce detailed metrics and reports – all in an automated fashion.

MATE’s approach can be seen as bringing many of the benefits of the Copilot Studio Kit into a single, code-first testing environment. Rather than a Power App solution, MATE is a pure .NET 9 application that you can run in a container stack. This design choice means MATE operates outside the constraints of the Power Platform, giving developers more flexibility in how and where they run their tests.

Let’s break down how MATE works and how it addresses key testing challenges:

• Direct Line Integration (Live Agent Testing): MATE connects to the agent through its Direct Line API endpoint. This is the same interface used by real chat channels (like Teams or a custom web chat). By using Direct Line, MATE ensures it's testing the deployed agent exactly as end-users experience it. The tool can send a sequence of user messages and receive the agent’s replies in turn, thereby automating full multi-turn conversations. This addresses the challenge of multi-turn flows by allowing complex scenario scripts to be executed automatically. It’s effectively like an automated “test chat” but running dozens or hundreds of predefined conversations unattended.
• Test Case Definition and Multi-turn Flows: In MATE, you can define a test case with multiple steps of user input (representing a conversation) and the expected outcomes. Expected outcomes can include:
  • Expected Intent and Entities – i.e., which topic or action the agent should trigger and which key data (entities) it should extract.
  • Acceptance Criteria – specific conditions that constitute a pass/fail for the test (for example, certain keywords must appear in the answer, or a certain API call must be made).
  • Reference Answer – an ideal answer text or outline for comparison. Each test case can be labeled with a priority or category, useful for organizing large test suites (e.g., “P1 critical flows”, “Edge cases”, etc.). By supporting multi-step conversations in test cases, MATE ensures you can test end-to-end agent behavior, not just isolated single-turn Q&A.

• “Model-as-a-Judge” Evaluations: One of MATE’s most powerful features is using an AI model to evaluate the quality of the agent’s response. Rather than relying only on hard-coded checks (exact matches or simple contained keywords), MATE sends the agent’s answer along with the reference answer and validation criteria to a Large Language Model (LLM) – for instance, an Azure OpenAI GPT-4 model – which acts as an impartial judge. This AI Judge scores the response across multiple evaluation dimensions:

  • Task Success: Did the agent fulfill the user’s request or solve the user’s problem?
  • Intent Match: Did the agent correctly understand what the user was asking for?
  • Factuality: Is the information provided true and accurate (no hallucinations or incorrect data)?
  • Helpfulness/Completeness: Is the answer complete, well-structured, and does it address the user’s need effectively?
  • Safety/Compliance: Does the response avoid policy violations (no sensitive data exposure, no disallowed content)? Each of these dimensions is scored (e.g. 0.0 to 1.0), and MATE can apply configurable weightings to decide if a test passes or fails overall. For example, you may require 0.9+ on Task Success and Intent Match, tolerate a lower score on style metrics like Helpfulness, and demand a perfect score on Safety. This approach directly tackles the challenge of evaluating nondeterministic generative answers: even if the agent’s wording differs from the expected answer, the AI Judge can still determine that the answer is essentially correct and useful. Conversely, if the agent’s response is irrelevant or contains errors, the AI Judge will assign low scores, causing the test to fail. This method provides a nuanced, context-aware evaluation that traditional automated tests struggle to achieve. (Internally, the AI Judge uses prompt-based prompting of an LLM with the expected answer or criteria to get these scores.)

• In Addition, MATE also supports other judge types, such as:

  • RubricsJudge – A fully deterministic judge that evaluates responses using explicit rules such as Contains, NotContains, and Regex, making it ideal for compliance, safety, and reproducible pass/fail checks.
  • HybridJudge – A cost‑efficient combination judge that first gates responses with deterministic rubrics and then applies an LLM for deeper qualitative scoring only where needed.
  • CopilotStudioJudge – A Copilot‑Studio‑specific LLM judge that is citation‑ and grounding‑aware, aligning evaluations with Copilot Studio’s default reasoning and response patterns:
  • GenericJudge – A lightweight, zero‑cost judge based on simple keyword and regex matching, intended for fast smoke tests and offline or CI scenarios

• Automated Test Generation from Documentation: Authoring a comprehensive set of test cases can be labor-intensive. MATE addresses this by allowing you to upload documents (PDFs or text files) that are relevant to your agent’s domain or knowledge base. It then automatically:

  • Extracts text content from the documents (using a PDF parser).
  • Indexes and chunks the content for semantic analysis (using a Lucene-based index).
  • Generates potential questions and answers from the content using an LLM. The outcome is a set of suggested Q&A pairs or even multi-turn conversation scenarios derived from the documentation. For example, if you upload a product FAQ PDF, MATE can generate likely customer questions and the correct answers from that PDF. These can be reviewed and added to your test suites. This feature helps broaden test coverage automatically, ensuring the agent is tested on real knowledge it’s supposed to have, and catching gaps where it might not respond correctly. It’s an intelligent way to keep tests in sync with content. (Notably, Copilot Studio Kit in the Power Platform also introduced an AI-based test generation in preview, which uses the agent’s topics and knowledge to generate example questions. MATE provides a similar capability but on external docs and with full control of the generated cases.)

• Detailed Reporting and Analysis: After executing tests, MATE provides rich metrics and logs. In the Web Dashboard, you can see overall pass rates, success trends over time, and drill down into individual test runs. Each test run retains the transcript of the conversation and the scores for each evaluation dimension, so you can inspect exactly where a particular test failed. This addresses transparency: instead of just “Test 5 failed”, you can see that it failed because, say, Factuality scored low (perhaps the agent gave a wrong detail), and even read the conversation to diagnose the issue. MATE’s Runs view lets you compare results between runs – useful for spotting regressions after an update. All test data (test cases, results, transcripts, etc.) are stored in a local PostgreSQL database for quick retrieval and can be queried or exported for additional analysis.

• Web UI and CLI for Different Use Cases: MATE offers two interfaces:

  • A Web Application (built with ASP.NET Blazor Server) for an interactive experience. This is ideal for exploratory testing, configuring your test suites, and reviewing results. The UI includes a setup wizard for initial configuration (entering your agent’s Direct Line credentials and your AI model info) to generate the necessary settings file. Testers can use the web UI to kick off test runs on-demand, monitor progress, and view results in real time.
  • A Command-Line Interface (CLI) tool for automation. The CLI allows you to run tests as part of scripts or pipelines. For example, you can incorporate dotnet run --suite "Regression Suite" into a DevOps or GitHub Actions pipeline, so that whenever the agent’s bot is updated or its content changes, the test suite runs and verifies everything still works. The CLI returns an exit code indicating success or failure (0 if all tests passed, non-zero if any test failed), which CI systems can use to pass/fail a build. This enables true CI/CD for AI agents – a failed test can halt a deployment, preventing flawed agent versions from going live.

• Containerized and Extensible Architecture: MATE is designed to be run in a self-hosted manner, giving teams full control. It doesn’t require a SaaS backend or a Dataverse environment – you just need a machine that can reach the internet for calling the agent service and the AI model endpoint. This avoids many of the Power Platform licensing constraints associated with the Copilot Studio Kit (discussed later). The architecture is modular by design, with separate components (projects) for domain logic, data storage, core services, web UI, and CLI. This modularity not only enforces clean separation of concerns, but also sets the stage for supporting multiple types of agents in the future. In fact, MATE’s roadmap includes extending support to other agent frameworks beyond Copilot Studio – the core logic (test execution, AI judging, etc.) can be adapted to different agent APIs by swapping out the integration layer. Early code commits already hint at multi-agent support being developed and even an upcoming “red teaming” module for adversarial testing (there are structural hooks in the codebase for this, though the feature is not yet implemented). This means MATE is not a one-off tool but a growing platform for comprehensive AI agent testing across the board.

MATE Architecture at a Glance

Internally, MATE is built with a modern software architecture using the latest Microsoft stack:

• .NET 9 with C# – providing performance and cross-platform support.
• ASP.NET Core Blazor Server for the web front-end – delivering a rich interactive UI for managing tests and viewing results.
• Entity Framework Core (with PostgreSQL) – for the local database that stores test cases, results, transcripts, etc., ensuring persistence without requiring an external DB server.
• Azure AI OpenAI SDK – to connect to the AI Judge model hosted on Azure’s AI services (Azure OpenAI “Foundry”). This is how MATE queries an LLM for evaluation of answers.
• Lucene.NET – used for full-text indexing in the document-driven test generation feature, to find relevant content in uploaded docs for question generation.
• PDF processing libraries (e.g., UglyToad.PdfPig) – to extract text from PDF documents for test generation.
• Serilog – for structured logging of events and errors, helping with diagnosing issues in test executions.

The solution is divided into several components (projects) reflecting a modular design: a Domain layer for core models and interfaces, a Data layer for database access, a Core services layer (implementing the judge logic, execution engine, etc.), a WebUI for the front-end, and a CLI project for the command-line interface. This modularity makes it easier to maintain and extend specific parts (for example, adding a new agent connector could be done by introducing a new service in the Core or a new API integration, without touching the UI or data layers).

Overall, MATE is engineered to be a scalable, extensible test harness for AI agents. It’s currently focused on Copilot Studio agents, but its principles apply broadly. Next, we’ll compare MATE to the Copilot Studio Kit – the established testing solution from Microsoft’s Power CAT team – to understand their differences and use cases.

Comparing MATE with Copilot Studio Kit

Microsoft’s Power CAT Copilot Studio Kit is an existing solution aimed at testing and managing Copilot Studio agents. It’s a Power Platform solution (managed package) that provides a canvas app or model-driven app interface, along with Dataverse entities and Power Automate flows, enabling test case creation, automated test runs via the Direct Line API, and analytics (such as conversation transcripts, dashboards, etc.). The Copilot Studio Kit was instrumental in early adoption of agent testing – it allowed makers to do things like bulk import test cases (via Excel), run them from a UI, and even integrate with Azure DevOps pipelines via Power Platform build tools.

However, the Copilot Studio Kit has some inherent characteristics stemming from its Power Platform foundation. Below is a comparison of MATE vs. Copilot Studio Kit across key dimensions:

Aspect	MATE (Multi-Agent Test Environment)	Copilot Studio Kit (Power CAT)
Deployment Model	.NET application (containarized). Runs locally or in cloud; launched via web UI or CLI on demand.	Power Platform managed solution. Deployed to a Dataverse environment; accessed via Power Apps interface.
Licensing & Costs	source-available - CC BY-NC 4.0 . Requires .NET runtime and an Azure OpenAI endpoint (for AI Judge) which may incur usage costs. No special Power Platform licensing needed beyond having a Copilot Studio agent to test.	Provided by Microsoft Power CAT as a sample solution (available on GitHub). However, requires Power Platform premium licenses: a Dataverse environment, and for certain features, AI Builder credits (for generative answer analysis). Usage of Dataverse and Power Automate in the kit might consume capacity or require specific licenses.
Technology Stack	Modern .NET 9 stack; Blazor Web UI, CLI tool, local PostgreSQL DB. Integrates with Azure services (OpenAI) for evaluation. Highly customizable and extendable by developers (source code available).	Low-code Power App + Dataverse. Relies on standard Power Platform tech (model-driven app or canvas app, Dataverse tables, Power Automate flows, AI Builder for some AI tasks). Customization is limited to what Power Platform allows.
Test Creation	Supports manual creation of test cases via UI or by defining JSON/CSV, etc., and auto-generation of test cases from documents using LLMs. Test cases can include multi-turn dialogues in one case. Organized into test suites for batch execution.	Supports manual test case input (through the app or via Excel import/export). Also supports multi-turn test cases and offers some AI-assisted generation of test questions from agent topics/knowledge (in Preview, via the Agent Evaluation integration). Test cases stored in Dataverse; grouping of tests supported (by test set).
Test Execution	Runs tests externally by connecting to the agent’s Direct Line channel (or Web Channel with secret & bot ID). Offers a CLI for headless execution (suitable for CI pipelines) and a web interface for interactive runs. Test results are stored locally and displayed in the web UI with analytics.	Executes tests through Copilot Studio’s Direct Line API as well, orchestrated by Power Automate flows under the hood. Typically run on-demand from the app’s interface. There is integration for pipelines via Power Platform build tools, though this is more complex to set up. Results are stored in Dataverse and can be viewed via in-app dashboards or exported.
Evaluation Methodology	AI-driven semantic evaluation: uses a GPT-based AI Judge to score responses on multiple quality dimensions (task success, intent match, factual correctness, etc.). This allows flexible, semantic comparisons rather than simple exact matches. Configurable pass thresholds provide fine-grained control. Also supports explicit pass/fail rules (acceptance criteria) where needed.	Rule-based and some AI: supports exact or partial response matching, checking for expected keywords or presence of attachments, etc. For generative answers, the kit uses AI Builder to compare the agent’s answer with a reference answer for similarity. It also retrieves telemetry from Application Insights to help explain failures. Plan validation tests examine the agent’s action plan against expected tools (for orchestration scenarios).
Modularity & Extensibility	Designed to be modular and extensible. The core can be extended to new agent types (plans to support other AI agent frameworks are in progress). The evaluation component (AI Judge) can be pointed to different models or adapted with different prompts. Being source-available, organizations can modify or extend MATE (e.g., add custom evaluation metrics, integrate with other data sources) to fit their needs.	Focused scope, limited extensibility. The Copilot Studio Kit is specific to Copilot Studio agents and deeply tied to the Power Platform environment structure. It’s not architected to test arbitrary other agents. Customizing it generally means modifying the Power App or creating new Dataverse fields/flows, which requires Power Platform expertise.
Data & Infrastructure	Stores test artifacts and results in a local database within the application. No cloud infrastructure needed to get started; data stays within the user’s environment. For scaling up, the application could be hosted on a server or in Kubernetes (containerization support is under development). Because it’s self-hosted, data sovereignty and privacy can be managed internally.	Relies on Dataverse for storing tests and results, and optionally uses other services (App Insights, SharePoint, etc.) for logs and knowledge management. This provides seamless integration if you’re already within the Microsoft ecosystem, but it requires that all data be in the Power Platform cloud.

Table: Comparison of MATE vs. Copilot Studio Kit across key aspects of testing functionality and usage.

As seen above, MATE and Copilot Studio Kit share the same goal – improving agent quality through automated testing – but they differ in implementation approach. MATE is more developer-oriented, offering flexibility, openness, and extensibility, whereas the Copilot Studio Kit is maker-friendly, integrated in the Power Platform with a ready-to-use interface but comes with platform constraints.

From a Microsoft perspective, the Copilot Studio Kit was a bridge solution to empower agent creators with testing capabilities before deeper platform features were available. Now, with Agent Evaluation built directly into Copilot Studio (currently in preview), some capabilities of the kit are being absorbed into the product itself – for instance, AI-generated test queries and built-in execution of test sets. Still, the Kit provides additional tooling (like dashboards, inventory, governance features) that are useful in complex environments.

see articles:

Ship Copilot Studio Agents with Confidence: Master Automated Testing with the Copilot Studio Kit

Testing Copilot Studio Agents: Copilot Studio Kit vs. Agent Evaluation (Preview)

MATE, on the other hand, is an independent effort to provide a robust testing harness that can evolve fast and go beyond what the closed-source product features offer. It is not limited by Power Platform’s boundaries (for example, one could imagine integrating MATE with other LLM evaluation criteria, or hooking it up to monitor backend APIs invoked by the agent). Additionally, MATE’s modular nature means it could incorporate other agent types into the same testing dashboard. For example, if you have a fleet of different AI bots – some built in Copilot Studio, some using Azure OpenAI Orchestration, some third-party – MATE could theoretically be extended to test them all in one place, whereas the Copilot Studio Kit is only for Copilot Studio agents.

When to use which? If you are a Power Platform maker or IT admin who wants a straightforward, supported way to test Copilot Studio agents and you’re already comfortable with Power Apps and Dataverse, the Copilot Studio Kit is a solid choice. It integrates nicely with the environment (and your data, logs, etc.) and doesn’t require coding to use. However, you’ll need the necessary licenses and some patience to configure the environment, and you won’t be able to easily customize how tests are evaluated beyond what Microsoft provides.

If you are a developer or dev team looking for a more flexible, code-driven approach – especially if you want to integrate agent testing into a DevOps pipeline or extend testing to specialized scenarios – MATE is very appealing. It does require .NET and some setup, but it gives you full control. You can run it locally for rapid iteration, include it in automated builds, and tweak it to your needs. There’s also no dependency on having a Power Platform environment or any particular license. You do need access to an Azure OpenAI service (or you could swap in another LLM API if desired) to leverage the AI judge, but that is relatively straightforward for most enterprise scenarios.

Roadmap and Future Enhancements in MATE

MATE is an evolving project, and there are several notable enhancements on the roadmap:

• Support for Additional Agent Types: As of now, MATE supports testing Microsoft Copilot Studio agents exclusively, because it specifically uses Copilot’s Direct Line API and related assumptions (like the concept of “topics” and Dataverse knowledge base) in its current version. However, the architecture is being extended to accommodate other agent platforms. Future versions are expected to introduce modules for other agent types – for example, the ability to test Microsoft Agent Framework agents, or even agents built with entirely different frameworks. This will broaden MATE’s applicability across various “agentic AI” solutions used within Microsoft and beyond, making it a one-stop testing hub for heterogeneous AI systems.
• Integrated Red Teaming: In addition to “blue team” style functional testing (checking that the agent does what it’s supposed to), MATE aims to incorporate “red teaming” capabilities. Red teaming in AI refers to attacking or stress-testing the agent with malicious or unexpected inputs to probe its defenses and safety measures. This can include testing the agent’s response to prompt injections, inappropriate content requests, or attempts to trick the agent into breaking rules. The goal is to ensure the agent is robust against misuse or adversarial users. The MATE codebase already contains the foundation for a Red Teaming module, but this is currently just a skeleton (non-functional in the current release). Once completed, this feature will allow users to run a suite of adversarial tests (perhaps using predefined malicious prompts or common attack patterns) against their agents and get a report on vulnerabilities or policy compliance issues. This is a critical part of AI system testing, especially for enterprise scenarios, and its inclusion will differentiate MATE further by offering a more comprehensive safety evaluation than what is currently possible with the Copilot Studio Kit or built-in Agent Evaluation (which, so far, focus on correctness and performance rather than adversarial robustness).
• Cloud and Scalable Deployment: Presently, MATE runs as a local docker stack. Looking forward, the project plans to simplify deployment on Azure, likely via containerization. Kubernetes support is on the roadmap, meaning you might be able to deploy MATE as a set of containers (web app, background worker, etc.) in an AKS (Azure Kubernetes Service) or similar environment. This will enable team-wide usage at scale – multiple testers or developers could share a MATE instance, run tests concurrently, and store results in a central location, much like a web application service. Cloud deployment will also facilitate integration with other services (for example, connecting to Azure DevOps for automatic test triggers, or scaling out the AI Judge component).
• UI/UX and Usability Improvements: As an source-available project, MATE will continue to refine its user interface and ease of use. Features on the horizon could include richer test editing experiences (perhaps a visual conversation flow editor), more analytics dashboards (trend of agent performance over time, flakiness of certain tests, etc.), and integration with agent design tools (for example, pulling in agent topics or suggesting tests based on recent real user conversations – aligning with how built-in Agent Evaluation reuses Test Pane interactions).

Conclusion

Testing AI agents is no longer optional – it’s a necessity for any organization that wants to confidently deploy AI solutions. Agent Development Solutions empower the creation of sophisticated AI Agents, but ensuring these agents function correctly, safely, and efficiently requires going beyond manual testing or one-off trials. This is where testing frameworks like Copilot Studio Kit and MATE come into play.

MATE (Multi-Agent Test Environment) represents a next-generation approach to agent testing. It addresses the limitations of earlier tools by adopting a fully modular, code-first architecture that can keep pace with the rapidly changing AI landscape. By using MATE, developers and testers gain the ability to thoroughly automate conversations with their agents, evaluate responses with the help of AI, generate tests from existing knowledge, and integrate all this into continuous delivery pipelines. The outcome is a higher degree of assurance that your Copilot Studio agent will perform as expected when it’s in production – responding correctly to user queries, using the right tools, and staying within the guardrails.

In comparison to the Power Platform-based Copilot Studio Kit, MATE offers more flexibility, extensibility, and independence. You won’t be constrained by specific licensing or environment setups, and you can tailor the tool to your needs. On the other hand, it’s a more technical solution that may require developer effort to set up and maintain, whereas the Copilot Studio Kit is more turn-key if you’re already within Microsoft’s ecosystem. It’s encouraging to see both approaches available, as they cater to different audiences.

Ultimately, MATE’s importance goes beyond just testing Copilot Studio agents. It signifies an evolving philosophy in the AI agent world: that testing and evaluation should be first-class citizens in the development lifecycle of AI systems, just as they are in traditional software development. With AI models and agents becoming increasingly central to applications, tools like MATE help ensure we can trust these systems through systematic validation. MATE’s deep integration of AI for testing (using an AI to test another AI) is an innovative approach that can significantly enhance the rigor of evaluations.

In summary, MATE enables teams to ship Copilot Studio agents (and, in the future, other AI agents) with greater confidence. It provides the means to catch issues early, improve agents iteratively based on test feedback, and guard against regressions as agents evolve. By combining the power of automation with the wisdom of AI judging, MATE exemplifies a “test smarter” strategy for the era of generative AI – ensuring that our intelligent agents are not only smart, but also reliable, safe, and effective when they go to work for us.