Intro:
We built a lightweight regression-testing pipeline for our Retrieval-Augmented Generation (RAG) agent using Power Automate, Copilot Studio (or Execute Agent), and an AI Builder evaluation prompt. The goal: turn a curated set of knowledge-base (KB) questions into a repeatable test pack, run them automatically after KB or instruction changes (or model upgrades), and deliver deterministic pass/fail reports to the product owner.
How it works (high level):
Work with the product owner to define a small, focused regression pack: representative KB questions + the expected/desired responses or required attributes (formatting, tone, required facts). Keep these single-turn questions to start.
- Store the test pack (Excel/Dataverse/SharePoint list). Each row is a test case: ID, question, expected snippet or attributes, and acceptance criteria.
- Automate with Power Automate (flow shown in the screenshot): a) Manually trigger or schedule the flow. b) GetTestCases (read test pack from Excel/Dataverse). c) Scope > Apply to each test case: d) Compose input payload (question + metadata). e) Execute Agent and wait (run the RAG agent for that single-turn question). f) Run a prompt (AI Builder or LLM) that evaluates the agent response against the acceptance criteria.
Update a row with the result (Pass/Fail, metrics, evaluator JSON).
After loop: aggregate results, calculate pass rate, and send a report to the product owner (email/Teams/Files).
The evaluation prompt (core idea) We used an evaluator prompt that positions the LLM as an impartial critic and returns a strict JSON evaluation. This is inspired by proven LLM-as-judge patterns and gives us a deterministic, machine-readable output to gate deployments.
- Output is machine-readable (JSON/CSV) so results can be tracked and automated.
- Each test shows scores and a short reason so the product owner sees why something failed.
- Clear PASS/FAIL and a short failure reason make go/no-go decisions easy. Every result links back to the original test case so fixes are quick to find.
- Include run info (date, model/KB version) so failures can be reproduced.
- Show a one-line summary (e.g., "28/30 passed — 93%") for a fast health check.
- Prioritize failures by impact so the product owner knows what to fix first.
- Attach remediation notes (short next steps) so engineers know how to resolve issues.
A clear, structured evaluation report turns a flood of agent outputs into actionable intelligence — empowering product owners to make fast, confident decisions about content changes, model updates, and releases.
Top comments (0)