TimeWarden: An autonomous multi-agent with Copilot Studio

Time-Off Management with Autonomous AI in Microsoft Copilot Studio

After watching Ignite 2024 this October, I was amazed by the capabilities showcased in the Copilot autonomous agent presentations. Intrigued by their potential, I set out to explore and test these features, which led me to create a project designed to deepen my understanding of this innovative technology. This is how TimeWarden came to life: a multi-agent solution developed using Copilot Studio.

Copilot Studio

Copilot Studio provides a powerful platform for building autonomous AI agents that understand work processes and act on behalf of users. By combining managed SaaS infrastructure, advanced AI models, a low-code design interface, and a vast library of prebuilt connectors, these agents can automate tasks, support business roles, and enhance efficiency across various applications and data sources.

Key Benefits

• Efficiency: Autonomous agents handle repetitive tasks, freeing up employees to focus on more complex and creative work.
• Accuracy: By automating processes, agents reduce the risk of human error, ensuring more accurate and consistent outcomes.
• Scalability: Agents can be scaled to support large organizations, handling high volumes of tasks without additional human resources.
• Cost Savings: Automating routine tasks can lead to significant cost savings by reducing the need for manual labor.
• Improved Customer Experience: Agents can provide real-time support and information, enhancing customer satisfaction.

Characteristics

• Autonomous Functionality: Operating continuously, they proactively monitor business events and execute actions without manual intervention.
• Dynamic Workflow Planning: Their ability to create and adjust execution strategies in real time enables flexibility in complex environments.
• User-Friendly Integration: With low-code customization, organizations of all sizes can tailor these agents to address specific operational challenges.

These innovations mark a significant step toward creating AI-first business processes, promising to streamline operations and drive measurable business value.

TimeWarden

TimeWarden is an autonomous multi-agent system designed to manage employee time-off requests by cross-referencing them with a Jira project. It intelligently determines whether to approve or reject requests automatically or escalate them for project manager approval based on specific conditions.

It is not intended for production use but serves as a proof of concept (POC) for exploring the potential of this technology. Due to this, it has multiple limitations, and the scenario is constrained.

Additionally, TimeWarden can be extended or updated as needed. For example, while the current trigger is an email-based time-off request, it could be replaced with a Power Platform Form App or any other trigger supported by Copilot Studio. In this case, I chose email as a simplified scenario, allowing TimeWarden and its supporting agents to interpret and categorize the message.

Description

One of the first steps in creating a Copilot is providing a description of the agent, outlining its purpose, target audience, and desired outcomes.

Instructions

The Instructions section defines how the agent behaves, outlining its tasks and the approach it takes to complete them. Below are the instructions for TimeWarden:

Step 1: Checking the Email

• Verifies if the email is a time-off request. If not, it stops processing.
• Checks if the sender belongs to the Contoso project team. If not, it sends a rejection email explaining why.
• Ensures all required details (dates, hours, reason, type of leave) are provided. If anything is missing, the requester receives an email listing the missing fields.

Step 2: Reviewing the Sprint Schedule

• Retrieves the latest sprint schedule from Jira to assess how the request fits into the project timeline.
• Categorizes the request into: – Current Sprint (happening now) – Future Sprint (upcoming) – Outside Sprint (not affecting any sprint)

Step 3: Handling Current Sprint Requests

• Checks if the employee has sufficient time-off balance.
• If everything is fine, the request is sent to the manager for approval.
• If approved, the system updates the records and notifies the requester.

Step 4: Managing Requests Outside a Sprint

• If no sprint is affected and the leave balance is sufficient, the request is automatically approved.
• If the balance is insufficient, the request is rejected, and the employee is notified.

Step 5: Handling Future Sprint Requests

• Assesses whether the absence impacts sprint capacity.
• If the request does not reduce sprint availability below 80%, it is approved automatically.
• Otherwise, it is escalated to a manager for review.

At every step, TimeWarden ensures employees and managers stay informed via automated emails, streamlining time-off management.

Notice that intentionally a simplified version is provided rather than the exact instructions from Copilot Studio. One key lesson learnt through experimentation was prompt engineering , which let me to rework and refine multiple times those instructions to make them more precise and detailed so that Copilot Studio could accurately understand and execute them.

Prompt engineering is the practice of designing and refining inputs (prompts) to guide AI models, like ChatGPT or Copilot, to generate accurate, relevant, and useful outputs. It involves structuring questions, instructions, or contextual information in a way that optimizes the AI’s response.

Autonomous Triggers

Autonomous Triggers in Copilot Studio are event-driven mechanisms that allow AI agents to take action automatically, without requiring direct user input. These triggers detect specific conditions-such as an incoming email, a database update, or a scheduled event-and initiate workflows or responses accordingly, having Copilot Studio hundreds of those.

TimeWarden detects incoming emails that are likely time-off requests and parses their content to extract all the necessary information to create a time-off request.

Dynamic Agent Plans

Copilot Studio allow AI agents to create and adjust execution strategies on the fly based on real-time data and evolving conditions. Instead of following a fixed workflow, these agents analyze the situation, determine the best course of action, and adapt as needed.

So depending on the use case on the instructions they will do one thing or another for example let’s see a TimeWarden example:

• An employee requests time off.
• The agent checks sprint schedules and workload balance dynamically.
• If the request fits within capacity, it auto-approves it.
• If it risks project delays, the agent escalates it to a manager for review.
• If a manager rejects the request, the agent suggests alternative dates based on team availability.

Activity

When testing your agent Copilot Studio provides an Activity Map for every session, offering a visual representation of an agent’s sequence of inputs, decisions, and outputs. This helps in identifying issues and improving workflows.

Users can also access the Activity page to review:

• Session details
• Step-by-step activities within a session

Actions

Actions in Copilot Studio are pre-built functionalities that can be used to extend and automate workflows. They allow you to perform specific tasks, such as generating code, making API calls, manipulating data, or integrating with other services, all through a simple interface.

There are different type of actions and hundreds which could be used, but the ones used by TimeWarden are custom Power Automate flows (except for the Send Mail action)

Power Automate

As mentioned before, TimeWarden’s actions use Power Automate flows, as well as other features from Power Automate like custom connectors, AI Hub, and custom prompts, which we will explain later. This integration allows Copilot Studio to automate tasks, connect with external services, and enhance AI-driven workflows, enabling Copilots to fetch data, update records, and execute complex actions seamlessly.

Custom Connectors

A Custom Connector in Power Automate allows you to create your own connectors to integrate services or APIs that aren’t already available in Power Automate’s built-in connectors.

Since the Jira connector didn’t provide the needed actions needed and BambooHR connector didn’t exist, 2 custom connectors were created with the needed actions to integrate those APIs with the flows.

Flows

A flow in Power Automate is an automated process that connects different apps and services to perform tasks without manual intervention. There are different flows but the ones used are Cloud Flows which connect TimeWarden Copilot through events which map to those Copilot actions, properly specifying input and output parameters (if specified).

Approvals

TimeWarden uses Microsoft Teams Approvals (powered by Power Platform), allowing users to create, manage, and track approval requests directly within Teams.

Depending on the scenario, a manager approval may be required to create a time-off request. In this case, the manager will receive a Teams notification containing all the relevant details provided by the user, allowing them to approve or reject the request.

Multi Agent

One of the challenges with the initial version of TimeWarden was that it attempted to handle all tasks within the Instructions prompt. This approach proved ineffective in certain cases, such as categorizing emails or identifying missing fields. The solution was to introduce specialized agents dedicated to specific tasks, working together under the coordination of TimeWarden, which serves as the main orchestrator. This led to the creation of three additional agents:

• Time Off Request Categorizer
• Email Time Off field extractor
• Sprint classifier

All these agents are available in the AI Hub of Power Automate, a centralized platform that allows users to integrate AI-driven capabilities into their automation workflows. It offers access to pre-built AI models, custom AI models, and Azure OpenAI services.

Time Off Request Categorizer

Below is a picture displaying the actual prompt. This prompt essentially analyzes a text (such as an email) and classifies it into various categories, including Vacation, Sick, etc. The prompt’s response is provided in natural language.

In the next picture, you can see that it is activated within the email trigger flow, and its output is directed to the TimeWarden Copilot.

Email Time Off field extractor

As we can see in the following prompt image, this agent analyzes an email to determine if any fields are missing, like Start Date or End Date. Additionally, it takes into account whether a Time off Type parameter is specified, without needing to check if that field is missing.

As before, it is triggered within the email flow, and its output is sent to TimeWarden Copilot.

Sprint classifier

Another agent with an interesting functionality classifies a sprint as current, future, or outside based on a JSON containing all sprint data, along with the start and end dates of a time-off request.

The following image depicts a simple flow triggered by a Copilot. The Sprint Classifier agent is invoked with parameters received from the Copilot, and its output is then sent back to TimeWarden.

Lessons Learnt

During the development apart from learning about the technologies some lessons applying to LLM and AI were learnt.

• Attempting to perform specialized actions, like recognizing whether an email was a time-off request with established categories, didn’t work as expected. This led to the creation of additional agents (prompts) that specialized in these tasks, enabling them to be successfully completed.
• Prompt engineering provided the opportunity to explore and evaluate other LLMs, such as DeepSeek, Qwen, Copilot, and OpenAI. I leveraged them to help describe instructions for Copilot when the initial instructions were unclear or too vague for Copilot Studio to understand.
• Even though Copilot Studio allows up to 8,000 characters for instructions, TimeWarden was limited to 4,400 characters, not even accounting for the prompts of other agents. This made me realize that for complex projects, instructions must be broken down into many smaller agents. Despite that, it still feels like not enough input space for a complex system. I hope Microsoft will increase this limit in the future.
• Copilot Studio requires a high level of clarity for certain tasks, such as forcing it to stop the entire process or not execute specific parts of the instructions (i.e., conditional branches).
• As Copilot Studio is still in preview, some tasks required workarounds, like disabling topics, as they should never be invoked by an autonomous agent.

Conclusions

TimeWarden demonstrates the transformative potential of integrating autonomous agents into everyday business processes. By leveraging Copilot Studio and some specialized agents, TimeWarden automates time-off request management. Throughout this project, several important insights have come to light:

1. Multi-Agent Collaboration: Breaking down complex tasks into specialized agents not only improved performance but also underscored the importance of a modular approach in designing scalable AI systems.
2. Prompt Engineering Mastery: The iterative refinement of prompts was crucial for aligning AI outputs with specific business logic. This process highlighted both the power and the current limitations of AI-driven instructions, prompting continuous enhancements.
3. Scalability and Flexibility Challenges: Although Copilot Studio provides powerful automation features, its limitation contraints and the requirement for explicit control in workflows pose challenges that need to be overcome to scale more complex solutions.

Overall, TimeWarden stands as a compelling proof of concept that not only streamlines time-off management but also lays the groundwork for next-generation AI-powered business automation. Additionally, the continuous evolution of Copilot Studio and autonomous agents is likely to inspire further innovation in how organizations harness AI to drive measurable business value.

Source code

You can find all the source code with a detailed install instructions on:

GitHub - zodraz/TimeWarden: A multi-agent Copilot