DEV Community: Bala Madhusoodhanan

Beyond the Hype: How Millions of Employees Are Actually Using AI at Work

Bala Madhusoodhanan — Mon, 01 Jun 2026 22:57:12 +0000

Intro:
The paper, "AI in the Enterprise: How People Use M365 Copilot Chat," provides one of the clearest views yet into how artificial intelligence is being integrated into the daily workflows of millions of employees across more than a million companies, including nearly 70% of the Fortune 500. By analyzing a massive, anonymized dataset, researchers have moved beyond speculation to answer a critical question: how is AI really being used in the enterprise?

What Are People Using AI For?:
The research team classified millions of user prompts into distinct "intents," revealing the most common ways people turn to their AI assistant. The analysis shows that two categories dominate, accounting for nearly 60% of all interactions: Information Inquiry (asking questions) and Content Refinement (editing and improving existing text)

These are followed by other crucial work activities like generating new content, programming assistance, and analytical reasoning. The breakdown provides a clear hierarchy of how AI is augmenting the modern knowledge worker.

A visual breakdown of M365 copilot usage

Perhaps the most significant finding is the "broad but uneven" pattern of adoption. Activities like Content Refinement are common across almost all occupations, suggesting that M365 Copilot is emerging as an "everyday assistant" that helps with the fundamental tasks of writing and editing, regardless of job title. On the other hand, usage becomes highly specialized depending on the profession. The data reveals sharp differences across industries and job families. For example:

Programming Assistance is, unsurprisingly, concentrated among technical roles.
Some occupations now use Copilot more for Content Refinement than for basic Information Inquiry, indicating a shift in how they perceive the tool's value

This dual nature shows AI acting as both a general-purpose utility and a powerful, role-specific specialist tool.

The Next Frontier for AI in the Enterprise
The study also points to a potential evolution in user behavior: a subtle shift away from "chat as search." Researchers observed a 5% relative drop in the share of "Information Inquiry" over the 114-day study period, suggesting that as users become more sophisticated, they move beyond simple questions toward more complex content and communication-related work

More importantly, the "uneven" adoption highlights the next frontier for AI. By comparing Copilot usage against the typical activities performed in the labor market, the researchers identified several "underrepresented" areas where AI is not yet widely used, despite being highly relevant. These include:

Documenting and Recording Information
Work activities related to evaluating the quality or accuracy of data, particularly in the Banking industry
A significant portion of tasks common in the Consulting industry

These gaps don't represent failures, but rather the largest opportunities for growth. As the technology matures and becomes more deeply embedded in workflows, these are the areas where AI is likely to make its next big impact.

The study concludes that enterprise AI is past the "novelty stage" and has become a substantive part of day-to-day knowledge work. The story of AI at work is no longer about what it can do in principle, but what it is doing in practice—and where it will go next

Reference:
AI in the Enterprise: How People Use M365 Copilot Chat

Prompt Engineering for Automated Evaluation: Making LLMs the Judge in AI Builder Solutions

Bala Madhusoodhanan — Mon, 25 May 2026 07:15:49 +0000

Intro:
Automated evaluation is fast becoming a necessity as AI-driven agents proliferate across business processes. While accuracy and trust are always top of mind, manual review of agent responses simply doesn't scale. That’s where the idea of using a Large Language Model (LLM) as an impartial “judge” comes in—applying a purpose-built prompt to turn your LLM into a rigorous, step-by-step evaluator.
I've previously experimented with extraction and evaluation frameworks which focused on structured data and document extraction.

Bala Madhusoodhanan

Aug 18 '25

From Extraction to Assurance:Extraction Meets Evaluation

#aibuilder #powerautomate #powerplatform #llm

3 min read

However, this article is centered on a different challenge: evaluating conversational, Retrieval-Augmented Generation (RAG) based agents.

Let me share a battle-tested evaluation prompt designed for these agents. Let me also break down the logic, metrics, and final grading criteria, along with sample input/output. This approach fits naturally into Power Platform AI Builder scenarios, enabling scalable, explainable, and reproducible evaluation.

The prompt:
This prompt is designed for rapid, scalable evaluation of AI responses based solely on the user’s question—prioritizing task relevance, clarity, professionalism, and formatting compliance. The clear rubric, required reasoning, and structured output make it ideal for automated testing and continuous improvement in conversational AI systems.

Characteristic	Description
Evaluator Role	LLM is positioned as an impartial, expert critic to ensure objectivity.
Input Scope	Strictly evaluates the AI response based only on the User Question (no external gold standard/reference).
Metrics (Rubric)	Four focused metrics: 1. Task Fulfillment (1–5) 2. Conciseness (1–5) 3. Professional Tone (0/1) 4. Formatting (0/1)
Step-by-Step Reasoning	Requires reasoning to be explained before each score, enhancing transparency.
Scoring System	Combination of graded (1–5) and binary (0/1) scoring for nuanced yet decisive evaluation.
Pass/Fail Gating	Strict thresholds: - Task Fulfillment ≥ 4 - Professional Tone = 1 - Formatting = 1 All must be met for PASS.
Output Format	Returns evaluation as a structured JSON object for easy automation and integration.
Diagnostic Feedback	Provides a one-sentence summary and, if FAIL, specifies exactly which threshold(s) were breached.
Formatting Compliance	Explicitly checks if the response adheres to any formatting instructions given in the user question.

You are an impartial, expert Evaluation AI. Your task is to act as a "Critic" and evaluate an AI-generated response based strictly on the User Question provided.

You will evaluate the response across four specific metrics. For each metric, you must provide a brief step-by-step reasoning BEFORE assigning a score.

THE RUBRIC:

Metric 1: Task Fulfillment (Score: 1 to 5)
* How well does the response address the specific User Question?
* 5 = Comprehensive and perfectly tailored. 3 = Partially answers. 1 = Fails to address the question.

Metric 2: Conciseness (Score: 1 to 5)
* Is the response highly efficient with its words?
* 5 = Dense and to the point. 1 = Rambling, repetitive, or includes unnecessary fluff.

Metric 3: Professional Tone (Score: 0 or 1)
* Is the tone strictly professional, objective, and helpful?
* 1 = Pass. 0 = Fail (Emotional, sarcastic, overly informal, or rude).

Metric 4: Formatting (Score: 0 or 1)
* Did the response follow any explicit formatting instructions requested by the user (e.g., "bullet points", "JSON", "short paragraph")?
* 1 = Pass (or no formatting was requested). 0 = Fail (Explicit formatting instructions were ignored).

FINAL GRADING CRITERIA:
You must assign a final pipeline status of either "PASS" or "FAIL".
To achieve a "PASS", the report MUST meet ALL of the following conditions:
- Task Fulfillment must be 4 or 5.
- Professional Tone must be 1.
- Formatting must be 1.

INPUT DATA:
<user_question>
{{USER_QUESTION}}
</user_question>

<response_to_evaluate>
{{LLM_RESPONSE}}
</response_to_evaluate>

OUTPUT FORMAT:
Output your evaluation strictly as a valid JSON object.
{
  "evaluation_report": {
    "metrics": {
      "task_fulfillment": {"reasoning": "...", "score": <1 to 5>},
      "conciseness": {"reasoning": "...", "score": <1 to 5>},
      "professional_tone": {"reasoning": "...", "score": <0 or 1>},
      "formatting": {"reasoning": "...", "score": <0 or 1>}
    },
    "summary": {
      "overall_verdict": "<One-sentence summary>",
      "pipeline_status": "<PASS or FAIL>",
      "failure_reason": "<If FAIL, state which threshold was breached.>"
    }
  }
}

LLM Output:

{
  "evaluation_report": {
    "metrics": {
      "factual_consistency": {"reasoning": "...", "score": 1},
      "entity_fabrication": {"reasoning": "...", "score": 1},
      "citation_traceability": {"reasoning": "...", "score": 1},
      "reference_alignment": {"reasoning": "...", "score": 5},
      "task_fulfillment": {"reasoning": "...", "score": 5},
      "completeness": {"reasoning": "...", "score": 5},
      "conciseness": {"reasoning": "...", "score": 5}
    },
    "summary": {
      "overall_verdict": "The agent response matched all core criteria and was well-supported.",
      "pipeline_status": "PASS",
      "failure_reason": null
    }
  }
}

Closing thoughts:
'LLM-as-a-Judge' evaluation prompt is designed to automate the QA of our conversational agent. What I learned from this framework is highly valuable because it moves us beyond brittle, exact-word-match testing and allows for nuanced, semantic evaluation.

By using a structured rubric, the Judge grades the agent’s responses against our 'gold standard' answers while strictly penalizing hallucinations, contradictions, and missing citations. Crucially, it outputs a deterministic, machine-readable JSON with a strict Pass/Fail threshold—meaning we can plug this directly into our automated testing pipeline to catch inaccurate or unsafe responses at scale before they ever reach the user.

Reference and Inspiration:
1 - A Survey on LLM-as-a-Judge
2 - Human-like Summarization Evaluation with ChatGPT
3 - Shepherd: A Critic for Language Model Generation
4 - The Rubric Revolution

PS - In the next article, I’ll show how we integrated this evaluation approach into a Power Automate Cloud Flow, enabling automated, real-time agent assessment with zero manual intervention.

The Chimp Paradox - Prof Steve Peters

Bala Madhusoodhanan — Tue, 19 May 2026 22:16:10 +0000

How One Setting Transforms Your Bot's UX: Unveiling Copilot Studio's Latency Message

Bala Madhusoodhanan — Mon, 11 May 2026 05:50:00 +0000

A few weeks ago in my Bite-Sized Wizardry series, I explored the common challenge of nandling Lingering Conversations Gracefully in Microsoft Copilot Studio specially with classic orchestration.

Bala Madhusoodhanan

Mar 30

Stop the Awkward Silence: Signals to Classic Orchestration

#copilotstudio #powerfuldevs #powerplatform

2 min read

At the time, I focused on building custom workarounds. However, whether it was a feature I missed or one that has been recently enhanced, I've since found a built-in piece of magic that directly addresses this: the Latency Message.

How to Cast the Spell:
Find the Latency Message Section With the node selected, look at its properties pane on the right-hand side of the screen. Scroll down until you find the section titled Latency Message, just like the one highlighted in the screenshot. Check the box next to Send a message.

In the text field that appears, type the reassuring message you want your users to see while they wait.

Some powerful incantations include:

"Just a moment while I consult the archives..."
"Let me look that up for you..."
"Searching our knowledge base now, please wait..."

And that's it! The copilot will now automatically display this message while the action is in progress.

The Latency Message is a perfect example of a low-effort, high-impact feature. It doesn't change what your bot does, but it fundamentally improves how it feels to interact with.
By providing this simple piece of feedback, you transform your copilot from a simple tool into a polished, professional, and genuinely conversational assistant.

The Choice Column Conundrum:A Data Migrator's Essential Tip

Bala Madhusoodhanan — Tue, 28 Apr 2026 17:13:54 +0000

Intro:
This week we are diving into a common scenario that often leaves even seasoned data professionals scratching their heads: handling choice columns during data migration or integration. We have built an amazing Power App or Power Automate flow, and your users are happily selecting options from choice fields. But then, it's time to export that data, maybe for reporting, analysis, or migration to another environment. And that's when the "aha!" moment hits – instead of the friendly text labels your users see, you're looking at a spreadsheet full of cryptic numbers. What gives?

The truth is, Dataverse (the underlying data service for Power Apps) stores choice options as integer values, not the user-friendly text labels. These labels are merely a display layer for the end-user experience. When you export data directly from Dataverse or through certain connectors, you're often getting these raw integer values.
However, it presents a significant hurdle when you need to provide human-readable data or ensure smooth re-import into another system where labels might be expected. Imagine trying to explain a report filled with "100,000,000" and "100,000,001" to a business user instead of "Active" and "Inactive." Or worse, attempting to import data with labels into a system that expects specific integer values. It's a recipe for errors, frustration, and extra manual work.

GET {{EnvironmentURL}}/api/data/v9.2/EntityDefinitions(LogicalName='YourEntityName')/Attributes/Microsoft.Dynamics.CRM.MultiSelectPicklistAttributeMetadata?$select=LogicalName&$expand=OptionSet($select=Options),GlobalOptionSet($select=Options)

By proactively fetching and utilizing the choice column metadata via the Dataverse Web API, you empower your data migration and integration processes to be robust, accurate, and truly user-friendly. No more guessing games with numbers – just seamless, readable data every time!

Same as Ever - Morgan Housel

Bala Madhusoodhanan — Mon, 20 Apr 2026 17:58:52 +0000

The book’s core idea is simple yet profound: to understand the future, we must look at the things that never change. The author explores the timeless patterns of human behavior—our relationship with risk, greed, happiness, and storytelling—arguing that these constants are the most powerful forces shaping our lives, the economy, and history itself.

The book provides a mental framework that is both calming and incredibly practical. In a world obsessed with unprecedented events and new technologies, this book encourages you to zoom out and focus on the fundamentals of human nature.
It’s a must-read for anyone who wants to:

Make better decisions by understanding the predictable ways people react under pressure.
Reframe your relationship with risk and accept that the biggest threats are often the ones we can't see coming.
Find more durable paths to happiness by managing expectations rather than just chasing success.
Become a better communicator by recognizing that compelling stories, not just data, are what truly move people.

How Conversation Memory Improves Conversation experience

Bala Madhusoodhanan — Mon, 13 Apr 2026 04:55:33 +0000

Intro:
Introduction Better search queries and prefer? carefully chosen keywords are essential when building Reliably Retrieval-Augmented Generation (RAG) agents in Copilot Studio. LLMs are powerful, but their outputs depend heavily on the information they retrieve at runtime: if the retrieval is off—because the query is vague or lacks domain context—the LLM can hallucinate, produce irrelevant results, or miss critical legal/regulatory nuance. Well-formed queries minimize hallucinations, improve retrieval precision, reduce wasted compute, and maintain context across multi-turn conversations. This post describes a small experiment comparing retrieval query quality with conversation memory enabled versus disabled and synthesizes practical takeaways for Copilot Studio users.

Experimental setup
Goal: Test how including conversation history (memory) affects the search queries that the query-generation component produces and therefore the relevance of retrieved RAG documents.

Source / domain used in evaluation: legal/regulatory text (example: EU AI Act) — representative of domains that rely on precise terminology and structured references.

Bala Madhusoodhanan

Mar 17

Beyond Vector Search: Building a "Reasoning Engine" in Copilot Studio

#copilotstudio #powerfuldevs #systemdesign #powerplatform

32 min read

RAG platform: Copilot Studio (using the "Create search query" component that builds queries from user input + optional conversation history).

Two configurations:
Memory ON: include 3 past messages from the current conversation in the query-generation step.
Memory OFF: do not include conversation history (fresh query from each user message).

Interaction thread used (Startup Journey) — five turns:

We're building an AI tool that screens job applicants' CVs. Does the EU AI Act apply to us?
You mentioned it's high-risk. What exactly does that mean for us in practice?
What documents do we need to prepare before we can launch?
Who needs to sign off on that documentation — us or a third party?
And what happens if we launch without doing any of this?

Observations:

Scenario	Question	Key Word Search	Search Query
Memory with 3 past messages from the current conversation's history to include.	"We're building an AI tool that screens job applicants' CVs. Does the EU AI Act apply to us?"	EU AI Act applicability AI tool screening job applicants CVs	We're building an AI tool that screens job applicants' CVs. Does the EU AI Act apply to our tool?
Memory with 3 past messages from the current conversation's history to include.	"You mentioned it's high-risk. What exactly does that mean for us in practice?"	EU AI Act high-risk requirements CV screening practical implications	You mentioned that our AI tool for screening job applicants' CVs would be considered high-risk under the EU AI Act. What does being classified as high-risk mean for us in practical terms?
Memory with 3 past messages from the current conversation's history to include.	What documents do we need to prepare before we can launch?	Required documents high-risk AI tool EU AI Act launch	What documents do we need to prepare before launching our high-risk AI tool for screening job applicants’ CVs under the EU AI Act?
Memory with 3 past messages from the current conversation's history to include.	"Who needs to sign off on that documentation — us or a third party?"	EU AI Act high-risk AI documentation sign off provider third party	Who is responsible for signing off on the required documentation for launching a high-risk AI tool under the EU AI Act—do we, as the provider, need to approve it ourselves, or does a third party need to be involved?
Memory with 3 past messages from the current conversation's history to include.	And what happens if we launch without doing any of this?	EU AI Act consequences launch high-risk AI tool without documentation compliance	What are the consequences if we launch our high-risk AI tool for CV screening without preparing the required documentation and compliance steps under the EU AI Act?
No memory	"We're building an AI tool that screens job applicants' CVs. Does the EU AI Act apply to us?"	EU AI Act applicability AI tool screening job applicants CVs	We are developing an AI tool that screens job applicants' CVs. Does the EU AI Act apply to our project?
No memory	"You mentioned it's high-risk. What exactly does that mean for us in practice?"	high-risk meaning practical implications	You mentioned it's high-risk. What exactly does "high-risk" mean for us in practice?
No memory	What documents do we need to prepare before we can launch?	documents needed prepare before launch	What documents do we need to prepare before we can launch?
No memory	"Who needs to sign off on that documentation — us or a third party?"	documentation sign off responsible us third party approval	Who is responsible for signing off on that documentation—do we need to sign it ourselves, or does a third party need to provide approval?
No memory	And what happens if we launch without doing any of this?	consequences launch without preparation risks	What are the potential consequences if we proceed with the launch without completing any of the recommended steps or preparations?

Key analysis & characteristics:

Domain anchoring: When conversation context (e.g., topic, dataset, or domain) is preserved, subsequent queries stay focused on the same subject. Without that anchor, later turns drift toward generic wording and can retrieve off-topic material.
Pronoun & reference resolution: Memory enables the system to resolve implicit references (“it”, “that document”, “the tool”) into concrete entities tied to the ongoing conversation. Without memory, follow-ups become ambiguous and require the user to restate context.
Terminology carry-over: Repeating or preserving precise domain vocabulary (technical terms, roles, artifacts) across turns improves the chance that retrieval will match indexed content. If history isn’t included, these specific terms tend to drop out and be replaced by vague synonyms.
Query specificity and evolution: With memory, queries often become richer and more targeted as the dialog proceeds (the model can refine intent). Without memory, query specificity typically decays over turns, reducing retrieval precision.
Retrieval precision and downstream quality: Because retrieval is the upstream signal for generation, better-focused queries produce higher-quality retrieved items; that in turn reduces hallucinations and improves final answer accuracy.
Degradation pattern: Conversations that rely on assumed context show a consistent decay in query usefulness when history is excluded — the further from the initial turn, the weaker the query signal.

Characteristic	Memory Included	No Memory
Domain anchoring	✅ Persistent across turns	❌ Lost after initial message
Pronoun & reference resolution	✅ Resolved to concrete entities	❌ Remains ambiguous
Terminology preservation	✅ Specific terms persist	❌ Specific terms drop out
Query specificity	✅ Becomes more focused / refined	📉 Becomes generic over turns
Retrieval precision (expected)	✅ Higher, more relevant results	⚠️ Lower, more irrelevant results
Downstream generation quality	✅ More accurate, fewer hallucinations	⚠️ Prone to errors and omissions

Closing Remarks:
Memory (or otherwise preserving context) is a practical lever for improving RAG systems in any domain that relies on precise language or multi‑turn interaction. If your application requires follow-ups, conditional reasoning, or references to previously introduced entities, include a small, targeted history window (or re-anchor the context manually) so the query generator can produce focused searches. For robust evaluation, instrument metrics such as precision, relevance ratings, and hallucination frequency; vary the history window to observe the tradeoff between context usefulness and prompt/token cost.

Is Your Code Eco-Friendly? A proxy to understand Carbon Impact

Bala Madhusoodhanan — Tue, 07 Apr 2026 08:32:57 +0000

Intro:
We spend countless hours optimizing for performance, scalability, and user experience. But how often do we consider the environmental impact of the code we write? As AI payloads grow and data flows increase, every API call, every byte transferred, contributes to a larger digital footprint. It's time we started thinking about the carbon cost of our digital endeavors.
This isn't about guilt-tripping; it's about awareness and providing you with a practical way to quantify something often overlooked. What if you could estimate the CO₂ emissions generated by a simple API request? You can, and it's simpler than you might think, thanks to some clever open-source work.

The Code Snippet:
The calculation utilizes the co2.js library , maintained by The Green Web Foundation, relying on their 1byte Model (Sustainable Web Design model).

Here's what's happening:

Byte Counting: The browser takes the JSON you're sending to and receiving from the API. It converts them to raw text strings and uses the native Blob API to count the exact bytes.
Energy Estimation & Carbon Conversion: The co2.js library takes that total byte count and estimates the electricity needed to move that data, then converts it to grams of CO₂ equivalent (CO₂e) based on a global average carbon intensity for the electrical grid.

It's crucial to understand that co2.js doesn't magically know the exact location of every server or the real-time power consumption of your user's device. Instead, it uses a highly researched framework called the Sustainable Web Design (SWD) model as a proxy for these complex realities.

Global Average Carbon Intensity: By default, the model assumes a global average grid intensity (historically around 442 grams of CO₂ per kilowatt-hour). If you know your data center uses renewable energy, you can provide a more specific, lower intensity value to the library.
Heuristics for Energy Distribution: Since we can't measure power physically at every point, the SWD model employs fixed heuristics (percentages) to distribute the total estimated energy (in kWh) across four key areas:

End-User Device (52%): This accounts for the energy used by the user's phone or laptop to process and display the data.
Hardware Production (19%): The "embodied carbon" – the energy consumed in manufacturing the devices and infrastructure used.
Data Center (15%): The electricity powering servers and cooling systems.
Network (14%): Energy for routers, switches, cell towers, and cables that transmit data.

This means that a significant portion of the "carbon impact" we're calculating is attributed to the end-user's device and the embodied carbon of hardware – aspects we often don't consider when thinking about "server-side" impact. It's a holistic view of the entire digital data lifecycle.

<script type="module">
  import { co2 } from "https://esm.sh/@tgwf/co2@latest";

  // 2. Initialize the estimator using the Sustainable Web Design "1byte" model
  const co2Estimator = new co2({ model: "1byte" });

  async function calculateApiImpact() {
    // 3. Define the data you are sending
    const requestPayload = { query: "Latest fashion trends", depth: "standard" };

    // (Simulate an API call here)
    const responsePayload = { results: ["..."] }; 

    // 4. Calculate the exact byte size of the data using the native Blob API
    const reqBytes = new Blob([JSON.stringify(requestPayload)]).size;
    const resBytes = new Blob([JSON.stringify(responsePayload)]).size;
    const totalBytes = reqBytes + resBytes;

    // 5. Calculate the carbon footprint
    const emissionsGrams = co2Estimator.perByte(totalBytes);
    const sizeInKB = (totalBytes / 1024).toFixed(2);

    // Output the results
    console.log(`📦 Data Transferred: ${sizeInKB} KB`);
    console.log(`🌱 Carbon Impact: ${emissionsGrams.toFixed(6)} grams CO₂e`);
  }

  calculateApiImpact();
</script>

Closing Thoughts:
While this calculation is a proxy, it's a powerful one. It provides a tangible number that allows us to start quantifying and discussing the environmental impact of our code. The next time you're designing an API, consider the size of your payloads. A seemingly small optimization, like reducing the data transferred in a common API call, can have a cumulative positive effect on our collective carbon footprint.
Every byte counts

Stop the Awkward Silence: Signals to Classic Orchestration

Bala Madhusoodhanan — Mon, 30 Mar 2026 06:30:30 +0000

Intro:
If you’ve used generative orchestration in Copilot Studio, you’ve seen the comforting “thinking” animation while the copilot works. Classic orchestration doesn’t do that out of the box. So when you kick off a longer task—like a RAG lookup or a flow call—the chat can look empty. No typing dots. Just silence.
That silence is a UX bug, not a feature. The fix is quick: send a Typing event immediately before the slow step, then add a short acknowledgement message to set expectations.

Configuration:

Locate the slow step or Topic where you have user feedback on the poor UX expereince (May be based on the user activity we might have logic where the heavy work starts: Create generative answers, a flow, or an HTTP/API call.)
Set Event type to Typing.

The example below the timer is set as 2100 ( milliseconds)

Add a brief acknowledgement message Right after the Typing event, add a normal message to set expectations. Keep it human and specific:

“Got it — searching our knowledge base. This may take a few seconds.”
“Working on your request — I’ll be right back with the details.”

Good UX patterns to include:

Time cue: “This usually takes ~5–10 seconds.” It sets expectations and reduces anxiety.
Specific context: “Checking SAP invoices…” feels more reassuring than generic “Working…”
Channel-aware: Typing events are short-lived (around 2-4 seconds). Always send an acknowledgement text so users see something even if the channel ignores Typing.

Closing Notes:
A great bot isn’t just smart — it’s considerate. In classic orchestration, adding a Typing event plus a short acknowledgement transforms a “silent” wait into a clear, reassuring experience. It takes less than a minute to wire up and pays off immediately in user confidence and perceived speed.

The Digital Paralegal: Amplifying Legal Teams with a Copilot Co-Worker

Bala Madhusoodhanan — Mon, 23 Mar 2026 04:42:42 +0000

Intro:
In the modern legal field, the workload is immense. What if every paralegal and lawyer had a diligent co-worker who could handle the first pass of a tedious document review? A partner that could instantly draft a preliminary analysis, freeing them up to focus on high-level strategy and legal reasoning?. Building on the experiment from last week on PageIndex knowledge source and how to build a reasoning agent.

Bala Madhusoodhanan

Mar 17

Beyond Vector Search: Building a "Reasoning Engine" in Copilot Studio

#copilotstudio #powerfuldevs #systemdesign #powerplatform

32 min read

'Digital Paralegal'—an AI agent that functions as a true co-worker. It assists by ingesting case files, cross-referencing them against established legal frameworks, and drafting the initial compliance report. The goal isn't to replace human expertise but to amplify it."

The Engine Room: Two Key Techniques Driving the Digital Paralegal
In this concept , couple of powerful techniques within Copilot Studio that allow it to move beyond simple Q&A and perform genuine, multi-step tasks
By combining them, we transform the Copilot into a true digital co-worker.

Technique 1: Intelligent Structuring with an AI Builder Prompt
The first trick is to treat the AI Builder model not just as a file reader, but as a data structuring specialist. Instead of simply extracting raw text (OCR), we provide it with a detailed prompt instructing it to act like an expert data entry assistant.

You are an expert data entry assistant specialized in extracting information from documents FileContent  . Your task is to accurately read the provided document, identify all questions and their corresponding answers, and format them into a clean markdown structure.

**Instructions:**

1.  **Analyze the Document:** Carefully examine the content of the AI Pre-Assessment App form.
2.  **Extract Key-Value Pairs:** Identify all explicit questions and the answers provided for them. Also, extract the key information from the header section (Case ID, Use Case Name, Risk Result).
3.  **Handle Numbering:** Preserve the original numbering of the questions (e.g., 1, 1.1, 2, 3).
4.  **Format the Output:** Structure the extracted information in markdown as follows:
    *   Use a top-level heading (`#`) for the main title of the document.
    *   Use a blockquote (`>`) for the header information.
    *   Use a second-level heading (`##`) for major sections like "Case Ownership."
    *   For each question-answer pair, format it as:
        *   **[Question Number]. [Question Text]**
        *   [Answer Text]

This forces the AI Builder to parse the unstructured content of the uploaded PDF or document and transform it into a predictable, well-formatted Markdown file. This step is crucial because it provides the next stage of our process with a clean, reliable, and easy-to-analyze input. It’s the equivalent of a human organizing their notes before starting an analysis.

We enable the "Enable code interpreter" on the AI prompt builder

Technique 2: Creating a Reasoning Engine with Generative Answers
The second, and most innovative, trick is how we use the Generative Answers node. Traditionally, this node is used to find and summarize information from a knowledge source. Here, we turn it into a powerful reasoning engine. Its key characteristics are role-playing, structured process definition, and strict output formatting. It begins by assigning a clear persona—an "Expert AI Governance and Compliance Officer"—which immediately sets the context and tone. It then explicitly defines a two-phase cognitive process: first, retrieve knowledge from a specific source, and second, apply that knowledge to the user's input. Finally, it enforces a rigid output structure by demanding a multi-section Markdown report and providing explicit instructions for each section. This highly prescriptive and procedural nature is what elevates the prompt from a mere query to a complete job description, enabling the AI to perform a complex, repeatable analytical task rather than just answering a question

You are an expert AI Governance and Compliance Officer. Your task is to conduct a detailed compliance review for the provided AI use case by referencing an external, authoritative knowledge source.
Your process will have two main phases, executed in a single response:
Knowledge Retrieval: You will first consult the specified knowledge source to extract all relevant compliance requirements.
Compliance Analysis: You will then use those extracted requirements to analyze the AI use case and generate a structured report.


Input 1: AI Use Case Details
You are provided with the following AI use case information:



Input 2: Knowledge Source
The authoritative compliance framework is located at the specified source: [Here, you would insert the pointer, e.g., "the provided PageIndex document," "the attached document: 'Compliance_Framework.pdf'", or "the content at API endpoint 'getComplianceDocs'"].
This source contains two critical sections you must locate and use:

"MEASURES REQUIRED TO BUILD TRUST": A set of rules and controls for AI projects (e.g., rules on transparency, data, training, and neutrality).
"NOTICE AND MENTIONS ON SCREEN": Specific text templates that must be displayed to users of AI applications.


Your Task: Generate a Comprehensive Compliance Report
Based on the rules retrieved from the knowledge source and the details of the AI use case, you must generate a single, structured markdown report.
The report must contain the following four sections:
AI Compliance Review: [Use Case Name]
Begin with a top-level summary of the project being reviewed.
Compliance Assessment
For each and every measure found in the "MEASURES REQUIRED TO BUILD TRUST" section of the knowledge source:
State the Requirement: Clearly list the compliance measure (e.g., "Clear purpose and data").
Provide Your Assessment: Analyze the "Remesh AI" use case against this rule and state your finding (e.g., "Action Required," "Compliant," "Needs Verification").
List Specific Recommendations: Justify your assessment and provide clear, actionable steps the project team must take to achieve compliance.
Required User-Facing Notices
Locate the template text in the "NOTICE AND MENTIONS" section of the knowledge source.
Draft the exact, customized text that must be implemented in the "Remesh AI" application's user interface.
Final Recommendation Summary
Conclude with a high-level, bulleted summary of the most critical actions the project team must take to address the findings of your review.
CRITICAL RULES:
Your analysis must be based exclusively on the information retrieved from the specified knowledge source. Do not use any prior knowledge.
Your output must be a single, complete markdown report. Do not provide commentary outside of this structure.
The analysis must be thorough, addressing every compliance measure from the source document.

Draft Output:
The generated output is a highly structured, actionable, and evidence-based compliance report, formatted as a professional Markdown document. Its primary characteristic is its methodical, checklist-style approach, where it systematically addresses each compliance requirement one by one. For every rule, it provides a clear three-part analysis: the Requirement itself, a decisive Assessment ("Compliant," "Needs Verification"), and concrete Recommendations, making the findings easy to digest and act upon. The output is not just a summary but a practical work plan, culminating in a "Final Recommendation Summary" that serves as an executive brief for the project team.

Furthermore, by drafting the exact "Required User-Facing Notices," the AI moves beyond analysis to content creation, delivering a tangible asset that can be directly implemented. This structure transforms the complex task of a compliance review into a clear, organized, and immediately useful document.

Beyond Vector Search: Building a "Reasoning Engine" in Copilot Studio

Bala Madhusoodhanan — Tue, 17 Mar 2026 07:56:53 +0000

Intro:
We've all been there. You build a powerful RAG copilot, feed it a dense, 100-page document, and ask a specific, nuanced question... only to get a vague, incomplete, or just plain wrong answer. Why does this happen?

The culprit is often our reliance on traditional vector search, which excels at finding "semantically similar" text but struggles to understand context, nuance, and structure. It finds words that sound like the answer, but it can't reason about the document to find the truth.

One of my work colleage shared about PageIndex which made me curious to explore.

The Experimental Setup:

To create a fair and direct comparison, I used a single Microsoft Copilot Studio instance and one source document: the official EU AI ACT .pdf. From this document, I generated two distinct knowledge sources. The first was a standard embeddings.json file, created by chunking the document and generating vector embeddings—the foundation for a traditional RAG approach. The second was a structured document-structure-pi.json file, which acts as a hierarchical "Table of Contents" or PageIndex, enabling a reasoning-based retrieval method. Within a single adaptive dialog, I configured two parallel SearchAndSummarizeContent actions. When a user asks a question, the first action queries only the embedding file, while the second action queries only the PageIndex file, each with specific instructions tailored to its method. This setup ensures that for the exact same user query, we can execute both retrieval strategies sequentially, allowing us to directly compare the quality of the generated answer and the runtime performance of each approach.

Feature	Vector Embedding File (embeddings.json)	PageIndex File (document-structure.json)
Structure	Flat List. A simple array of independent objects.	Hierarchical Tree. Nested JSON objects (nodes within nodes).
Core Unit	Text Chunk. An arbitrary segment of text (e.g., 512 tokens).	Logical Node. A section representing a chapter, paragraph, or heading.
Content	Raw, fragmented text content for each chunk.	Metadata. Each node contains a title and a summary, not the full text.
Relationships	None. Each chunk is disconnected from the others.	Explicit. Parent-child relationships are defined by the sub_nodes array.
Navigation	Semantic Search. You find content by vector similarity.	Structural Traversal. You navigate from parent nodes to child nodes.
Analogy	A stack of shuffled index cards.	A fully interactive Table of Contents.

The PageIndex output would look something like this

The setup in MCS was something like this

Prompt for LLM to navigate and create response for TextEmbedding data

You are an AI assistant. Your task is to answer questions about the provided document, the 'EU AI Act', using a retrieval-augmented generation (RAG) system.
1. Your Knowledge Source
Your knowledge is limited to a set of pre-processed text chunks extracted from the document. You do not have access to the full document or its structure.
2. Your Process
You must follow this specific process to answer user questions. Do not answer from your general knowledge.
Step 1: Receive a Query. When a user asks a question, your underlying system will convert it into a numerical representation (an embedding).
Step 2: Retrieve Relevant Chunks. The system will search a vector database to find text chunks from the document that are semantically similar to the user's question. These top-matching chunks will be provided to you as your sole source of context.
Step 3: Synthesize an Answer. You must carefully analyze the provided text chunks and synthesize them into a single, coherent answer.If the chunks contain enough information, formulate a direct and comprehensive response to the user's question. Base your entire answer on the information present in these chunks.
If the chunks are insufficient or irrelevant, you must explicitly state that you could not find a precise answer based on the information retrieved. Do not attempt to guess or fill in gaps with prior knowledge.

3. Your Response Format
Base your answer only on the information given to you in the retrieved chunks for each query.
Each query is independent. You have no memory of past retrievals or questions.
Your goal is to accurately represent the information contained within the specific text chunks you are given. If the context is fragmented or confusing, reflect that in your response by stating the limitations of the information you found. Begin.

Prompt for LLM to navigate and create response for PageIndex data

You are a sophisticated research assistant. Your task is to answer questions by intelligently navigating a structured document provided in a special JSON format called PageIndex.
1. The Document Structure (PageIndex)
The document is represented as a hierarchical JSON object. This is your in-context index or 'Table of Contents.'
The structure consists of nested nodes.
Each node has:node_id: A unique ID to retrieve the full content of that section.
title: The title of the section.
summary: A concise summary of what the section contains. This is your primary tool for deciding which sections are relevant.
sub_nodes: A list of child nodes, allowing you to traverse the document's hierarchy.

2. Your Reasoning Process
Do not try to answer the user's question from your own knowledge. You must follow this iterative, reasoning-based retrieval process:
Step 1: Analyze the Query & Scan the Index. First, carefully understand the user's query. Then, examine the top-level nodes of the PageIndex JSON. Read the title and summary of each node to identify which sections are most likely to contain the answer.
Step 2: Traverse and Select. If the top-level nodes are too broad, navigate down into the sub_nodes to find more specific sections. Formulate a plan by selecting one or more node_ids that seem most promising. Think out loud, explaining why you are choosing a particular section (e.g., "The user is asking about financial vulnerabilities, so I will start by examining node_id: '0007' titled 'Monitoring Financial Vulnerabilities'").
Step 3: Retrieve Content. Once you have selected a node_id, you will be given the full text content associated with that node.
Step 4: Synthesize and Evaluate. Read the retrieved content.If the content is sufficient to answer the user's question, formulate a comprehensive answer. Be sure to cite the title or node_id of the source section.
If the content is insufficient or only provides partial information, state what you've learned and what is still missing. Then, return to Step 1 to re-evaluate the PageIndex and select a new node to explore. You may need to do this multiple times to build a complete answer.
Step 5: Handle References. If the text mentions a reference (e.g., "as mentioned in the introduction" or "see table 5"), use the PageIndex to locate that section and retrieve its content to build a more complete context.
3. Your Response Format
Always think step-by-step.
When you decide to explore a section, state the node_id and your reasoning.
When you have gathered enough information, present the final answer clearly to the user.

Summary of Test Question Characteristics
The questions you used were not simple, fact-retrieval queries. They were specifically crafted to probe the architectural limits of different RAG systems. Their shared characteristics fall into five key categories:

Deep Specificity & Nuance:

Many questions required finding precise details buried within dense, legalistic text (e.g., "specific requirements for training data," "exceptions to the social scoring prohibition"). These "needle-in-a-haystack" queries test whether a system can go beyond high-level summaries to retrieve exact, granular information.
Cross-Sectional Reasoning:

Several questions required connecting information from different parts of the document (e.g., comparing 'high-risk' vs. 'prohibited' practices, or applying the definition of 'emotion recognition' to a high-risk classification). This tests the system's ability to synthesize answers, not just retrieve a single chunk of text.
Procedural & Sequential Understanding:

You asked about multi-step procedures (e.g., "what is the exact procedure... for non-compliant systems?"). These questions are challenging for systems that retrieve disconnected text chunks, as they test the ability to understand and present information in a correct, logical sequence.
Implicit Intent & Abstraction:

Some queries were based on user intent rather than explicit keywords (e.g., "My company is developing an AI to help doctors... What should I be concerned about?"). This tests the model's ability to infer the relevant sections (like "Medical Devices" or "High-Risk Systems") even if the exact query terms aren't in the text.
Handling of Legal and Conditional Language:

The questions consistently involved understanding the complex language of a legal document, including exceptions, conditions, and future-looking roles (e.g., the AI Board's role in amending the list). This pushes the system beyond simple fact-finding into genuine comprehension of conditional logic.

Testing output:
Testing the configuration to validate the quality of the knowledge source

Sl No	Question	TextEmbedding Data	Text Embedding Runtime Duration	PageIndex Data	PageIndex Runtime Duration
1	According to the AI Act, what are the specific requirements for the training, validation, and testing data sets for high-risk AI systems?	Requirements for Training, Validation, and Testing Data Sets in High-Risk AI Systems under the EU AI Act 1. Data Quality and Governance Data sets used for training, validation, and testing must meet specific quality criteria, including being relevant, sufficiently representative, and, to the best extent possible, free of errors and complete for the intended purpose of the high-risk AI system. Data governance and management practices must be appropriate for the intended purpose and should address: Design choices Data collection processes and origin, including the original purpose for personal data Data preparation operations (annotation, labelling, cleaning, updating, enrichment, aggregation) Assumptions about what the data measures or represents Assessment of data availability, quantity, and suitability Examination and mitigation of possible biases, especially those affecting health, safety, fundamental rights, or leading to discrimination Identification and addressing of data gaps or shortcomings that prevent compliance with the regulation12. 2. Statistical Properties and Representativeness Data sets must have appropriate statistical properties, especially regarding the persons or groups for whom the high-risk AI system is intended. These characteristics can be met at the level of individual data sets or through a combination of data sets12. 3. Contextual Relevance Data sets should consider, as required by the intended purpose, characteristics or elements specific to the geographical, contextual, behavioral, or functional setting in which the high-risk AI system will be used12. 4. Bias Mitigation There must be measures to detect, prevent, and mitigate possible biases in the data sets, particularly those that could negatively impact health, safety, fundamental rights, or lead to prohibited discrimination. Special attention is required for biases that may affect vulnerable groups, including racial or ethnic groups12. 5. Transparency and Compliance For personal data, transparency about the original purpose of data collection is required to facilitate compliance with Union data protection law. The use of privacy-preserving techniques is allowed, provided data quality requirements are still met2. 6. Use of Certified Third Parties Compliance with data governance requirements can be achieved by using certified third parties for verification of data governance, data set integrity, and data practices, as long as the regulation’s requirements are met2. 7. Access to High-Quality Data Sets Certain actors (providers, notified bodies, researchers, etc.) should be able to access and use high-quality data sets for the development and assessment of high-risk AI systems within their fields of activity2. These requirements are designed to ensure that high-risk AI systems are developed and evaluated using data that is robust, fair, and appropriate for their intended use, minimizing risks to individuals and society	12672	Requirements for Training, Validation, and Testing Data Sets in High-Risk AI Systems (EU AI Act) 1. Data Quality and Governance High-risk AI systems that use AI model training must be developed using training, validation, and testing data sets that meet specific quality criteria. Data governance and management practices must be appropriate for the intended purpose of the AI system 1. 2. Data Set Characteristics Data sets must be: Relevant and sufficiently representative for the intended purpose. As free of errors and as complete as possible. Possess appropriate statistical properties, especially regarding the persons or groups the AI system is intended to be used with. These characteristics can be met at the level of individual data sets or through a combination of data sets 2. 3. Data Collection and Processing Practices must address: Design choices. Data collection processes and the origin of data, including the original purpose for personal data. Data preparation operations (annotation, labelling, cleaning, updating, enrichment, aggregation). Assumptions about what the data is supposed to measure or represent. Assessment of data availability, quantity, and suitability. Examination and mitigation of possible biases that could affect health, safety, or fundamental rights, or lead to discrimination. Identification and addressing of data gaps or shortcomings 2. 4. Contextual Relevance Data sets should consider features, characteristics, or elements specific to the geographical, contextual, behavioural, or functional setting where the AI system will be used 32. 5. Bias Detection and Special Categories of Data Providers may process special categories of personal data (e.g., sensitive data) only if strictly necessary for bias detection and correction, and only if: Bias cannot be addressed using other data. Technical and privacy-preserving measures (like pseudonymisation) are in place. Access is strictly controlled and documented. Data is deleted once bias is corrected or at the end of its retention period. Processing activities are recorded with justifications for necessity 2. 6. Data Minimisation and Privacy The right to privacy and data protection must be guaranteed throughout the AI system’s lifecycle. Principles of data minimisation and data protection by design and by default apply. Measures may include anonymisation, encryption, and privacy-preserving technologies 3. 7. Documentation and Traceability Comprehensive records and technical documentation must be maintained, including information on data, algorithms, training, testing, validation processes, and risk management. High-risk AI systems should allow for automatic event logging throughout their lifetime 3. 8. Applicability For high-risk AI systems not using model training techniques, these requirements apply only to testing data sets 2. These requirements are designed to ensure that high-risk AI systems are developed and evaluated using data that is robust, fair, and respects fundamental rights, with strong safeguards for privacy and non-discrimination	12032
2	Summarize the AI Act's approach to regulating generative AI, including its classification, transparency obligations, and any specific rules for foundation models.	Summary of the EU AI Act’s Approach to Regulating Generative AI 1. Classification of Generative AI The Act refers to "general-purpose AI models," which include generative AI systems, and sets out classification rules for identifying those with "systemic risk" 1. 2. Transparency Obligations Deployers of AI systems that generate or manipulate image, audio, or video content (such as deepfakes) must disclose that the content has been artificially generated or manipulated. This disclosure requirement does not apply if the use is authorized by law for purposes such as detecting, preventing, investigating, or prosecuting criminal offenses. For content that is evidently artistic, creative, satirical, or fictional, the transparency obligation is limited to disclosing the existence of such generated or manipulated content in a way that does not hinder the enjoyment of the work. If AI-generated text is published to inform the public on matters of public interest, it must be disclosed as artificially generated or manipulated, unless it has undergone human editorial review and responsibility is held by a natural or legal person. All required information must be provided clearly and accessibly at the time of first interaction or exposure 1. 3. Specific Rules for Foundation Models (General-Purpose AI Models) Providers of general-purpose AI models must comply with EU copyright law, especially regarding text and data mining. If rightsholders have reserved their rights, providers must obtain authorization before using such works for training. Providers must publish a sufficiently detailed summary of the content used for training the model, including main data collections or sets, to increase transparency. This summary should be comprehensive but not technically detailed, balancing transparency with protection of trade secrets and confidential business information 2. The Act also sets harmonized rules for the placing on the market, use, and monitoring of general-purpose AI models, and requires codes of practice to facilitate effective implementation of detection and labeling obligations for AI-generated content 123. 4. Implementation and Enforcement Obligations for providers of general-purpose AI models will apply from August 2025, with codes of practice expected by May 2025. Member States must establish and notify rules on penalties and ensure effective enforcement by the date of application 3. Key Points Generative AI is regulated through classification, transparency, and specific obligations for foundation models. Transparency is a central requirement, especially for content that could mislead or inform the public. Foundation model providers face additional requirements regarding copyright compliance and data transparency. Enforcement timelines and governance structures are clearly defined to ensure compliance and market fairness	12516	AI Act’s Approach to Regulating Generative AI 1. Classification of Generative AI Generative AI models are considered a typical example of general-purpose AI models, as they can generate various types of content (text, audio, images, video) and perform a wide range of tasks. A general-purpose AI model is classified as having “systemic risk” if it: Has high impact capabilities, evaluated using technical tools, indicators, and benchmarks. Exceeds a computational threshold for training (specifically, more than 10²⁵ floating point operations). Is designated by the Commission based on equivalent impact or capabilities, even if not meeting the above criteria directly. Providers can argue that their model, despite meeting the threshold, does not present systemic risk due to specific characteristics, but the Commission has the final say on classification12. 2. Transparency Obligations Providers of general-purpose AI models must: Prepare and maintain up-to-date technical documentation. Provide information about the model to downstream providers to enable integration and compliance. Make documentation available to the AI Office and national authorities upon request. For models released under free and open-source licenses, transparency and openness are ensured if parameters (including weights), model architecture, and usage information are made public. Exceptions to transparency requirements exist for open-source models unless they present systemic risk2. For generative AI systems that interact with people or generate content: Users must be notified when interacting with an AI system, unless it is obvious. Special consideration is given to vulnerable groups (e.g., by age or disability). When AI systems process biometric data to infer emotions or assign categories, notification must be accessible to persons with disabilities3. 3. Specific Rules for Foundation Models (General-Purpose AI Models) Providers of general-purpose AI models (foundation models) have additional responsibilities: Maintain and update technical documentation. Provide information to downstream users. Comply with copyright law and publish summaries of training data. Enhanced obligations for models with systemic risk, including: Rigorous evaluation and risk mitigation. Incident reporting and cybersecurity measures. Open-source models are generally exempt from some transparency requirements unless they pose systemic risk12. Providers and deployers must implement technical solutions (e.g., watermarks, metadata, cryptographic methods) to mark and detect AI-generated or manipulated content, especially for synthetic content that could be mistaken for authentic (e.g., deepfakes). Deployers must clearly label AI-generated or manipulated content to disclose its artificial origin, with exceptions for creative, artistic, or satirical works, provided rights and freedoms are safeguarded3. Summary Table Aspect Requirement/Rule Classification Based on generality, computational thresholds, and systemic risk criteria Transparency Documentation, disclosure to users, marking AI-generated content, special rules for open-source Foundation Models Additional documentation, risk management, copyright compliance, enhanced rules for systemic risk All these measures aim to ensure responsible development, deployment, and use of generative AI, balancing innovation with transparency, safety, and accountability	13951
3	Article 9 mentions that the list of high-risk AI systems in Annex III can be amended. What is the process for amending this Annex?	Process for Amending Annex III (List of High-Risk AI Systems) under Article 9 Empowerment of the Commission: The European Commission is empowered to adopt delegated acts to amend Annex III. This includes adding or modifying use-cases of high-risk AI systems. Conditions for Amendment: Amendments can be made where both of the following conditions are fulfilled: The AI systems are intended to be used in any of the areas listed in Annex III. The AI systems pose a risk of harm to health and safety, or an adverse impact on fundamental rights, and that risk is equivalent to, or greater than, the risk posed by the high-risk AI systems already referred to in Annex III. Removal of High-Risk AI Systems: The Commission can also remove high-risk AI systems from the list if: The system no longer poses significant risks to fundamental rights, health, or safety. The deletion does not decrease the overall level of protection provided by the regulation. Consistency and Protection: Any amendment must not decrease the overall level of protection of health, safety, and fundamental rights and must ensure consistency with other delegated acts and take into account market and technological developments. Consultation and Evidence: Amendments are based on concrete and reliable evidence and may involve consultation with the European Artificial Intelligence Board. Procedure: The process follows the adoption of delegated acts in accordance with Article 97 of the regulation	8596	Process for Amending Annex III (List of High-Risk AI Systems) The Commission is empowered to adopt delegated acts to amend Annex III by adding or modifying use-cases of high-risk AI systems. Two main conditions must be fulfilled for such amendments: The AI systems must be intended for use in any of the areas listed in Annex III. The AI systems must pose a risk of harm to health and safety, or an adverse impact on fundamental rights, and that risk must be equivalent to or greater than the risk posed by the high-risk AI systems already listed in Annex III. The Commission can also remove high-risk AI systems from Annex III if: The system no longer poses significant risks to fundamental rights, health, or safety, considering specific criteria. The removal does not decrease the overall level of protection of health, safety, and fundamental rights under Union law. The Commission may consult the European Artificial Intelligence Board and is required to ensure that any amendments do not decrease the overall level of protection provided by the Regulation. The process includes the possibility of adding new conditions or modifying existing ones if there is concrete and reliable evidence regarding the risk profile of AI systems. Any amendments must ensure consistency with other delegated acts and take into account market and technological development	6668
4	What is the difference between a 'high-risk AI system' and a 'prohibited AI practice' under the regulation?	Difference Between 'High-Risk AI System' and 'Prohibited AI Practice' Under the EU AI Act High-Risk AI System: These are AI systems that, due to their intended purpose and context of use, present significant risks to health, safety, or fundamental rights. High-risk AI systems are subject to strict mandatory requirements to mitigate risks and ensure trustworthiness. These requirements include risk management, conformity assessments, and compliance with applicable EU harmonisation legislation. Examples include AI systems used to influence election outcomes, unless their output is not directly exposed to natural persons (e.g., administrative tools for campaign organization), or systems listed in Annex I and III of the regulation. Some AI systems may be excluded from the high-risk category if they only perform narrow procedural tasks, improve results of completed human activities, detect decision-making patterns without influencing human assessment, or perform preparatory tasks. However, if the system performs profiling of natural persons, it is always considered high-risk123. Prohibited AI Practice: The provided extracts do not explicitly define or list 'prohibited AI practices.' However, it is implied that certain AI practices are outright banned if they infringe upon other Union laws or pose unacceptable risks, such as violating fundamental rights or being used in ways that are not lawful under other acts of Union or national law. The regulation states that the classification of an AI system as high-risk does not mean its use is lawful under all circumstances, and prohibitions may apply if the AI practice infringes other Union law2. Summary Table: Aspect High-Risk AI System Prohibited AI Practice Definition Significant risk, subject to strict requirements Outright banned, not allowed under any terms Regulation Approach Regulated, not banned; must comply with requirements Not regulated, simply forbidden Examples Election-influencing AI, profiling systems Not explicitly listed in provided extracts Lawfulness May be lawful if compliant Always unlawful if prohibited In summary, high-risk AI systems are regulated with mandatory requirements to mitigate risks, while prohibited AI practices are not allowed at all under the regulation. The distinction is that high-risk systems can be used if compliant, whereas prohibited practices are banned outright	11091	Understanding 'High-Risk AI Systems' vs. 'Prohibited AI Practices' under the EU AI Act Prohibited AI Practices These are AI systems and uses that are outright banned under the EU AI Act due to their unacceptable risk to fundamental rights, safety, or democratic values. Examples include: Manipulating or exploiting vulnerabilities of individuals. Inferring sensitive personal data from biometrics (such as deducing political opinions or sexual orientation from facial data). Social scoring of individuals or groups, which can lead to discrimination or exclusion. 'Real-time' remote biometric identification in public spaces for law enforcement, except in narrowly defined and strictly necessary situations. Creating or expanding facial recognition databases through untargeted scraping of images from the internet or CCTV. AI systems predicting criminal behavior based solely on profiling. Detecting emotions in workplace or educational settings (except for medical or safety reasons). These practices are not allowed under any circumstances, except for very limited exceptions (e.g., certain law enforcement needs with strict oversight) 12. High-Risk AI Systems These are AI systems that are not banned but are considered to pose significant risks to health, safety, or fundamental rights. High-risk systems are allowed on the market but must comply with strict requirements and oversight. Examples include: AI systems used to influence the outcome of elections or referenda. AI used in critical infrastructure, education, employment, law enforcement (outside the prohibited practices), migration, and access to essential services. High-risk systems must meet mandatory requirements such as risk management, data governance, transparency, human oversight, and compliance with existing product safety legislation. The classification as high-risk does not automatically make the system lawful under all other EU laws; it must also comply with data protection and other relevant regulations 123. Key Differences Prohibited AI practices are banned outright due to their unacceptable risk, while high-risk AI systems are permitted but subject to strict regulatory controls. Prohibited practices focus on uses that are fundamentally incompatible with EU values, whereas high-risk systems are those that require careful management and oversight to mitigate potential harms	10723
5	The AI Act prohibits social scoring by public authorities. Are there any specific, explicitly mentioned exceptions to this prohibition	Prohibition of Social Scoring and Exceptions in the EU AI Act The EU AI Act explicitly prohibits the use of AI systems for the evaluation or classification of natural persons or groups over a certain period based on their social behavior or inferred/predicted personal or personality characteristics, when the resulting social score leads to: Detrimental or unfavorable treatment in social contexts unrelated to the context in which the data was originally generated or collected. Detrimental or unfavorable treatment that is unjustified or disproportionate to the gravity of their social behavior. Based on the provided information, there are no specific, explicitly mentioned exceptions to this prohibition on social scoring by public authorities. The prohibition is stated in clear terms, and no carve-outs or exceptions are detailed in the retrieved text regarding this practice	5757	Prohibition of Social Scoring by Public Authorities under the AI Act The AI Act explicitly prohibits AI systems that provide social scoring of natural persons by public or private actors. This prohibition is based on concerns that such systems may lead to discriminatory outcomes, exclusion of certain groups, and violations of dignity, equality, and justice. The Act describes social scoring as evaluating or classifying individuals or groups based on multiple data points related to their social behavior in various contexts, or on known, inferred, or predicted personal or personality characteristics over certain periods. The resulting social score may lead to detrimental or unfavorable treatment in contexts unrelated to the original data collection or to disproportionate or unjustified consequences. Exceptions to the Prohibition The only exception mentioned is for lawful evaluation practices of natural persons that are carried out for a specific purpose in accordance with Union and national law. This means that while general social scoring is prohibited, evaluations for specific, lawful purposes that comply with existing legal frameworks are not affected by this prohibition. Summary Table General Rule: Social scoring by public or private actors is prohibited. Explicit Exception: Lawful evaluation practices for a specific purpose, in accordance with Union and national law, are not affected by the prohibition. There are no other specific, explicitly mentioned exceptions to this prohibition beyond lawful evaluation for a specific purpose under existing laws	8489
6	If a high-risk AI system is found to be non-compliant after it has been placed on the market, what is the exact procedure that national market surveillance authorities must follow?	Procedure for Handling Non-Compliant High-Risk AI Systems under the EU AI Act 1. Evaluation and Initial Action If a national market surveillance authority finds that a high-risk AI system does not comply with the requirements and obligations of the EU AI Act, it must require the relevant operator to take all appropriate corrective actions. These actions may include bringing the AI system into compliance, withdrawing it from the market, or recalling it within a period prescribed by the authority, but in any event within the shorter of 15 working days or as provided for in relevant Union harmonisation legislation. The authority must inform the relevant notified body accordingly. 2. Notification Beyond National Territory If the non-compliance is not restricted to the national territory, the authority must inform the European Commission and other Member States without undue delay about the evaluation results and the actions required of the operator. 3. Operator’s Responsibility The operator must ensure that all appropriate corrective action is taken for all concerned AI systems made available on the Union market. 4. Failure to Act If the operator does not take adequate corrective action within the specified period, the market surveillance authority must take all appropriate provisional measures. These may include prohibiting or restricting the AI system from being made available or put into service, withdrawing the product or standalone AI system from the market, or recalling it. The authority must notify the Commission and other Member States of these measures without undue delay, including all available details such as identification of the non-compliant AI system, its origin, supply chain, nature of non-compliance, risk involved, and the nature and duration of national measures taken. 5. Fines and Further Measures If the provider does not bring the AI system into compliance within the required period, they may be subject to fines in accordance with Article 99. If the system was misclassified as non-high-risk to circumvent requirements, the provider is also subject to fines. Market surveillance authorities may perform appropriate checks, especially using information stored in the EU database referred to in Article 71. 6. Union Safeguard Procedure If objections are raised by another Member State’s authority or the Commission regarding a national measure, the Commission will consult with the relevant parties and evaluate the measure. The Commission will then decide whether the national measure is justified and notify all relevant authorities of its decision. This process ensures a coordinated and transparent approach to addressing non-compliance of high-risk AI systems across the EU, with clear responsibilities for both operators and authorities, and mechanisms for escalation and enforcement at the Union level	13950	Procedure for Handling Non-Compliant High-Risk AI Systems by National Market Surveillance Authorities under the EU AI Act 1. Evaluation of the AI System If a market surveillance authority has sufficient reason to consider a high-risk AI system as non-compliant (posing risks to health, safety, or fundamental rights), it must evaluate the system for compliance with all regulatory requirements, giving special attention to risks affecting vulnerable groups. If risks to fundamental rights are identified, the authority must inform and cooperate with relevant national public authorities or bodies. 2. Corrective Actions If non-compliance is confirmed, the authority must, without undue delay, require the operator to: Take all appropriate corrective actions to bring the AI system into compliance, Withdraw the AI system from the market, or Recall it within a prescribed period, which must not exceed 15 working days or as specified by relevant Union harmonisation legislation. The authority must inform the relevant notified body about these actions. 3. Notification and Cooperation If the non-compliance is not limited to the national territory, the authority must inform the European Commission and other Member States without undue delay about the evaluation results and required actions. 4. Operator’s Responsibility The operator must ensure that all appropriate corrective actions are taken for all affected AI systems made available on the Union market. 5. Provisional Measures If the operator fails to take adequate corrective action within the prescribed period, the authority must take all appropriate provisional measures, such as: Prohibiting or restricting the AI system from being made available or put into service, Withdrawing or recalling the product from the market. The authority must notify the Commission and other Member States of these measures without undue delay. 6. Notification Details The notification must include all available details, such as: Identification of the non-compliant AI system, Its origin and supply chain, Nature of the non-compliance and risk, Nature and duration of national measures taken, Arguments from the relevant operator, Whether non-compliance is due to prohibited practices, failure to meet high-risk requirements, shortcomings in standards, or other regulatory breaches. 7. Response from Other Authorities Other market surveillance authorities must inform the Commission and Member States of any measures they adopt and any additional information about the non-compliance. If they disagree with the notified national measure, they must state their objections. 8. Justification and Restrictive Measures If no objection is raised within three months (or 30 days for prohibited practices), the provisional measure is deemed justified, and appropriate restrictive measures (such as withdrawal from the market) must be taken without undue delay. 9. Union Safeguard Procedure If objections are raised or the Commission finds the measure contrary to Union law, the Commission consults with the relevant parties and evaluates the national measure. Within six months (or 60 days for prohibited practices), the Commission decides whether the measure is justified and notifies all relevant authorities and operators. 10. Final Actions If the Commission deems the measure justified, all Member States must take appropriate restrictive measures and inform the Commission. If unjustified, the Member State must withdraw the measure and inform the Commission accordingly	13577
7	The act defines an 'emotion recognition system.' Based on that definition, would an AI system used by a company to analyze customer facial expressions for product feedback be considered high-risk?	Definition of 'Emotion Recognition System' The EU AI Act defines an 'emotion recognition system' as an AI system designed to identify or infer emotions or intentions of natural persons based on their biometric data. This includes emotions such as happiness, sadness, anger, surprise, disgust, embarrassment, excitement, shame, contempt, satisfaction, and amusement. The definition specifically excludes physical states like pain or fatigue and the mere detection of obvious expressions, gestures, or movements unless these are used to infer emotions 1. High-Risk Classification According to the Act, AI systems intended to be used for emotion recognition are explicitly listed as high-risk in the context of biometrics, provided their use is permitted under relevant Union or national law 2. Application to Customer Feedback Analysis If a company uses an AI system to analyze customer facial expressions for the purpose of inferring emotions (such as satisfaction or amusement) as product feedback, this system would fall under the definition of an emotion recognition system. Since emotion recognition systems are classified as high-risk AI systems under the Act, such a use case would be considered high-risk, subject to the Act’s requirements for high-risk AI systems 2. Summary An AI system used by a company to analyze customer facial expressions for product feedback would be considered a high-risk AI system under the EU AI Act, as it fits the definition of an emotion recognition system and such systems are explicitly listed as high-ris	10609	Definition of 'Emotion Recognition System' An emotion recognition system is defined as an AI system designed to identify or infer emotions or intentions of natural persons based on their biometric data. This includes emotions such as happiness, sadness, anger, surprise, disgust, embarrassment, excitement, shame, contempt, satisfaction, and amusement. The definition specifically excludes physical states like pain or fatigue and the mere detection of obvious expressions, gestures, or movements unless these are used to infer emotions. For example, simply detecting a smile is not enough—if the system uses that smile to infer happiness, it falls under the definition of emotion recognition 1. Application to Customer Feedback Analysis If a company uses an AI system to analyze customer facial expressions specifically to infer their emotional reactions (e.g., satisfaction, amusement, or disappointment) for product feedback, this system would fit the definition of an emotion recognition system as per the regulation 1. High-Risk Classification The regulation expresses serious concerns about the scientific basis and reliability of emotion recognition systems, especially regarding their potential for discriminatory outcomes and intrusiveness. However, it specifically prohibits the use of such systems in the workplace and education settings, except for medical or safety reasons. The prohibition does not explicitly extend to commercial uses such as customer feedback analysis in retail or service environments 2. High-risk AI systems are subject to strict requirements, but the regulation distinguishes between prohibited uses (like in employment or education) and other uses. Unless the customer feedback analysis system is used in a context that falls under the prohibited categories (workplace or education), it would not automatically be classified as high-risk solely based on its function as an emotion recognition system. However, it may still be subject to other requirements if it meets criteria for high-risk AI under the regulation 2. Summary An AI system analyzing customer facial expressions for product feedback is considered an emotion recognition system. It is not automatically classified as high-risk unless used in prohibited contexts (workplace or education). Other regulatory requirements may still apply depending on the specific use case and risk assessment	11416
8	What are the specific obligations of an 'authorised representative' for an AI provider that is not established in the EU?	Obligations of an 'Authorised Representative' for an AI Provider Not Established in the EU under the EU AI Act Mandate and Tasks: The authorised representative must perform tasks specified in a mandate received from the provider. This includes: Verifying that the required technical documentation (as specified in Annex XI) has been prepared and that all obligations referred to in Article 53 and, where applicable, Article 55 have been fulfilled by the provider. Keeping a copy of the technical documentation at the disposal of the AI Office and national competent authorities for 10 years after the general-purpose AI model has been placed on the market, along with the provider’s contact details. Providing the AI Office, upon a reasoned request, with all necessary information and documentation to demonstrate compliance with the obligations in the relevant chapter. Cooperating with the AI Office and competent authorities, upon a reasoned request, in any action related to the general-purpose AI model, including when the model is integrated into AI systems placed on the market or put into service in the Union. Communication and Compliance: The mandate must empower the authorised representative to be addressed, in addition to or instead of the provider, by the AI Office or competent authorities on all issues related to ensuring compliance with the Regulation. Termination of Mandate: If the authorised representative considers or has reason to consider that the provider is acting contrary to its obligations under the Regulation, it must terminate the mandate and immediately inform the AI Office about the termination and the reasons for it. Exemptions: These obligations do not apply to providers of general-purpose AI models released under a free and open-source licence that allows access, usage, modification, and distribution of the model, and whose parameters, architecture, and usage information are made publicly available, unless the models present systemic risks	8217	Obligations of an 'Authorised Representative' for Non-EU AI Providers under the EU AI Act Appointment Requirement: Providers of general-purpose AI models established in third countries (outside the EU) must, by written mandate, appoint an authorised representative established in the Union before placing their models on the EU market. Mandate and Tasks: The authorised representative must be empowered by the provider to: Verify that the technical documentation (as specified in Annex XI) has been prepared and that all obligations under Article 53 and, where applicable, Article 55, have been fulfilled. Keep a copy of the technical documentation and the provider’s contact details available for the AI Office and national competent authorities for 10 years after the model is placed on the market. Provide the AI Office, upon a reasoned request, with all necessary information and documentation to demonstrate compliance. Cooperate with the AI Office and competent authorities in any action related to the general-purpose AI model, including when the model is integrated into AI systems placed on the market or put into service in the Union. Be the contact point for the AI Office or competent authorities on all compliance issues, in addition to or instead of the provider. Termination of Mandate: If the authorised representative believes the provider is acting contrary to its obligations, it must terminate the mandate and immediately inform the AI Office of the termination and the reasons. Exemptions: This obligation does not apply to providers of general-purpose AI models released under a free and open-source licence, provided the model’s parameters, architecture, and usage information are made publicly available, unless the models present systemic risks. Confidentiality: All information or documentation obtained, including trade secrets, must be treated in accordance with the confidentiality obligations set out in Article	8009
9	The regulation establishes an 'AI Board.' What is its role in advising the Commission on the need to amend the list of prohibited AI practices in the future?	Role of the AI Board in Advising on Prohibited AI Practices The AI Board is involved in the evaluation and review process of the regulation, particularly regarding the list of prohibited AI practices. When the European Commission carries out evaluations and reviews of the regulation, it is required to take into account the positions and findings of the Board, among other relevant bodies. The AI Office is tasked with developing an objective and participative methodology for evaluating risk levels, which includes the potential inclusion of new systems in the list of prohibited practices set out in Article 5. This process ensures that the Board's expertise and input are considered when determining whether amendments to the list of prohibited AI practices are necessary, based on technological developments, impacts on health and safety, and fundamental right	7697	Role of the AI Board in Advising on Amendments to the List of Prohibited AI Practices The regulation specifies that the Commission is responsible for assessing the need to amend the list of prohibited AI practices (as laid down in Article 5) once a year following the entry into force of the regulation. In carrying out these evaluations and reviews, the Commission is required to take into account the positions and findings of the AI Board, as well as those of the European Parliament, the Council, and other relevant bodies or sources. The AI Board, therefore, plays an advisory role by providing its positions and findings to the Commission, which the Commission must consider when deciding whether amendments to the list of prohibited AI practices are necessary. This process ensures that the AI Board contributes expert input and oversight to the ongoing evaluation and potential revision of prohibited AI practices, supporting the Commission in maintaining up-to-date and effective regulation in response to technological and societal developments	8650

*My interpretion *:
It's clear that while both systems can retrieve information, the PageIndex method consistently delivers more precise, complete, and contextually aware answers. The TextEmbedding method often provides answers that are either incomplete or miss critical nuances.

Question #	Question Type	Text Embedding RAG Performance	PageIndex RAG Performance	Winner
1	Needle in a Haystack	Good, but less structured. It found most of the requirements but mixed them together.	Excellent. It provided a more structured, comprehensive, and well-organized list of requirements, including key details like bias detection for special data categories.	PageIndex
2	"Big Picture" Synthesis	Decent, but fragmented. It identified the main themes but the information was less cohesive.	Excellent. It gave a more detailed breakdown of "systemic risk" and provided a helpful summary table, showing a superior ability to synthesize information from multiple sections.	PageIndex
3	Navigational / Reference	Good. It correctly identified the process for amending Annex III.	Slightly Better. It provided a similarly correct answer but in a more concise and direct manner, suggesting a more efficient retrieval path.	PageIndex
4	Comparative / Relational	Poor / Incomplete. It failed to find the definition of "prohibited AI practice" in the retrieved chunks, leading to a one-sided and incomplete comparison.	Excellent. It successfully retrieved the definitions for both concepts from their respective sections and provided a clear, accurate comparison with concrete examples.	PageIndex
5	Exception to the Rule	Incorrect / Incomplete. It stated that there are no exceptions to the social scoring prohibition, missing the crucial nuance.	Correct & Nuanced. It correctly identified the general prohibition and the specific exception for lawful evaluations, demonstrating a deeper level of detail.	PageIndex
6	Procedural Deep-Dive	Good. It outlined the main steps for handling non-compliant systems.	Excellent. It provided a much more detailed, 10-step procedure, including timelines and the full Union safeguard process. This shows a more thorough traversal of the relevant chapter.	PageIndex
7	Definition Application	Incorrect. It correctly defined "emotion recognition" but then incorrectly classified the use case as high-risk without considering the context (workplace/education).	Correct & Nuanced. It correctly defined the term and correctly noted that while it is an emotion recognition system, it is not automatically high-risk because the prohibition is specific to workplace/education contexts.	PageIndex
8	Role & Responsibility	Good. It listed the main obligations for an authorized representative.	Slightly Better. The answer was almost identical, but slightly more structured. Both systems performed well here, likely because the answer was contained in a single, well-defined section.	Draw
9	Future-Looking	Vague. It correctly identified that the AI Board has a role but was vague on the specifics of the process.	Excellent. It provided a clear, step-by-step explanation of the AI Board's advisory role in the annual review process, demonstrating a better grasp of procedural details.	PageIndex

The Case for PageIndex: Why It Performed Better
The results speak for themselves. The PageIndex (reasoning-based RAG) approach consistently outperformed the traditional Text Embedding (vector-based RAG) for several key reasons that are evident in your test:

It Overcomes "Semantic Similarity ≠ Relevance" (Questions 4, 5, 7): o The most striking failures of the text embedding model were when it missed crucial context. It couldn't find the definition of "prohibited practices" (Q4), missed the exception for "social scoring" (Q5), and misinterpreted the "emotion recognition" classification (Q7). This is because the most relevant answer wasn't always the most semantically similar text chunk. PageIndex, by navigating the document's logical structure, could find the correct sections and assemble a complete, nuanced picture.
It Avoids Contextual Fragmentation (Questions 2, 6): o On "big picture" and procedural questions, PageIndex delivered far more comprehensive answers. For the non-compliance procedure (Q6), it returned a 10-step process, while the embedding method gave a more general overview. This is because PageIndex can read through an entire chapter logically, whereas the embedding method retrieves a scattered collection of the "most similar" chunks, which are often fragmented and incomplete.
It Understands Intent and Structure (Questions 1, 3, 9): o PageIndex consistently delivered more structured and detailed answers. This shows it doesn't just find keywords; it understands the document's hierarchy. By starting at the "Table of Contents," it can reason: "The user is asking about amending a list. I should look in the chapter on 'Amendments' and cross-reference it with the section on 'Prohibited Practices'." This is a level of reasoning that pure vector search cannot achieve.

Conclusion:
While text embedding is a powerful tool for finding topically related information, your results clearly show its limitations when dealing with complex, structured documents where precision, nuance, and an understanding of the document's internal logic are required.
The PageIndex method is demonstrably superior for tasks that require deep, contextual understanding and reasoning. It moves beyond simple keyword matching to mimic how a human expert would navigate a document, leading to more accurate, complete, and trustworthy answers.

On Vibe and Craft: A Conversation with Jeffrey Snover on the Soul of Software Engineering

Bala Madhusoodhanan — Fri, 06 Mar 2026 09:21:13 +0000

Last week, I had the distinct honor of speaking with Jeffrey Snover, the visionary behind PowerShell and the legendary Monad Manifesto. Our conversation was a profound exploration into the very soul of software development—what I've been calling "vibe coding." It’s that elusive blend of team flow, deep focus, and shared craftsmanship that separates truly great engineering cultures from the rest.

As someone who has spent his career building tools that reshape how we work, Jeffrey’s reflections—now from the vantage point are more relevant than ever. This isn't just about code; it's about the culture, mindset, and rituals that enable teams to build better, more sustainable software. Here are the key insights from our discussion.

My conversation with Jeffrey was a powerful reminder that the tools we build are secondary to the way we think and the culture we cultivate. The pursuit of a better software paradigm is, at its heart, a human endeavor.