DEV Community: She11 QA

Organizing Application-Specific Artifacts in Automation Frameworks

She11 QA — Thu, 27 Aug 2026 07:09:49 +0000

Maintaining a clean and modular repository structure is essential when scaling test automation frameworks across multiple applications. A core practice is isolating application-specific artifacts from shared resources.

Folder Purpose

The HL-RLS_SpecificArtifacts directory stores files dedicated strictly to an individual application. This includes app-specific configurations, test data, documentation, and automation scripts directly tied to that project.

Separating these assets ensures independence, allowing teams to work autonomously without breaking global configurations.

What to Include vs. What to Exclude

Include (Application-Specific)

Configuration Files: Settings unique to the application (e.g., environment variables, app-specific configurations).
Test Data: Datasets used solely by this application's test cases (e.g., custom mock data).
Documentation: Workflows, setup guides, or instructions exclusive to the app.
Automation Artifacts: Unique test suites, helper scripts, or framework extensions tailored to this application.

Exclude (Move to Shared Directories)

Global/Shared Configurations: Store multi-app settings in Shared Resources/Configs.
Generic Test Data: Place reusable mock data in Shared Resources/Reusable-Test-Data.
Shared Libraries & Utilities: Move general helper scripts or shared libraries to Shared Resources/Libraries or Automated-Utilities.
Temporary Files: Avoid committing draft logs, unreviewed code, or personal developer files.

Examples

Good Examples

ApplicationConfig.yml – Defines app-specific deployment variables, URLs, or API keys.
TestData-SpecificApp.json – A mock dataset simulating specific user behavior for this application.
CustomScripts.sh – Tailored automation steps for server setup or specific integrations.

Anti-Examples (Belong in Shared Folders)

GlobalConfig.json – Shared config file across all projects (move to Shared Resources/Configs).
CommonLoggingLibrary.js – Reusable logging utility (move to Shared Resources/Libraries).
TestData-Common.json – Generic test dataset used across multiple test suites (move to Shared Resources/Reusable-Test-Data).

Key Takeaways & Best Practices

Isolation: Keep global and application-level concerns separate so teams can work independently without cross-contaminating shared logic.
Maintenance: Regularly review and purge outdated, unused, or draft files to preserve workspace clarity.
Exclusivity: Only keep files in this directory if they apply exclusively to the target application.

HL-RLS ( High-Level Release-Specific artifacts/configurations.)

Building an AI Engineering Observability Platform for Test Automation

She11 QA — Wed, 26 Aug 2026 06:57:58 +0000

Building an AI Engineering Observability Platform for Test Automation

Tracking static productivity percentages (e.g., 75%–80% savings) is no longer enough to prove real enterprise value. To provide transparency, governance, and business ROI, you must convert your AI-driven test automation framework into an AI Engineering Observability Platform.

Current Gap in the Model

Many teams showcase an agentic setup:

✅ Context Agent
✅ Test Case Agent
✅ Feature File Agent
✅ Page Object Agent
✅ Step Definition Agent

While this tracks estimated effort reduction (e.g., 45 hrs → 9.5 hrs), stakeholders often ask:

"How do we know AI actually did the work?"
"How many tokens were consumed?"
"What was generated daily?"
"What was the total cost?"
"How much effort did we save?"

Key Metrics to Track

1. Agent Utilization Metrics

Agent	Executions	Success Rate	Avg Runtime	Tokens Used
Context Agent	45	98%	35 sec	120K
Test Case Agent	60	95%	50 sec	850K
Feature Agent	40	99%	20 sec	150K
Page Object Agent	35	96%	45 sec	400K
Step Definition Agent	40	97%	30 sec	300K

2. Daily Productivity Output

Metric	Manual	AI	Savings
Test Cases Created	15	100	6.6X
Feature Files Created	5	50	10X
Step Definitions	20	200	10X
Page Objects	5	40	8X

3. Time Savings Calculation

Log every LLM execution with telemetry attributes:
{
"user": "Tester1",
"agent": "Test Case Agent",
"input_tokens": 3500,
"output_tokens": 6500,
"model": "GPT-4o",
"execution_time": "42 sec"
}

Effort Comparison Example:

Context Generation: 4 hrs (Manual) vs. 30 mins (AI)
Test Case Creation: 8 hrs (Manual) vs. 1 hr (AI)
Feature File Creation: 4 hrs (Manual) vs. 20 mins (AI)
Total Manual Effort: 16 hrs | AI Effort: 1.8 hrs | Net Time Saved: 14.2 hrs

4. Quality Improvements

KPI	Before AI	After AI
Test Coverage	65%	90%
Automation Coverage	50%	85%
Defect Leakage	12%	5%
Rework Rate	18%	7%

Future-State Architecture (MCP-Enabled)

Implement the Model Context Protocol (MCP) and telemetry to route logs from your agents directly to visualization tools like Power BI:

Azure OpenAI -> Orchestrator
Agents: Context, Test Case, Feature, Step, Page Object
Telemetry Layer: Logs Prompts, Token Usage, Runtime Metrics, Cost Metrics, User Metrics, Generated Assets
Dashboard: Power BI / Custom Observability Suite

Core Steering Committee KPIs

When presenting to stakeholders, focus on these 8 KPIs:

AI Adoption Rate (% of automation work generated by AI)
Tokens Consumed
Cost per Story
Automation Assets Generated
Hours Saved
Productivity Improvement %
Automation Coverage Increase
Defect Reduction %

Executive Summary Example:
"During July, the AI Automation Factory executed 5,200 agent workflows, consumed 42M tokens, generated 3,800 automation assets, reduced manual effort by 78%, saved 620 engineering hours, and improved automation coverage from 58% to 86%."

A Unified KPI Framework for Automation Testing with Playwright & JavaScript

She11 QA — Wed, 26 Aug 2026 06:48:04 +0000

Measuring the impact of test automation goes beyond simple pass/fail ratios. To demonstrate real engineering excellence and business value, automation metrics must capture execution speed, suite stability, test coverage, maintenance cost, and CI/CD integration.

Here is a comprehensive, unified KPI framework designed specifically for Playwright & JavaScript automation suites.

📊 Executive KPI Targets

Category	Metric	Target
Execution Speed	Runtime Reduction	50% ↓
Efficiency	Throughput	+40% ↑
Stability	Flaky Tests	< 3%
Reliability	Retry Dependency	< 5%
Coverage	Automation Coverage	80%+
Quality	Defect Leakage	20–30% ↓
Productivity	Script Dev Time	30% ↓
CI/CD	Pipeline Time	40% ↓
ROI	Automation ROI	Positive (3–6 months)
Cost	Manual Effort Reduction	30–50% ↓

1. Execution Efficiency & Speed

Test Execution Time Reduction: Target 40–60% reduction vs legacy frameworks like Selenium. $$\text{Reduction \%} = \frac{\text{Old Time} - \text{New Time}}{\text{Old Time}} \times 100$$
Parallel Execution Efficiency: Measure tests executed per hour and parallel thread utilization. $$\text{Efficiency \%} = \frac{\text{Sequential Time} - \text{Parallel Time}}{\text{Sequential Time}} \times 100$$
Test Throughput: Maximize total test cases executed per CI window.
CI/CD Pipeline Cycle Time: Aim for a 30–40% total reduction in build + test execution duration.

2. Stability & Reliability

Flaky Test Rate: Keep flaky tests under 2–3% by leveraging Playwright's native auto-waiting and resilient locators. $$\text{Flakiness \%} = \frac{\text{Flaky Tests}}{\text{Total Tests}} \times 100$$
Retry Dependency Ratio: Track the percentage of tests passing only after retries to minimize false positives.
Failure Root Cause Accuracy: Target >90% of test failures pointing directly to genuine application defects rather than script instability.

3. Coverage Metrics

Automation Coverage: Maintain 80%+ regression coverage across all functional scenarios.
Cross-Browser & Device Coverage: Measure test runs across Chromium, Firefox, and WebKit using Playwright’s native multi-browser support and device emulation.
API + UI Balance: Track distribution across API-level setup and UI assertion layers.

4. Quality & Defect Metrics

Defect Detection Efficiency (DDE): $$\text{DDE \%} = \frac{\text{Defects Found in Testing}}{\text{Total Defects}} \times 100$$
Defect Leakage Reduction: Target a 20–30% reduction in production defects.
Mean Time to Detect (MTTD): Catch defects post-commit within minutes, not hours.

5. Productivity & Maintainability

Script Development Time: Aim for ~30% faster scripting using Playwright Codegen and concise JS APIs.
Reusability Index: Track percentage of shared fixtures, hooks, and Page Object components. $$\text{Reusability \%} = \frac{\text{Reusable Components}}{\text{Total Code}} \times 100$$
Maintenance Effort: Monitor hours spent updating selectors and test logic per sprint.

6. Cost & ROI Calculation

Automation ROI: $$\text{ROI} = \frac{\text{Manual Effort Saved} - \text{Automation Cost}}{\text{Automation Cost}}$$
Resource Optimization: Target a 30–50% reduction in manual regression QA effort, achieving positive ROI within 3 to 6 months.

🚀 Why Playwright Delivers High ROI

Faster execution through isolated browser contexts and parallel execution.
Stable test runs via auto-waiting and built-in network mocking.
Simplified debugging using Trace Viewer, automated video capture, and step-by-step DOM snapshots.
Native CI/CD integration designed for continuous delivery setups.

Building an Automated QA KPI Dashboard for Playwright & BDD Pipelines

She11 QA — Wed, 26 Aug 2026 06:41:36 +0000

Tracking test automation metrics manually often leads to outdated figures and missed engineering gaps. To solve this, automated reporting directly from your test suites—such as Playwright and Cucumber—provides clear visibility into health, execution speed, and coverage.

Below is a breakdown of how to structure an Automation KPI Dashboard to streamline test metrics, track trends, and establish actionable engineering goals.

Executive Summary Dashboard

KPI Metric	Target	Current Value	Status	Trend
Total Test Cases	100% coverage	85%	🟡 Partial	↗️ Up
Automated Test Coverage	90%+	78%	🟡 Partial	↗️ Up
Pass Rate (Last Run)	95%+	92%	🟡 Partial	↔️ Stable
Avg. Execution Time	< 30 min	28 min	🟢 Good	↘️ Down
Flaky Test Rate	< 2%	1.5%	🟢 Good	↔️ Stable
Defects Detected	—	3	🟡 Review	↔️ Stable
CI/CD Pipeline Success	100%	98%	🟡 Partial	↗️ Up

Key Metric Breakdowns

1. Coverage & Execution

Total Test Suite: 120 tests (94 Automated, 26 Manual).
Latest Run (2026-05-29): 94 executed — 87 passed, 7 failed, 0 skipped.

2. Flakiness Tracking

Flaky Tests (Last 10 Runs): 2 scenarios identified.
Top Offenders:
1. Scenario A: UI timeout issues.
2. Scenario B: Data synchronization lag.

3. Defect Detection & CI/CD Performance

Defect Lifecycle: 3 opened, 1 closed (Avg. resolution time: 2 days).
Pipeline Health: 98% success rate, 12 min average build time.
Primary Cause of Pipeline Failure: Dependency resolution errors.

Execution & Pass Rate Trends (Last 6 Runs)

Run Date	Pass %	Fail %	Flaky %	Duration (min)
2026-05-29	92%	8%	2%	28
2026-05-28	91%	9%	2%	29
2026-05-27	90%	10%	3%	30
2026-05-26	89%	11%	3%	31
2026-05-25	88%	12%	4%	32
2026-05-24	87%	13%	4%	33

Next Engineering Action Items

Automation Expansion: Push total automated coverage past 90%.
Flakiness Mitigation: Refactor explicit waits and isolation for UI timeout and data sync scenarios.
Pipeline Stability: Resolve dependency caching errors to bring CI/CD success to 100%.
Optimization: Lower execution suite duration below 25 minutes using parallel run setups.

Implementation Note: This dashboard context can be auto-generated by parsing execution JSON outputs (from frameworks like Playwright or Cucumber BDD) directly into your CI/CD reporting artifacts after every major test cycle or sprint.

GenAI in Test Automation: Accelerating Testing with GitHub Copilot & Agentic Solutions

She11 QA — Tue, 25 Aug 2026 08:48:23 +0000

GenAI in Test Automation: Accelerating Testing with GitHub Copilot & Agentic Solutions

Generative AI is shifting how quality assurance teams design, generate, and maintain test automation suites. By using GitHub Copilot (GHCP) alongside agentic workflows, software teams can significantly cut setup time while maintaining production-grade standards.

Here is a practical framework for leveraging GenAI and agentic solutions to accelerate test automation.

🛠️ Tooling & Core Capabilities

Primary Tool: GitHub Copilot (GHCP)

Framework & Language Support

Frameworks: Selenium, Playwright, Cypress
Languages: Java, JavaScript, Python
Scope: Web & API application testing
Architectural Patterns: Generates modular scripts using Page Object Model (POM), BDD, and Data-Driven Testing paradigms.

⚡ Modes of Operation

Interactive Mode: Human-in-the-loop workflow tailored for prompt refinement and continuous validation.
Agentic Mode: Autonomous end-to-end test generation, execution, and reporting pipeline.
Edit Mode: Quick prompt-driven corrections and inline code modifications.

Crucial Rule: Human-in-the-loop oversight is mandatory to validate, monitor, and correct generated outputs against real-world domain requirements.

📋 Implementation Workflow

Step 1: Pre-requisites & Setup

Configure GitHub Copilot, GHCP, and relevant automation-oriented VSIX extensions in your IDE.
Format test cases in Markdown (Test ID, Objective, Steps, Expected Results, Test Data). Keep individual files under 20 MB.
Collect element locators (JSON, Excel, or Markdown format) and application configurations.

Step 2: Prompt Engineering & Execution

Use prompt templates populated with your framework details, target language, and app context.
Set explicit constraints, rules, and coding style guides within the prompt.
Process prompts through GHCP agentic workflows to generate test cases and code scripts.
Execute generated test suites, evaluate performance metrics, and iteratively refine prompts.

💡 Key Use Cases

Scenario & Case Generation: Automatically derive BDD or non-BDD user stories, scenarios, and test cases.
Manual to Automated Scripting: Rapidly translate manual test documentation into executable code.
Suite Maintenance: Automate locator fixes and script updates whenever application flows change.
Intelligent Prioritization: Target tests based on risk, business criticality, and execution frequency.
Data & Reporting: Automate test data creation and post-execution reporting.

📌 Best Practices Checklist

Human Oversight: Always inspect AI-generated code for edge cases, logical errors, and adherence to company coding standards.
Track Impact Metrics: Measure success by tracking output accuracy, test coverage improvements, bug reduction rates, and overall engineering time saved.
Document Structured Inputs: Provide clean input context (BRD documents, locator files, code samples) to minimize hallucination risks during script generation.

Essential Git Commands for Feature Branch Workflow

She11 QA — Tue, 25 Aug 2026 08:40:14 +0000

How to Switch, Sync, and Merge Git Branches in VS Code

Keeping your feature or QA branch up to date with the main codebase is essential for preventing merge conflicts later down the line. Here is a quick reference guide on how to switch branches, fetch new remotes, and pull the latest changes into your branch using VS Code and Git terminal.

1. How to Switch Branches in VS Code

Using the Command Palette:

Press Ctrl + Shift + P (Windows/Linux) or Cmd + Shift + P (Mac).
Type and select Git: Checkout to...
Select your target branch (e.g., your-feature-branch).

Using the Terminal:

If the branch hasn't been fetched locally yet:

# Fetch the specific branch from remote
git fetch origin your-feature-branch

# Switch to the branch
git checkout your-feature-branch

2. Syncing Your Branch with the Latest Main Code

Follow these 5 steps to update your local working branch with the latest upstream code from main:

Step 1: Switch to the Main Branch

git checkout main

Step 2: Pull Remote Changes

git pull origin main

Step 3: Return to Your Feature Branch

git checkout your-feature-branch

Step 4: Merge Main into Your Branch

git merge main

Handling Conflicts: If VS Code highlights merge conflicts, edit the files to resolve them, then stage and commit:
git add .
git commit -m "Merge main into your-feature-branch"

Step 5: Push Updates to Remote

git push origin your-feature-branch

Quick Summary Checklist

git checkout main → Jump to main
git pull origin main → Get latest changes
git checkout → Return to your working branch
git merge main → Bring latest main changes into your branch
git push origin → Push updated work to remote

Setting Up a Playwright JavaScript BDD Automation Framework with Cucumber

She11 QA — Tue, 25 Aug 2026 08:31:19 +0000

Setting up a scalable test automation framework using Playwright JavaScript with BDD (Behavior-Driven Development) via Cucumber helps streamline testing and maintain consistency across complex test suites.

Here is a step-by-step guide to onboarding and initializing this test environment.

Prerequisites & Tools

Make sure you have the following software installed:

Recommended VS Code Extensions

Open the Extensions tab in VS Code (Ctrl + Shift + X / Cmd + Shift + X) and install:

Cucumber (Gherkin) Support — Syntax highlighting and step definition navigation for .feature files.
JavaScript / TypeScript Support — Auto-completion and language tooling.
ESLint (Optional) — Code linting to maintain style rules and clean syntax.

Setup & Initialization Commands

Clone the repository, navigate to your root workspace, and execute the following commands in your integrated terminal:

1. Install required project dependencies

npm install

2. Download and configure required Playwright browser binaries

npx playwright install --with-deps

3. Install Excel utility packages for handling test data

npm install exceljs --save

4. Optional security/vulnerability audit

npm audit

5. Launch tests in Playwright's UI Mode for debugging

npm run test/ui

Best Practices for BDD Automation Suites

Align Feature Files with Test Specifications: Always map .feature Gherkin steps directly against your functional requirements sheet to prevent missing coverage.
Centralize Documentation: Keep detailed architecture notes and setup guides inside the README.md file and a dedicated /doc folder for faster onboarding.
Data Handling: Use structured files (such as Excel via exceljs or JSON) to separate test scripts from dynamic test data.

Fixing PowerShell Script Execution Policy Issue When Activating Python venv

She11 QA — Tue, 25 Aug 2026 08:26:09 +0000

When setting up a Python virtual environment on Windows, running .\venv\Scripts\activate in PowerShell often throws a script execution policy error (such as PSSecurityException or cannot be loaded because running scripts is disabled on this system).

Here is a quick walkthrough to resolve this issue safely without disabling your system's global security policies.

The Cause

Windows PowerShell restricts script execution by default under the Restricted policy to prevent malicious scripts from executing. Because virtual environment activation scripts (Activate.ps1) are local scripts, PowerShell blocks them.

Step-by-Step Fix

1. Open PowerShell

Launch PowerShell (or your integrated terminal in VS Code).

2. Update Execution Policy for Current User

Run the following command to allow locally generated scripts to execute for your Windows profile only:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Why -Scope CurrentUser?

Scope limiting ensures you only grant permission to your current user session, eliminating the need to alter global system policies or risk security system-wide.

Verification Workflow

Once the policy is set, you can create and activate your environment seamlessly:


# 1. Create your virtual environment
python -m venv venv

# 2. Activate the virtual environment
.\venv\Scripts\activate

# Output: Your prompt will now show the environment prefix:
# (.venv) PS C:\your-project-path>

Summary

Error: PowerShell blocks Activate.ps1.
Fix: Set -ExecutionPolicy RemoteSigned scoped to -Scope CurrentUser.
Security: Keeps remote unsigned scripts blocked while allowing local development tools to function smoothly.

Setting Up Playwright with JavaScript for Test Automation: Prerequisites & VS Code Extensions

She11 QA — Tue, 25 Aug 2026 08:20:04 +0000

Setting up a new test automation framework can be tricky if you miss essential tools and extensions. Below is a structured checklist to get your environment fully prepared for automation testing using Playwright with JavaScript.

1. Framework & Core Requirements

Framework Type: Playwright (JavaScript)
Configuration: settings.xml (shared via your internal communications channel)
Repository: xlc-portfolio-tech-shared-services/xlc-app-kc-test-automation

2. Development Environment Setup

IDE: Install Visual Studio Code.
Configuration Files: Ensure settings.xml and Framework.zip are downloaded and properly configured in your root workspace.

3. Recommended VS Code Extensions

To streamline writing BDD-style tests, running Playwright suites, and debugging, install the following extensions:

Playwright & Testing:
- Playwright Test Runner
- Playwright Test for VSCode
BDD & Gherkin / Cucumber Support:
- Cucumber
- Cucumber (Gherkin) Full Support
- Snippets and Syntax Highlight for Gherkin (Cucumber)
AI & GitHub Integration:
- GitHub Copilot
- GitHub Copilot Chat
- GitHub Actions
Code Quality & Utilities:
- ESLint
- XML Tools
- Rainbow CSV
- Excel Viewer
- Microsoft Edge Tools for VS Code

💡 Pro Tip

Make sure all team members maintain identical VS Code extension sets to prevent formatting inconsistencies and ensure smooth local test runs!

Building an AI Test Automation Factory: How We Reduced Automation Effort by 78% with Multi-Agent Systems & MCP

She11 QA — Sat, 22 Aug 2026 09:25:45 +0000

Traditional test automation frameworks often carry heavy maintenance costs, slow release cycles, and high knowledge dependency. By transitioning from standard script creation to a governed AI Test Automation Factory, engineering teams can shift their focus from writing boilerplate code to high-value validation and architectural optimization.

Here is an architectural breakdown of how multi-agent AI systems, governed telemetry, and Model Context Protocol (MCP) transform enterprise quality engineering.

The Problem: The 45-Hour Manual Bottleneck

Building a end-to-end BDD automation suite manually requires significant time per user story—often taking up to 45 hours across five distinct steps:

Context Generation & Requirements Review (~8 hrs)
Manual Test Case Design (~9 hrs)
Cucumber Feature File Creation (~8 hrs)
Page Object Model Generation (~8 hrs)
Step Definition Implementation (~10 hrs)

This traditional workflow creates coverage gaps, inconsistent code quality, and defect leakage.

The Solution: Multi-Agent AI Automation Pipeline

Instead of relying on single prompts, an AI Test Automation Factory routes requirement artifacts (BRDs / User Stories) through specialized agents:


[BRD / User Story] 
       │
       ▼
[Context Agent] ──► [Test Case Agent] ──► [Feature File Agent]
                                                  │
[Automation Suite] ◄── [Step Definition Agent] ◄── [Page Object Agent]

Context Agent: Parses acceptance criteria and enterprise domain knowledge.
Test Case Agent: Auto-generates exhaustive test scenario matrices.
Feature File Agent: Drafts standardized BDD Cucumber feature files.
Page Object & Step Def Agents: Constructs clean design patterns (POM) and matching step implementations.

Measurable ROI: Before vs. After AI

By replacing manual generation with agentic workflows, the effort to automate a scenario drops from 45 hours to 9.5 hours:

Phase	Manual Effort	AI-Driven Effort	Time Saved
Context Generation	8 hrs	2 hrs	75%
Test Design	9 hrs	2 hrs	78%
Feature File Creation	8 hrs	0.5 hrs	94%
Page Object Creation	8 hrs	2 hrs	75%
Step Definitions	10 hrs	3 hrs	70%
Total Effort	45 hrs	9.5 hrs	78% Reduction

Key Business Metrics:

Productivity Multiplier: 4X Faster Delivery
Test Coverage: Increased from 65% to 90%
Defect Leakage: Reduced from 12% to 5%
Overall Cost Footprint: Scaled down to 22% of original baseline

AI Governance & Observability

Enterprise deployment requires strict guardrails around LLM usage. A telemetry layer sits between the agents and executive reporting dashboards to monitor performance in real time:

Token & Usage Tracking: Daily audit trails for prompt/completion token consumption.
Cost & Adoption Monitoring: Sprint-by-sprint metrics tracking user engagement vs. API spend.
Executive Visibility: Real-time Power BI reporting reflecting total hours saved and generated code assets.

The Future: Autonomous Testing via MCP

The future of QA lies in moving from AI-Assisted generation to Autonomous Self-Healing Execution. Leveraging the Model Context Protocol (MCP) enables seamless enterprise knowledge integration, allowing agents to directly query system context, adjust broken locators automatically, and deliver a fully autonomous QA pipeline.

How to Configure Full Parallel Execution in TestNG via Maven (with Dynamic Thread Control)

She11 QA — Sat, 22 Aug 2026 09:16:16 +0000

Running automation test suites sequentially can quickly become a bottleneck in CI/CD pipelines. To optimize execution efficiency, you can enable full parallel execution using TestNG and Maven, allowing you to control thread counts dynamically right from the command line without modifying XML files every time.

Here is a step-by-step guide on how to configure your framework for flexible parallel execution, along with a comparison of native TestNG execution versus custom Allocator/Run Manager approaches.

Step-by-Step Configuration

1. Update testng_regression.xml

Modify the tag to set your default parallel mode and thread count. You can set parallel to methods, classes, tests, or instances based on your architecture:


<suite name="Regression" parallel="methods" thread-count="10">

2. Configure pom.xml for Dynamic Control

To override these settings at runtime without touching the codebase, map properties inside the block of the maven-surefire-plugin:


<configuration>
    <parallel>${parallel}</parallel>
    <threadCount>${threadCount}</threadCount>
</configuration>

Command-Line Usage Examples

Once configured, you can pass parameters dynamically via Maven:

Default Run (10 threads):

   mvn clean test -P runTestNGTests

Dynamic Thread Count (15 threads):


   mvn clean test -P runTestNGTests -DthreadCount=15

Change Parallel Mode at Runtime:


   mvn clean test -P runTestNGTests -Dparallel=classes -DthreadCount=15

High-Throughput Run (Match CPU cores, e.g., 24 threads):


   mvn clean test -P runTestNGTests -DthreadCount=24

Comparison: Native TestNG vs. Custom Excel Allocator

If your framework currently uses a custom Excel-based Run Manager alongside native TestNG, here is how the two approaches compare:

Feature / Aspect	Custom Allocator (Run Manager)	Native TestNG
Entry Point	allocator.Allocator.main() via Maven exec plugin	maven-surefire-plugin running testng.xml
Test Selection	Reads Excel sheets via custom properties	Reads testng_regression.xml classes/methods
Thread Management	Managed via custom ExecutorService	Native TestNG thread pool (parallel + thread-count)
Execution Command	mvn clean test -P runAllocator	mvn clean test -P runTestNGTests
Pros	Data-driven via Excel; multi-sheet aggregation support	Lightweight, faster startup, zero Excel dependencies
Cons	Requires global property tuning; code changes for structural updates	Restricted to TestNG parallel modes (methods/classes)

Which Approach Should You Choose?

Choose Custom Allocator if your suite relies heavily on fine-grained Excel-driven iteration control or multi-sheet test scheduling.
Choose Native TestNG if you want a cleaner footprint, faster execution loops without file-parsing overhead, and straightforward CLI thread scaling.

Complete End-to-End OpenTelemetry Setup Guide for AI Agents & Power BI Observability

She11 QA — Fri, 21 Aug 2026 09:14:14 +0000

OpenTelemetry E2E Setup Guide for AI Agents

This guide shows how to set up end-to-end OpenTelemetry observability for AI agents, from local tracing to production export. It covers the OpenTelemetry Collector, Python and Node.js instrumentation, agent-specific spans, LLM and tool-call tracing, validation, and production recommendations.

1. What You Are Instrumenting

For an AI agent, the most useful trace structure is a top-level agent run with child spans for every meaningful operation.


agent.run
|-- agent.plan
|-- llm.chat
|-- tool.call: knowledge_search
|-- retrieval.query
|-- memory.read
|-- llm.chat
|-- memory.write
`-- agent.finalize

At minimum, trace these operations:

agent.run : one full user request or autonomous task
llm.chat : each model call
tool.call : each tool invocation
retrieval.query : vector search, database lookup, or document retrieval
memory.read : agent memory lookup
memory.write : agent memory update
agent.handoff : transfer to another agent or human
agent.finalize : final response construction

2. Recommended Architecture


Agent Application
|
v
OTLP traces, metrics, logs
|
v
OpenTelemetry Collector
|
v Exporter
v
Observability Backend

Common observability backends include:

Jaeger
Grafana Tempo
Honeycomb
Datadog
New Relic
Azure Monitor
AWS X-Ray
Google Cloud Trace
Elastic Observability

For local development, you can start with the OpenTelemetry Collector plus a logging exporter. For production, send data from the Collector to your preferred backend.

3. Local OpenTelemetry Collector Setup

Create a file named otel-collector-config.yaml:


receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Run the Collector with Docker on Windows PowerShell:


docker run --rm -p 4317:4317 -p 4318:4318 \
  -v "${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:latest

Run the Collector on macOS or Linux:

docker run --rm -p 4317:4317 -p 4318:4318 \
  -v "$PWD/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:latest

The local OTLP endpoints are:

HTTP: http://localhost:4318/v1/traces
gRPC: localhost:4317

4. Python Agent Setup

Install dependencies:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

Create telemetry.py:


from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def configure_telemetry() -> None:
    resource = Resource.create({
        "service.name": "agent-service",
        "service.version": "1.0.0",
        "deployment.environment": "local",
    })

    provider = TracerProvider(resource=resource)
    exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

Instrument the agent:


from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from telemetry import configure_telemetry

configure_telemetry()
tracer = trace.get_tracer("agent-service")

def run_agent(user_input: str) -> str:
    with tracer.start_as_current_span("agent.run") as span:
        span.set_attribute("agent.name", "support-agent")
        span.set_attribute("agent.version", "1.0.0")
        span.set_attribute("agent.input.length", len(user_input))

        try:
            with tracer.start_as_current_span("agent.plan") as plan_span:
                plan = create_plan(user_input)
                plan_span.set_attribute("agent.plan.steps", len(plan))

            with tracer.start_as_current_span("llm.chat") as llm_span:
                llm_span.set_attribute("gen_ai.system", "openai")
                llm_span.set_attribute("gen_ai.operation.name", "chat")
                llm_span.set_attribute("gen_ai.request.model", "gpt-4.1")

                llm_response = call_llm(user_input)

                llm_span.set_attribute("gen_ai.response.model", "gpt-4.1")
                llm_span.set_attribute("gen_ai.usage.input_tokens", llm_response.input_tokens)
                llm_span.set_attribute("gen_ai.usage.output_tokens", llm_response.output_tokens)

            with tracer.start_as_current_span("tool.call") as tool_span:
                tool_span.set_attribute("tool.name", "knowledge_search")
                tool_span.set_attribute("tool.call_id", llm_response.tool_call_id)

                documents = search_documents(llm_response.tool_query)

                tool_span.set_attribute("tool.success", True)
                tool_span.set_attribute("retrieval.document_count", len(documents))

            with tracer.start_as_current_span("agent.finalize") as final_span:
                final_answer = produce_answer(llm_response, documents)
                final_span.set_attribute("agent.output.length", len(final_answer))

            return final_answer

        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            raise

5. Node.js or TypeScript Agent Setup

Install dependencies:


npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

Create telemetry.ts:

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

export const telemetrySdk = new NodeSDK({
  serviceName: "agent-service",
  traceExporter: new OTLPTraceExporter({
    url: "http://localhost:4318/v1/traces",
  }),
});

export async function startTelemetry() {
  await telemetrySdk.start();
}

export async function stopTelemetry() {
  await telemetrySdk.shutdown();
}

Instrument the agent:

import { SpanStatusCode, trace } from "@opentelemetry/api";
import { startTelemetry, stopTelemetry } from "./telemetry";

const tracer = trace.getTracer("agent-service");

export async function runAgent(userInput: string): Promise<string> {
  return tracer.startActiveSpan("agent.run", async span => {
    span.setAttribute("agent.name", "support-agent");
    span.setAttribute("agent.version", "1.0.0");
    span.setAttribute("agent.input.length", userInput.length);

    try {
      const plan = await tracer.startActiveSpan("agent.plan", async planSpan => {
        const result = await createPlan(userInput);
        planSpan.setAttribute("agent.plan.steps", result.length);
        planSpan.end();
        return result;
      });

      const llmResponse = await tracer.startActiveSpan("llm.chat", async llmSpan => {
        llmSpan.setAttribute("gen_ai.system", "openai");
        llmSpan.setAttribute("gen_ai.operation.name", "chat");
        llmSpan.setAttribute("gen_ai.request.model", "gpt-4.1");

        const response = await callLlm(userInput, plan);

        llmSpan.setAttribute("gen_ai.response.model", "gpt-4.1");
        llmSpan.setAttribute("gen_ai.usage.input_tokens", response.inputTokens);
        llmSpan.setAttribute("gen_ai.usage.output_tokens", response.outputTokens);
        llmSpan.end();

        return response;
      });

      const documents = await tracer.startActiveSpan("tool.call", async toolSpan => {
        toolSpan.setAttribute("tool.name", "knowledge_search");
        toolSpan.setAttribute("tool.call_id", llmResponse.toolCallId);

        const result = await searchDocuments(llmResponse.toolQuery);

        toolSpan.setAttribute("tool.success", true);
        toolSpan.setAttribute("retrieval.document_count", result.length);
        toolSpan.end();

        return result;
      });

      const finalAnswer = await tracer.startActiveSpan("agent.finalize", async finalSpan => {
        const answer = await produceAnswer(llmResponse, documents);
        finalSpan.setAttribute("agent.output.length", answer.length);
        finalSpan.end();
        return answer;
      });

      span.end();
      return finalAnswer;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: String(error),
      });
      span.end();
      throw error;
    }
  });
}

async function main() {
  await startTelemetry();

  try {
    await runAgent("How do I reset my password?");
  } finally {
    await stopTelemetry();
  }
}

6. Recommended Span Attributes

Use OpenTelemetry semantic conventions where possible.

GenAI Attributes

gen_ai.system
gen_ai.operation.name
gen_ai.request.model
gen_ai.response.model
gen_ai.request.temperature
gen_ai.request.max_tokens
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens

Agent Attributes

agent.name
agent.version
agent.run_id
agent.session_id
agent.step.name
agent.step.index
agent.output.length
agent.input.length

Tool Attributes

tool.name
tool.call_id
tool.success
tool.error.type
tool.retry_count

Retrieval Attributes

retrieval.system
retrieval.index.name
retrieval.query_count
retrieval.document_count
retrieval.top_k

Memory Attributes

memory.operation
memory.scope
memory.result_count

7. What Not To Capture

Do not capture sensitive or high-risk data as span attributes unless your organization has explicit approval, redaction, access control, and retention policies.

Avoid storing:

Full prompts
Full model responses
User secrets
API keys or tokens
Raw documents
Email addresses
Payment information
Health data
Authentication headers
Full tool outputs

Prefer safe metadata:

prompt length
response length
token counts
model name
tool name
status
latency
retry count
document count

8. Metrics To Add

Traces show what happened for one request. Metrics show aggregate behavior.

Recommended metrics:

agent.run.count
agent.run.duration
agent.run.error.count
llm.request.count
llm.request.duration
llm.token.input.count
llm.token.output.count
tool.call.count
tool.call.duration
tool.call.error.count
retrieval.query.count
retrieval.document.count
agent.handoff.count

Start with traces first, then add metrics once your span structure is stable.

9. Logging Setup

Use logs for discrete application events, but correlate them with traces.

Recommended log fields:

trace_id
span_id
agent_run_id
agent.name
event.name
status
error.type
error.message

Keep logs redacted. Avoid logging raw prompts, completions, credentials, or document bodies.

10. Validation Checklist

Use this checklist to verify the setup end to end.

Start the OpenTelemetry Collector.
Run one local agent request.
Confirm the Collector receives spans.
Confirm the root span is named agent.run.
Confirm LLM calls appear as llm.chat child spans.
Confirm tool calls appear as tool.call child spans.
Confirm errors are recorded on failed spans.
Confirm token usage attributes appear when available.
Confirm sensitive content is not exported.
Confirm traces reach your observability backend.

11. Production Collector Example

For production, route data through the Collector and export to your backend.


otlp:
  protocols:
    http:
    grpc:

processors:
  batch:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  otlphttp:
    endpoint: https://your-observability-backend.example.com
    headers:
      api-key: ${OBSERVABILITY_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

Set environment variables securely through your deployment platform, secret manager, or CI/CD system.

12. Production Hardening

Before production rollout:

Add sampling if trace volume is high.
Redact sensitive inputs and outputs.
Add service names and versions.
Add deployment environment attributes.
Use secure OTLP endpoints.
Store backend API keys in a secret manager.
Configure retention policies.
Add alerts for agent error rate and high latency.
Track model cost through token metrics.
Validate compliance requirements before exporting AI data.

13. Sampling Guidance

For development, sample everything.

For production, consider:

100% sampling for errors
100% sampling for low-volume critical workflows
Lower probabilistic sampling for high-volume successful requests
Tail sampling when supported by your backend or Collector distribution

Example parent-based trace ID ratio sampling in Python:


from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

provider = TracerProvider(
    sampler=ParentBased(TraceIdRatioBased(0.10))
)

This samples roughly 10% of new traces while preserving parent-child trace consistency.

14. Framework-Specific Notes

LangChain

Instrument the chain or agent executor as agent.run, then wrap model calls, retrievers, and tools as child spans. If you use callbacks, create spans inside callback handlers.

Semantic Kernel

Trace kernel invocation as agent.run, function calls as tool.call, planner execution as agent.plan, and AI service calls as llm.chat.
AutoGen or Multi-Agent Systems
Use one trace for the full multi-agent workflow. Each agent turn can be represented as agent.run or agent.step, with agent.name distinguishing participants. Handoffs should be explicit spans named agent.handoff.

Custom Agents

Create spans directly around your orchestration code. This usually gives the best signal because you know where planning, memory, retrieval, tool execution, and finalization happen.

15. Minimal Rollout Plan

Add the OpenTelemetry SDK to the agent service.
Send traces to a local Collector.
Add a root agent.run span.
Add child spans for LLM calls and tool calls.
Validate trace shape locally.
Add retrieval and memory spans.
Add token and latency attributes.
Add redaction rules.
Export from the Collector to your observability backend.
Add dashboards and alerts.

16. Example Dashboard Panels

Useful panels for agent operations:

Agent run count by agent name
Agent error rate
Agent p50, p95, and p99 latency
LLM latency by model
Input and output tokens by model
Tool call failure rate
Retrieval document count
Handoff count
Estimated model cost
Top failing tools

17. Power BI Dashboard For Usage And Tokens

Power BI should usually read from a queryable store, not directly from raw OpenTelemetry traces. Use OpenTelemetry for observability, then write a normalized usage table for reporting.

Recommended flow:


Agent Application
|
v
OpenTelemetry traces and metrics
|
v
OpenTelemetry Collector
|
v Export
v
Azure Monitor, Log Analytics, SQL, Fabric Lakehouse, or Data Warehouse
|
v Power BI semantic model
v
Power BI dashboard

Recommended Data Source Options

Use one of these patterns:

Azure Monitor or Log Analytics if your traces already go to Azure.
Application Insights if your application telemetry is already centralized there.
Azure SQL Database if you want simple relational reporting.
Microsoft Fabric Lakehouse or Warehouse if you want scalable analytics.
Databricks, Snowflake, or BigQuery if your organization already uses one of them.

For most teams, the easiest production setup is to store one row per LLM request in a table named agent_llm_usage.

Usage Table Schema

Create a reporting table with stable, low-cardinality columns.


CREATE TABLE agent_llm_usage (
    usage_id VARCHAR(100) NOT NULL,
    trace_id VARCHAR(100) NULL,
    span_id VARCHAR(100) NULL,
    timestamp_utc DATETIME2 NOT NULL,
    environment VARCHAR(50) NOT NULL,
    service_name VARCHAR(100) NOT NULL,
    agent_name VARCHAR(100) NOT NULL,
    agent_version VARCHAR(50) NULL,
    session_id VARCHAR(100) NULL,
    user_id_hash VARCHAR(100) NULL,
    model_provider VARCHAR(100) NULL,
    model_name VARCHAR(100) NOT NULL,
    operation_name VARCHAR(50) NOT NULL,
    input_tokens INT NOT NULL DEFAULT 0,
    output_tokens INT NOT NULL DEFAULT 0,
    total_tokens AS (input_tokens + output_tokens),
    estimated_cost_usd DECIMAL(18, 6) NULL,
    duration_ms INT NULL,
    status VARCHAR(30) NOT NULL,
    error_type VARCHAR(200) NULL,
    tool_name VARCHAR(100) NULL,
    retrieval_document_count INT NULL
);

Do not store raw prompts, raw completions, secrets, full document text, or emails in this table. Use hashed user identifiers if user-level reporting is required.

Writing Usage Records

OpenTelemetry spans should still include token attributes such as:


gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.request.model
gen_ai.system
agent.name
agent.session_id

For Power BI, also write a compact business reporting event when each LLM call finishes. In Python, the event can be inserted into SQL, sent to Event Hubs, or written to your analytics pipeline.


def record_llm_usage(response, context):
    usage_record = {
        "usage_id": context.usage_id,
        "trace_id": context.trace_id,
        "span_id": context.span_id,
        "timestamp_utc": context.timestamp_utc,
        "environment": context.environment,
        "service_name": "agent-service",
        "agent_name": context.agent_name,
        "agent_version": context.agent_version,
        "session_id": context.session_id,
        "user_id_hash": context.user_id_hash,
        "model_provider": response.provider,
        "model_name": response.model,
        "operation_name": "chat",
        "input_tokens": response.input_tokens,
        "output_tokens": response.output_tokens,
        "estimated_cost_usd": response.estimated_cost_usd,
        "duration_ms": response.duration_ms,
        "status": "success",
    }

    insert_usage_record(usage_record)

Cost Calculation

Keep model pricing in a separate table so costs can be updated without changing historical usage records.


CREATE TABLE model_pricing (
    model_provider VARCHAR(100) NOT NULL,
    model_name VARCHAR(100) NOT NULL,
    effective_from_utc DATETIME2 NOT NULL,
    input_cost_per_1k_tokens DECIMAL(18, 8) NOT NULL,
    output_cost_per_1k_tokens DECIMAL(18, 8) NOT NULL
);

The estimated cost formula is:


estimated_cost = (input_tokens / 1000 * input_price) + (output_tokens / 1000 * output_price)

You can calculate this in the ingestion pipeline, SQL view, Fabric notebook, or Power BI semantic model. Pipeline or SQL calculation is usually better because all reports use the same cost logic.

Power BI Data Model

Use a simple star schema.

Fact table:

agent_llm_usage

Dimension tables:

dim_date
dim_agent
dim_model
dim_environment
dim_tool

Relationships:

dim_date[date] -> agent_llm_usage[date]
dim_agent[agent_name] -> agent_llm_usage[agent_name]
dim_model[model_name] -> agent_llm_usage[model_name]
dim_environment[environment] -> agent_llm_usage[environment]
dim_tool[tool_name] -> agent_llm_usage[tool_name]

Core DAX Measures

Create these measures in Power BI:


Total Requests = 
COUNTROWS(agent_llm_usage)

Successful Requests = 
CALCULATE(
    COUNTROWS(agent_llm_usage),
    agent_llm_usage[status] = "success"
)

Failed Requests = 
CALCULATE(
    COUNTROWS(agent_llm_usage),
    agent_llm_usage[status] <> "success"
)

Failure Rate = 
DIVIDE([Failed Requests], [Total Requests])

Input Tokens = 
SUM(agent_llm_usage[input_tokens])

Output Tokens = 
SUM(agent_llm_usage[output_tokens])

Total Tokens = 
[Input Tokens] + [Output Tokens]

Estimated Cost USD = 
SUM(agent_llm_usage[estimated_cost_usd])

Average Duration MS = 
AVERAGE(agent_llm_usage[duration_ms])

Average Tokens Per Request = 
DIVIDE([Total Tokens], [Total Requests])

P95 Duration MS = 
PERCENTILEX.INC(
    agent_llm_usage,
    agent_llm_usage[duration_ms],
    0.95
)

Recommended Power BI Pages

Create these report pages:

Executive Overview
Token Usage
Cost Analysis
Agent Performance
Model Performance
Tool And Retrieval Usage
Errors And Reliability

Recommended visuals for the Executive Overview page:

Card: total requests
Card: total tokens
Card: estimated cost
Card: failure rate
Line chart: requests by day
Line chart: tokens by day
Bar chart: cost by agent
Bar chart: tokens by model
Table: top agents by cost, tokens, and failures

Recommended visuals for the Token Usage page:

Line chart: input tokens and output tokens over time
Stacked column chart: tokens by model
Matrix: agent name by model name with total tokens
Slicer: date range

Recommended visuals for the Cost Analysis page:

Slicer: environment
Slicer: agent name
Slicer: model name
Line chart: estimated cost by day
Bar chart: estimated cost by model
Bar chart: estimated cost by agent
Table: session or hashed user groups by cost
KPI: cost per request

Recommended visuals for the Errors And Reliability page:

Card: failed requests
Card: failure rate
Line chart: failures by day
Bar chart: errors by model
Bar chart: errors by tool
Table: error type, agent name, model name, and count

Refresh And Governance

Recommended refresh setup:

Development: manual refresh
Small production workload: scheduled refresh every 1 to 4 hours
High-volume production workload: Direct Lake, DirectQuery, or incremental refresh

Governance recommendations:

Use row-level security if teams should only see their own agents.
Store only hashed user IDs.
Keep prompt and completion content out of the reporting model.
Certify the semantic model once measures are validated.
Document model pricing assumptions and update them when provider pricing changes.

18. Troubleshooting

No Spans In Collector

Check:

Collector is running.
App exports to http://localhost:4318/v1/traces for OTLP HTTP.
Port 4318 is reachable.
The SDK is initialized before the agent runs.
The app shuts down cleanly so batch spans are flushed.

Root Span Exists But Child Spans Are Missing

Check:

Child spans are created inside the active context.
Async operations preserve context.
Spans are ended after work completes.
Exceptions do not skip span.end() in TypeScript.

Too Much Sensitive Data

Check:

No full prompts are added as attributes.
No raw tool outputs are added as attributes.
Logs do not contain secrets.
Collector processors or backend rules redact known sensitive fields.

High Trace Volume

Check:

Add sampling.
Reduce span count for noisy internal steps.
Keep attributes low-cardinality.
Avoid unique values such as full user IDs, emails, or raw queries in indexed attributes.

19. Final Recommended Baseline

A practical first production baseline is:


Root span:
agent.run

Child spans:
agent.plan
llm.chat
tool.call
retrieval.query
memory.read
memory.write
agent.finalize

Required attributes:
service.name
deployment.environment
agent.name
agent.version
gen_ai.system
gen_ai.request.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
tool.name
tool.success

This gives enough visibility to answer the most important operational questions: what happened, where time was spent, which model and tools were used, whether the run failed, and how much token usage it consumed.