DEV Community: She11 QA

GenAI in Test Automation: Accelerating Testing with GitHub Copilot & Agentic Solutions

She11 QA — Tue, 25 Aug 2026 08:48:23 +0000

GenAI in Test Automation: Accelerating Testing with GitHub Copilot & Agentic Solutions

Generative AI is shifting how quality assurance teams design, generate, and maintain test automation suites. By using GitHub Copilot (GHCP) alongside agentic workflows, software teams can significantly cut setup time while maintaining production-grade standards.

Here is a practical framework for leveraging GenAI and agentic solutions to accelerate test automation.

🛠️ Tooling & Core Capabilities

Primary Tool: GitHub Copilot (GHCP)

Framework & Language Support

Frameworks: Selenium, Playwright, Cypress
Languages: Java, JavaScript, Python
Scope: Web & API application testing
Architectural Patterns: Generates modular scripts using Page Object Model (POM), BDD, and Data-Driven Testing paradigms.

⚡ Modes of Operation

Interactive Mode: Human-in-the-loop workflow tailored for prompt refinement and continuous validation.
Agentic Mode: Autonomous end-to-end test generation, execution, and reporting pipeline.
Edit Mode: Quick prompt-driven corrections and inline code modifications.

Crucial Rule: Human-in-the-loop oversight is mandatory to validate, monitor, and correct generated outputs against real-world domain requirements.

📋 Implementation Workflow

Step 1: Pre-requisites & Setup

Configure GitHub Copilot, GHCP, and relevant automation-oriented VSIX extensions in your IDE.
Format test cases in Markdown (Test ID, Objective, Steps, Expected Results, Test Data). Keep individual files under 20 MB.
Collect element locators (JSON, Excel, or Markdown format) and application configurations.

Step 2: Prompt Engineering & Execution

Use prompt templates populated with your framework details, target language, and app context.
Set explicit constraints, rules, and coding style guides within the prompt.
Process prompts through GHCP agentic workflows to generate test cases and code scripts.
Execute generated test suites, evaluate performance metrics, and iteratively refine prompts.

💡 Key Use Cases

Scenario & Case Generation: Automatically derive BDD or non-BDD user stories, scenarios, and test cases.
Manual to Automated Scripting: Rapidly translate manual test documentation into executable code.
Suite Maintenance: Automate locator fixes and script updates whenever application flows change.
Intelligent Prioritization: Target tests based on risk, business criticality, and execution frequency.
Data & Reporting: Automate test data creation and post-execution reporting.

📌 Best Practices Checklist

Human Oversight: Always inspect AI-generated code for edge cases, logical errors, and adherence to company coding standards.
Track Impact Metrics: Measure success by tracking output accuracy, test coverage improvements, bug reduction rates, and overall engineering time saved.
Document Structured Inputs: Provide clean input context (BRD documents, locator files, code samples) to minimize hallucination risks during script generation.

Essential Git Commands for Feature Branch Workflow

She11 QA — Tue, 25 Aug 2026 08:40:14 +0000

How to Switch, Sync, and Merge Git Branches in VS Code

Keeping your feature or QA branch up to date with the main codebase is essential for preventing merge conflicts later down the line. Here is a quick reference guide on how to switch branches, fetch new remotes, and pull the latest changes into your branch using VS Code and Git terminal.

1. How to Switch Branches in VS Code

Using the Command Palette:

Press Ctrl + Shift + P (Windows/Linux) or Cmd + Shift + P (Mac).
Type and select Git: Checkout to...
Select your target branch (e.g., your-feature-branch).

Using the Terminal:

If the branch hasn't been fetched locally yet:

# Fetch the specific branch from remote
git fetch origin your-feature-branch

# Switch to the branch
git checkout your-feature-branch

2. Syncing Your Branch with the Latest Main Code

Follow these 5 steps to update your local working branch with the latest upstream code from main:

Step 1: Switch to the Main Branch

git checkout main

Step 2: Pull Remote Changes

git pull origin main

Step 3: Return to Your Feature Branch

git checkout your-feature-branch

Step 4: Merge Main into Your Branch

git merge main

Handling Conflicts: If VS Code highlights merge conflicts, edit the files to resolve them, then stage and commit:
git add .
git commit -m "Merge main into your-feature-branch"

Step 5: Push Updates to Remote

git push origin your-feature-branch

Quick Summary Checklist

git checkout main → Jump to main
git pull origin main → Get latest changes
git checkout → Return to your working branch
git merge main → Bring latest main changes into your branch
git push origin → Push updated work to remote

Setting Up a Playwright JavaScript BDD Automation Framework with Cucumber

She11 QA — Tue, 25 Aug 2026 08:31:19 +0000

Setting up a scalable test automation framework using Playwright JavaScript with BDD (Behavior-Driven Development) via Cucumber helps streamline testing and maintain consistency across complex test suites.

Here is a step-by-step guide to onboarding and initializing this test environment.

Prerequisites & Tools

Make sure you have the following software installed:

Recommended VS Code Extensions

Open the Extensions tab in VS Code (Ctrl + Shift + X / Cmd + Shift + X) and install:

Cucumber (Gherkin) Support — Syntax highlighting and step definition navigation for .feature files.
JavaScript / TypeScript Support — Auto-completion and language tooling.
ESLint (Optional) — Code linting to maintain style rules and clean syntax.

Setup & Initialization Commands

Clone the repository, navigate to your root workspace, and execute the following commands in your integrated terminal:

1. Install required project dependencies

npm install

2. Download and configure required Playwright browser binaries

npx playwright install --with-deps

3. Install Excel utility packages for handling test data

npm install exceljs --save

4. Optional security/vulnerability audit

npm audit

5. Launch tests in Playwright's UI Mode for debugging

npm run test/ui

Best Practices for BDD Automation Suites

Align Feature Files with Test Specifications: Always map .feature Gherkin steps directly against your functional requirements sheet to prevent missing coverage.
Centralize Documentation: Keep detailed architecture notes and setup guides inside the README.md file and a dedicated /doc folder for faster onboarding.
Data Handling: Use structured files (such as Excel via exceljs or JSON) to separate test scripts from dynamic test data.

Fixing PowerShell Script Execution Policy Issue When Activating Python venv

She11 QA — Tue, 25 Aug 2026 08:26:09 +0000

When setting up a Python virtual environment on Windows, running .\venv\Scripts\activate in PowerShell often throws a script execution policy error (such as PSSecurityException or cannot be loaded because running scripts is disabled on this system).

Here is a quick walkthrough to resolve this issue safely without disabling your system's global security policies.

The Cause

Windows PowerShell restricts script execution by default under the Restricted policy to prevent malicious scripts from executing. Because virtual environment activation scripts (Activate.ps1) are local scripts, PowerShell blocks them.

Step-by-Step Fix

1. Open PowerShell

Launch PowerShell (or your integrated terminal in VS Code).

2. Update Execution Policy for Current User

Run the following command to allow locally generated scripts to execute for your Windows profile only:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Why -Scope CurrentUser?

Scope limiting ensures you only grant permission to your current user session, eliminating the need to alter global system policies or risk security system-wide.

Verification Workflow

Once the policy is set, you can create and activate your environment seamlessly:


# 1. Create your virtual environment
python -m venv venv

# 2. Activate the virtual environment
.\venv\Scripts\activate

# Output: Your prompt will now show the environment prefix:
# (.venv) PS C:\your-project-path>

Summary

Error: PowerShell blocks Activate.ps1.
Fix: Set -ExecutionPolicy RemoteSigned scoped to -Scope CurrentUser.
Security: Keeps remote unsigned scripts blocked while allowing local development tools to function smoothly.

Setting Up Playwright with JavaScript for Test Automation: Prerequisites & VS Code Extensions

She11 QA — Tue, 25 Aug 2026 08:20:04 +0000

Setting up a new test automation framework can be tricky if you miss essential tools and extensions. Below is a structured checklist to get your environment fully prepared for automation testing using Playwright with JavaScript.

1. Framework & Core Requirements

Framework Type: Playwright (JavaScript)
Configuration: settings.xml (shared via your internal communications channel)
Repository: xlc-portfolio-tech-shared-services/xlc-app-kc-test-automation

2. Development Environment Setup

IDE: Install Visual Studio Code.
Configuration Files: Ensure settings.xml and Framework.zip are downloaded and properly configured in your root workspace.

3. Recommended VS Code Extensions

To streamline writing BDD-style tests, running Playwright suites, and debugging, install the following extensions:

Playwright & Testing:
- Playwright Test Runner
- Playwright Test for VSCode
BDD & Gherkin / Cucumber Support:
- Cucumber
- Cucumber (Gherkin) Full Support
- Snippets and Syntax Highlight for Gherkin (Cucumber)
AI & GitHub Integration:
- GitHub Copilot
- GitHub Copilot Chat
- GitHub Actions
Code Quality & Utilities:
- ESLint
- XML Tools
- Rainbow CSV
- Excel Viewer
- Microsoft Edge Tools for VS Code

💡 Pro Tip

Make sure all team members maintain identical VS Code extension sets to prevent formatting inconsistencies and ensure smooth local test runs!

Building an AI Test Automation Factory: How We Reduced Automation Effort by 78% with Multi-Agent Systems & MCP

She11 QA — Sat, 22 Aug 2026 09:25:45 +0000

Traditional test automation frameworks often carry heavy maintenance costs, slow release cycles, and high knowledge dependency. By transitioning from standard script creation to a governed AI Test Automation Factory, engineering teams can shift their focus from writing boilerplate code to high-value validation and architectural optimization.

Here is an architectural breakdown of how multi-agent AI systems, governed telemetry, and Model Context Protocol (MCP) transform enterprise quality engineering.

The Problem: The 45-Hour Manual Bottleneck

Building a end-to-end BDD automation suite manually requires significant time per user story—often taking up to 45 hours across five distinct steps:

Context Generation & Requirements Review (~8 hrs)
Manual Test Case Design (~9 hrs)
Cucumber Feature File Creation (~8 hrs)
Page Object Model Generation (~8 hrs)
Step Definition Implementation (~10 hrs)

This traditional workflow creates coverage gaps, inconsistent code quality, and defect leakage.

The Solution: Multi-Agent AI Automation Pipeline

Instead of relying on single prompts, an AI Test Automation Factory routes requirement artifacts (BRDs / User Stories) through specialized agents:


[BRD / User Story] 
       │
       ▼
[Context Agent] ──► [Test Case Agent] ──► [Feature File Agent]
                                                  │
[Automation Suite] ◄── [Step Definition Agent] ◄── [Page Object Agent]

Context Agent: Parses acceptance criteria and enterprise domain knowledge.
Test Case Agent: Auto-generates exhaustive test scenario matrices.
Feature File Agent: Drafts standardized BDD Cucumber feature files.
Page Object & Step Def Agents: Constructs clean design patterns (POM) and matching step implementations.

Measurable ROI: Before vs. After AI

By replacing manual generation with agentic workflows, the effort to automate a scenario drops from 45 hours to 9.5 hours:

Phase	Manual Effort	AI-Driven Effort	Time Saved
Context Generation	8 hrs	2 hrs	75%
Test Design	9 hrs	2 hrs	78%
Feature File Creation	8 hrs	0.5 hrs	94%
Page Object Creation	8 hrs	2 hrs	75%
Step Definitions	10 hrs	3 hrs	70%
Total Effort	45 hrs	9.5 hrs	78% Reduction

Key Business Metrics:

Productivity Multiplier: 4X Faster Delivery
Test Coverage: Increased from 65% to 90%
Defect Leakage: Reduced from 12% to 5%
Overall Cost Footprint: Scaled down to 22% of original baseline

AI Governance & Observability

Enterprise deployment requires strict guardrails around LLM usage. A telemetry layer sits between the agents and executive reporting dashboards to monitor performance in real time:

Token & Usage Tracking: Daily audit trails for prompt/completion token consumption.
Cost & Adoption Monitoring: Sprint-by-sprint metrics tracking user engagement vs. API spend.
Executive Visibility: Real-time Power BI reporting reflecting total hours saved and generated code assets.

The Future: Autonomous Testing via MCP

The future of QA lies in moving from AI-Assisted generation to Autonomous Self-Healing Execution. Leveraging the Model Context Protocol (MCP) enables seamless enterprise knowledge integration, allowing agents to directly query system context, adjust broken locators automatically, and deliver a fully autonomous QA pipeline.

How to Configure Full Parallel Execution in TestNG via Maven (with Dynamic Thread Control)

She11 QA — Sat, 22 Aug 2026 09:16:16 +0000

Running automation test suites sequentially can quickly become a bottleneck in CI/CD pipelines. To optimize execution efficiency, you can enable full parallel execution using TestNG and Maven, allowing you to control thread counts dynamically right from the command line without modifying XML files every time.

Here is a step-by-step guide on how to configure your framework for flexible parallel execution, along with a comparison of native TestNG execution versus custom Allocator/Run Manager approaches.

Step-by-Step Configuration

1. Update testng_regression.xml

Modify the tag to set your default parallel mode and thread count. You can set parallel to methods, classes, tests, or instances based on your architecture:


<suite name="Regression" parallel="methods" thread-count="10">

2. Configure pom.xml for Dynamic Control

To override these settings at runtime without touching the codebase, map properties inside the block of the maven-surefire-plugin:


<configuration>
    <parallel>${parallel}</parallel>
    <threadCount>${threadCount}</threadCount>
</configuration>

Command-Line Usage Examples

Once configured, you can pass parameters dynamically via Maven:

Default Run (10 threads):

   mvn clean test -P runTestNGTests

Dynamic Thread Count (15 threads):


   mvn clean test -P runTestNGTests -DthreadCount=15

Change Parallel Mode at Runtime:


   mvn clean test -P runTestNGTests -Dparallel=classes -DthreadCount=15

High-Throughput Run (Match CPU cores, e.g., 24 threads):


   mvn clean test -P runTestNGTests -DthreadCount=24

Comparison: Native TestNG vs. Custom Excel Allocator

If your framework currently uses a custom Excel-based Run Manager alongside native TestNG, here is how the two approaches compare:

Feature / Aspect	Custom Allocator (Run Manager)	Native TestNG
Entry Point	allocator.Allocator.main() via Maven exec plugin	maven-surefire-plugin running testng.xml
Test Selection	Reads Excel sheets via custom properties	Reads testng_regression.xml classes/methods
Thread Management	Managed via custom ExecutorService	Native TestNG thread pool (parallel + thread-count)
Execution Command	mvn clean test -P runAllocator	mvn clean test -P runTestNGTests
Pros	Data-driven via Excel; multi-sheet aggregation support	Lightweight, faster startup, zero Excel dependencies
Cons	Requires global property tuning; code changes for structural updates	Restricted to TestNG parallel modes (methods/classes)

Which Approach Should You Choose?

Choose Custom Allocator if your suite relies heavily on fine-grained Excel-driven iteration control or multi-sheet test scheduling.
Choose Native TestNG if you want a cleaner footprint, faster execution loops without file-parsing overhead, and straightforward CLI thread scaling.

Complete End-to-End OpenTelemetry Setup Guide for AI Agents & Power BI Observability

She11 QA — Fri, 21 Aug 2026 09:14:14 +0000

OpenTelemetry E2E Setup Guide for AI Agents

This guide shows how to set up end-to-end OpenTelemetry observability for AI agents, from local tracing to production export. It covers the OpenTelemetry Collector, Python and Node.js instrumentation, agent-specific spans, LLM and tool-call tracing, validation, and production recommendations.

1. What You Are Instrumenting

For an AI agent, the most useful trace structure is a top-level agent run with child spans for every meaningful operation.


agent.run
|-- agent.plan
|-- llm.chat
|-- tool.call: knowledge_search
|-- retrieval.query
|-- memory.read
|-- llm.chat
|-- memory.write
`-- agent.finalize

At minimum, trace these operations:

agent.run : one full user request or autonomous task
llm.chat : each model call
tool.call : each tool invocation
retrieval.query : vector search, database lookup, or document retrieval
memory.read : agent memory lookup
memory.write : agent memory update
agent.handoff : transfer to another agent or human
agent.finalize : final response construction

2. Recommended Architecture


Agent Application
|
v
OTLP traces, metrics, logs
|
v
OpenTelemetry Collector
|
v Exporter
v
Observability Backend

Common observability backends include:

Jaeger
Grafana Tempo
Honeycomb
Datadog
New Relic
Azure Monitor
AWS X-Ray
Google Cloud Trace
Elastic Observability

For local development, you can start with the OpenTelemetry Collector plus a logging exporter. For production, send data from the Collector to your preferred backend.

3. Local OpenTelemetry Collector Setup

Create a file named otel-collector-config.yaml:


receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Run the Collector with Docker on Windows PowerShell:


docker run --rm -p 4317:4317 -p 4318:4318 \
  -v "${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:latest

Run the Collector on macOS or Linux:

docker run --rm -p 4317:4317 -p 4318:4318 \
  -v "$PWD/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:latest

The local OTLP endpoints are:

HTTP: http://localhost:4318/v1/traces
gRPC: localhost:4317

4. Python Agent Setup

Install dependencies:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

Create telemetry.py:


from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def configure_telemetry() -> None:
    resource = Resource.create({
        "service.name": "agent-service",
        "service.version": "1.0.0",
        "deployment.environment": "local",
    })

    provider = TracerProvider(resource=resource)
    exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

Instrument the agent:


from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from telemetry import configure_telemetry

configure_telemetry()
tracer = trace.get_tracer("agent-service")

def run_agent(user_input: str) -> str:
    with tracer.start_as_current_span("agent.run") as span:
        span.set_attribute("agent.name", "support-agent")
        span.set_attribute("agent.version", "1.0.0")
        span.set_attribute("agent.input.length", len(user_input))

        try:
            with tracer.start_as_current_span("agent.plan") as plan_span:
                plan = create_plan(user_input)
                plan_span.set_attribute("agent.plan.steps", len(plan))

            with tracer.start_as_current_span("llm.chat") as llm_span:
                llm_span.set_attribute("gen_ai.system", "openai")
                llm_span.set_attribute("gen_ai.operation.name", "chat")
                llm_span.set_attribute("gen_ai.request.model", "gpt-4.1")

                llm_response = call_llm(user_input)

                llm_span.set_attribute("gen_ai.response.model", "gpt-4.1")
                llm_span.set_attribute("gen_ai.usage.input_tokens", llm_response.input_tokens)
                llm_span.set_attribute("gen_ai.usage.output_tokens", llm_response.output_tokens)

            with tracer.start_as_current_span("tool.call") as tool_span:
                tool_span.set_attribute("tool.name", "knowledge_search")
                tool_span.set_attribute("tool.call_id", llm_response.tool_call_id)

                documents = search_documents(llm_response.tool_query)

                tool_span.set_attribute("tool.success", True)
                tool_span.set_attribute("retrieval.document_count", len(documents))

            with tracer.start_as_current_span("agent.finalize") as final_span:
                final_answer = produce_answer(llm_response, documents)
                final_span.set_attribute("agent.output.length", len(final_answer))

            return final_answer

        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            raise

5. Node.js or TypeScript Agent Setup

Install dependencies:


npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

Create telemetry.ts:

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

export const telemetrySdk = new NodeSDK({
  serviceName: "agent-service",
  traceExporter: new OTLPTraceExporter({
    url: "http://localhost:4318/v1/traces",
  }),
});

export async function startTelemetry() {
  await telemetrySdk.start();
}

export async function stopTelemetry() {
  await telemetrySdk.shutdown();
}

Instrument the agent:

import { SpanStatusCode, trace } from "@opentelemetry/api";
import { startTelemetry, stopTelemetry } from "./telemetry";

const tracer = trace.getTracer("agent-service");

export async function runAgent(userInput: string): Promise<string> {
  return tracer.startActiveSpan("agent.run", async span => {
    span.setAttribute("agent.name", "support-agent");
    span.setAttribute("agent.version", "1.0.0");
    span.setAttribute("agent.input.length", userInput.length);

    try {
      const plan = await tracer.startActiveSpan("agent.plan", async planSpan => {
        const result = await createPlan(userInput);
        planSpan.setAttribute("agent.plan.steps", result.length);
        planSpan.end();
        return result;
      });

      const llmResponse = await tracer.startActiveSpan("llm.chat", async llmSpan => {
        llmSpan.setAttribute("gen_ai.system", "openai");
        llmSpan.setAttribute("gen_ai.operation.name", "chat");
        llmSpan.setAttribute("gen_ai.request.model", "gpt-4.1");

        const response = await callLlm(userInput, plan);

        llmSpan.setAttribute("gen_ai.response.model", "gpt-4.1");
        llmSpan.setAttribute("gen_ai.usage.input_tokens", response.inputTokens);
        llmSpan.setAttribute("gen_ai.usage.output_tokens", response.outputTokens);
        llmSpan.end();

        return response;
      });

      const documents = await tracer.startActiveSpan("tool.call", async toolSpan => {
        toolSpan.setAttribute("tool.name", "knowledge_search");
        toolSpan.setAttribute("tool.call_id", llmResponse.toolCallId);

        const result = await searchDocuments(llmResponse.toolQuery);

        toolSpan.setAttribute("tool.success", true);
        toolSpan.setAttribute("retrieval.document_count", result.length);
        toolSpan.end();

        return result;
      });

      const finalAnswer = await tracer.startActiveSpan("agent.finalize", async finalSpan => {
        const answer = await produceAnswer(llmResponse, documents);
        finalSpan.setAttribute("agent.output.length", answer.length);
        finalSpan.end();
        return answer;
      });

      span.end();
      return finalAnswer;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: String(error),
      });
      span.end();
      throw error;
    }
  });
}

async function main() {
  await startTelemetry();

  try {
    await runAgent("How do I reset my password?");
  } finally {
    await stopTelemetry();
  }
}

6. Recommended Span Attributes

Use OpenTelemetry semantic conventions where possible.

GenAI Attributes

gen_ai.system
gen_ai.operation.name
gen_ai.request.model
gen_ai.response.model
gen_ai.request.temperature
gen_ai.request.max_tokens
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens

Agent Attributes

agent.name
agent.version
agent.run_id
agent.session_id
agent.step.name
agent.step.index
agent.output.length
agent.input.length

Tool Attributes

tool.name
tool.call_id
tool.success
tool.error.type
tool.retry_count

Retrieval Attributes

retrieval.system
retrieval.index.name
retrieval.query_count
retrieval.document_count
retrieval.top_k

Memory Attributes

memory.operation
memory.scope
memory.result_count

7. What Not To Capture

Do not capture sensitive or high-risk data as span attributes unless your organization has explicit approval, redaction, access control, and retention policies.

Avoid storing:

Full prompts
Full model responses
User secrets
API keys or tokens
Raw documents
Email addresses
Payment information
Health data
Authentication headers
Full tool outputs

Prefer safe metadata:

prompt length
response length
token counts
model name
tool name
status
latency
retry count
document count

8. Metrics To Add

Traces show what happened for one request. Metrics show aggregate behavior.

Recommended metrics:

agent.run.count
agent.run.duration
agent.run.error.count
llm.request.count
llm.request.duration
llm.token.input.count
llm.token.output.count
tool.call.count
tool.call.duration
tool.call.error.count
retrieval.query.count
retrieval.document.count
agent.handoff.count

Start with traces first, then add metrics once your span structure is stable.

9. Logging Setup

Use logs for discrete application events, but correlate them with traces.

Recommended log fields:

trace_id
span_id
agent_run_id
agent.name
event.name
status
error.type
error.message

Keep logs redacted. Avoid logging raw prompts, completions, credentials, or document bodies.

10. Validation Checklist

Use this checklist to verify the setup end to end.

Start the OpenTelemetry Collector.
Run one local agent request.
Confirm the Collector receives spans.
Confirm the root span is named agent.run.
Confirm LLM calls appear as llm.chat child spans.
Confirm tool calls appear as tool.call child spans.
Confirm errors are recorded on failed spans.
Confirm token usage attributes appear when available.
Confirm sensitive content is not exported.
Confirm traces reach your observability backend.

11. Production Collector Example

For production, route data through the Collector and export to your backend.


otlp:
  protocols:
    http:
    grpc:

processors:
  batch:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  otlphttp:
    endpoint: https://your-observability-backend.example.com
    headers:
      api-key: ${OBSERVABILITY_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

Set environment variables securely through your deployment platform, secret manager, or CI/CD system.

12. Production Hardening

Before production rollout:

Add sampling if trace volume is high.
Redact sensitive inputs and outputs.
Add service names and versions.
Add deployment environment attributes.
Use secure OTLP endpoints.
Store backend API keys in a secret manager.
Configure retention policies.
Add alerts for agent error rate and high latency.
Track model cost through token metrics.
Validate compliance requirements before exporting AI data.

13. Sampling Guidance

For development, sample everything.

For production, consider:

100% sampling for errors
100% sampling for low-volume critical workflows
Lower probabilistic sampling for high-volume successful requests
Tail sampling when supported by your backend or Collector distribution

Example parent-based trace ID ratio sampling in Python:


from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

provider = TracerProvider(
    sampler=ParentBased(TraceIdRatioBased(0.10))
)

This samples roughly 10% of new traces while preserving parent-child trace consistency.

14. Framework-Specific Notes

LangChain

Instrument the chain or agent executor as agent.run, then wrap model calls, retrievers, and tools as child spans. If you use callbacks, create spans inside callback handlers.

Semantic Kernel

Trace kernel invocation as agent.run, function calls as tool.call, planner execution as agent.plan, and AI service calls as llm.chat.
AutoGen or Multi-Agent Systems
Use one trace for the full multi-agent workflow. Each agent turn can be represented as agent.run or agent.step, with agent.name distinguishing participants. Handoffs should be explicit spans named agent.handoff.

Custom Agents

Create spans directly around your orchestration code. This usually gives the best signal because you know where planning, memory, retrieval, tool execution, and finalization happen.

15. Minimal Rollout Plan

Add the OpenTelemetry SDK to the agent service.
Send traces to a local Collector.
Add a root agent.run span.
Add child spans for LLM calls and tool calls.
Validate trace shape locally.
Add retrieval and memory spans.
Add token and latency attributes.
Add redaction rules.
Export from the Collector to your observability backend.
Add dashboards and alerts.

16. Example Dashboard Panels

Useful panels for agent operations:

Agent run count by agent name
Agent error rate
Agent p50, p95, and p99 latency
LLM latency by model
Input and output tokens by model
Tool call failure rate
Retrieval document count
Handoff count
Estimated model cost
Top failing tools

17. Power BI Dashboard For Usage And Tokens

Power BI should usually read from a queryable store, not directly from raw OpenTelemetry traces. Use OpenTelemetry for observability, then write a normalized usage table for reporting.

Recommended flow:


Agent Application
|
v
OpenTelemetry traces and metrics
|
v
OpenTelemetry Collector
|
v Export
v
Azure Monitor, Log Analytics, SQL, Fabric Lakehouse, or Data Warehouse
|
v Power BI semantic model
v
Power BI dashboard

Recommended Data Source Options

Use one of these patterns:

Azure Monitor or Log Analytics if your traces already go to Azure.
Application Insights if your application telemetry is already centralized there.
Azure SQL Database if you want simple relational reporting.
Microsoft Fabric Lakehouse or Warehouse if you want scalable analytics.
Databricks, Snowflake, or BigQuery if your organization already uses one of them.

For most teams, the easiest production setup is to store one row per LLM request in a table named agent_llm_usage.

Usage Table Schema

Create a reporting table with stable, low-cardinality columns.


CREATE TABLE agent_llm_usage (
    usage_id VARCHAR(100) NOT NULL,
    trace_id VARCHAR(100) NULL,
    span_id VARCHAR(100) NULL,
    timestamp_utc DATETIME2 NOT NULL,
    environment VARCHAR(50) NOT NULL,
    service_name VARCHAR(100) NOT NULL,
    agent_name VARCHAR(100) NOT NULL,
    agent_version VARCHAR(50) NULL,
    session_id VARCHAR(100) NULL,
    user_id_hash VARCHAR(100) NULL,
    model_provider VARCHAR(100) NULL,
    model_name VARCHAR(100) NOT NULL,
    operation_name VARCHAR(50) NOT NULL,
    input_tokens INT NOT NULL DEFAULT 0,
    output_tokens INT NOT NULL DEFAULT 0,
    total_tokens AS (input_tokens + output_tokens),
    estimated_cost_usd DECIMAL(18, 6) NULL,
    duration_ms INT NULL,
    status VARCHAR(30) NOT NULL,
    error_type VARCHAR(200) NULL,
    tool_name VARCHAR(100) NULL,
    retrieval_document_count INT NULL
);

Do not store raw prompts, raw completions, secrets, full document text, or emails in this table. Use hashed user identifiers if user-level reporting is required.

Writing Usage Records

OpenTelemetry spans should still include token attributes such as:


gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.request.model
gen_ai.system
agent.name
agent.session_id

For Power BI, also write a compact business reporting event when each LLM call finishes. In Python, the event can be inserted into SQL, sent to Event Hubs, or written to your analytics pipeline.


def record_llm_usage(response, context):
    usage_record = {
        "usage_id": context.usage_id,
        "trace_id": context.trace_id,
        "span_id": context.span_id,
        "timestamp_utc": context.timestamp_utc,
        "environment": context.environment,
        "service_name": "agent-service",
        "agent_name": context.agent_name,
        "agent_version": context.agent_version,
        "session_id": context.session_id,
        "user_id_hash": context.user_id_hash,
        "model_provider": response.provider,
        "model_name": response.model,
        "operation_name": "chat",
        "input_tokens": response.input_tokens,
        "output_tokens": response.output_tokens,
        "estimated_cost_usd": response.estimated_cost_usd,
        "duration_ms": response.duration_ms,
        "status": "success",
    }

    insert_usage_record(usage_record)

Cost Calculation

Keep model pricing in a separate table so costs can be updated without changing historical usage records.


CREATE TABLE model_pricing (
    model_provider VARCHAR(100) NOT NULL,
    model_name VARCHAR(100) NOT NULL,
    effective_from_utc DATETIME2 NOT NULL,
    input_cost_per_1k_tokens DECIMAL(18, 8) NOT NULL,
    output_cost_per_1k_tokens DECIMAL(18, 8) NOT NULL
);

The estimated cost formula is:


estimated_cost = (input_tokens / 1000 * input_price) + (output_tokens / 1000 * output_price)

You can calculate this in the ingestion pipeline, SQL view, Fabric notebook, or Power BI semantic model. Pipeline or SQL calculation is usually better because all reports use the same cost logic.

Power BI Data Model

Use a simple star schema.

Fact table:

agent_llm_usage

Dimension tables:

dim_date
dim_agent
dim_model
dim_environment
dim_tool

Relationships:

dim_date[date] -> agent_llm_usage[date]
dim_agent[agent_name] -> agent_llm_usage[agent_name]
dim_model[model_name] -> agent_llm_usage[model_name]
dim_environment[environment] -> agent_llm_usage[environment]
dim_tool[tool_name] -> agent_llm_usage[tool_name]

Core DAX Measures

Create these measures in Power BI:


Total Requests = 
COUNTROWS(agent_llm_usage)

Successful Requests = 
CALCULATE(
    COUNTROWS(agent_llm_usage),
    agent_llm_usage[status] = "success"
)

Failed Requests = 
CALCULATE(
    COUNTROWS(agent_llm_usage),
    agent_llm_usage[status] <> "success"
)

Failure Rate = 
DIVIDE([Failed Requests], [Total Requests])

Input Tokens = 
SUM(agent_llm_usage[input_tokens])

Output Tokens = 
SUM(agent_llm_usage[output_tokens])

Total Tokens = 
[Input Tokens] + [Output Tokens]

Estimated Cost USD = 
SUM(agent_llm_usage[estimated_cost_usd])

Average Duration MS = 
AVERAGE(agent_llm_usage[duration_ms])

Average Tokens Per Request = 
DIVIDE([Total Tokens], [Total Requests])

P95 Duration MS = 
PERCENTILEX.INC(
    agent_llm_usage,
    agent_llm_usage[duration_ms],
    0.95
)

Recommended Power BI Pages

Create these report pages:

Executive Overview
Token Usage
Cost Analysis
Agent Performance
Model Performance
Tool And Retrieval Usage
Errors And Reliability

Recommended visuals for the Executive Overview page:

Card: total requests
Card: total tokens
Card: estimated cost
Card: failure rate
Line chart: requests by day
Line chart: tokens by day
Bar chart: cost by agent
Bar chart: tokens by model
Table: top agents by cost, tokens, and failures

Recommended visuals for the Token Usage page:

Line chart: input tokens and output tokens over time
Stacked column chart: tokens by model
Matrix: agent name by model name with total tokens
Slicer: date range

Recommended visuals for the Cost Analysis page:

Slicer: environment
Slicer: agent name
Slicer: model name
Line chart: estimated cost by day
Bar chart: estimated cost by model
Bar chart: estimated cost by agent
Table: session or hashed user groups by cost
KPI: cost per request

Recommended visuals for the Errors And Reliability page:

Card: failed requests
Card: failure rate
Line chart: failures by day
Bar chart: errors by model
Bar chart: errors by tool
Table: error type, agent name, model name, and count

Refresh And Governance

Recommended refresh setup:

Development: manual refresh
Small production workload: scheduled refresh every 1 to 4 hours
High-volume production workload: Direct Lake, DirectQuery, or incremental refresh

Governance recommendations:

Use row-level security if teams should only see their own agents.
Store only hashed user IDs.
Keep prompt and completion content out of the reporting model.
Certify the semantic model once measures are validated.
Document model pricing assumptions and update them when provider pricing changes.

18. Troubleshooting

No Spans In Collector

Check:

Collector is running.
App exports to http://localhost:4318/v1/traces for OTLP HTTP.
Port 4318 is reachable.
The SDK is initialized before the agent runs.
The app shuts down cleanly so batch spans are flushed.

Root Span Exists But Child Spans Are Missing

Check:

Child spans are created inside the active context.
Async operations preserve context.
Spans are ended after work completes.
Exceptions do not skip span.end() in TypeScript.

Too Much Sensitive Data

Check:

No full prompts are added as attributes.
No raw tool outputs are added as attributes.
Logs do not contain secrets.
Collector processors or backend rules redact known sensitive fields.

High Trace Volume

Check:

Add sampling.
Reduce span count for noisy internal steps.
Keep attributes low-cardinality.
Avoid unique values such as full user IDs, emails, or raw queries in indexed attributes.

19. Final Recommended Baseline

A practical first production baseline is:


Root span:
agent.run

Child spans:
agent.plan
llm.chat
tool.call
retrieval.query
memory.read
memory.write
agent.finalize

Required attributes:
service.name
deployment.environment
agent.name
agent.version
gen_ai.system
gen_ai.request.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
tool.name
tool.success

This gives enough visibility to answer the most important operational questions: what happened, where time was spent, which model and tools were used, whether the run failed, and how much token usage it consumed.

Setting Up End-to-End PDF Validation in Playwright and Cucumber BDD

She11 QA — Tue, 18 Aug 2026 13:08:36 +0000

Validating generated PDFs in automated end-to-end tests can be tricky. Here is a comprehensive guide on how to handle PDF downloading, parsing, and structured data assertion in a Web UI automation framework using Playwright, Cucumber (BDD), pdf-parse, and ajv schema validation.

Overview & Workflow

The automated end-to-end PDF validation flow follows these steps:

Navigate through the UI to trigger a PDF download.
Capture the download event using Playwright and store the PDF locally in reports/downloads/pdf/.
Extract and parse text from the downloaded file.
Perform assertions using static text matches, dynamic JSON data paths, or Regex extractions verified against JSON Schemas.

1. Prerequisites & Dependencies

Ensure your project has pdf-parse and ajv installed:

{
  "dependencies": {
    "pdf-parse": "^1.1.1",
    "ajv": "^8.17.1"
  }
}

Enable download handling in your Playwright configuration (setup/hooks.js):


// Browser context setup
acceptDownloads: true

2. Core Utility Helper

Create a helper module (utils/PdfHelper.js) to handle file downloads, text normalization, and schema validation:


// Core functions implemented in PdfHelper.js
- downloadPdfFromSelector(page, selector, options) // Captures Playwright download event & saves file
- parsePdf(filePath) / parsePdfBuffer(buffer)       // Normalizes raw text output
- assertTextContains(text, expectedValues, options) // Handles case-insensitive and whitespace-normalized checks
- extractBySchema(text, extractionRules)          // Extracts fields using JS Regex
- validateWithSchema(fieldData, schemaPath)        // Validates structure with AJV

3. BDD Step Definitions

Wire your Cucumber steps (step-definitions/ui/pdfValidationSteps.js) to interact with the helper and scenario context:


When the user downloads a PDF from selector "<selector>"
When the user parses the downloaded PDF
Then the downloaded PDF should contain text "<text>"
Then the downloaded PDF should contain values from json "<jsonPath>" at path "<dataPath>"
When the user extracts PDF fields using rules from "<rulesPath>"
Then the extracted PDF fields should match schema "<schemaPath>"
Then the downloaded PDF page count should be at least <n>

4. Extraction Rules & Schema Validation Example

For complex documents, define regex patterns to extract fields and validate them against a schema.

Regex Rules (test-data/json/pdfExtractionRules.sample.json):


{
  "rules": [
    {
      "key": "bondNumber",
      "pattern": "Bond\\s*Number\\s*:\\s*([A-Z0-9-]+)",
      "flags": "i",
      "group": 1
    }
  ]
}

Feature File Integration:

And the user downloads a PDF from selector "<stable-selector>"
And the user parses the downloaded PDF
Then the downloaded PDF should contain values from json "test-data/json/BondTestData.json" at path "accountName"
And the downloaded PDF page count should be at least 1

5. Running Tests & Best Practices

Add test scripts to your package.json:

{
  "scripts": {
    "test:ui:pdf:dry": "npx cucumber-js features/ui/pdfValidation.feature --import setup/hooks.js --import setup/assertions.js --import step-definitions/ui --dry-run",
    "test:ui:pdf:run": "npx cucumber-js features/ui/pdfValidation.feature --import setup/hooks.js --import setup/assertions.js --import step-definitions/ui -f progress -f json:reports/cucumber_report.json --parallel 1"
  }
}

Key Recommendations:

Selectors: Prefer stable data-testid attributes over text-only selectors.
Environment Safety: Store expected test data in JSON files rather than hardcoding values.
Security & Cleanup: Do not keep sensitive PDF payloads longer than necessary, and clean up downloads in post-test runs.

Checklist: Onboarding End-to-End Automation Frameworks to Harness CI

She11 QA — Tue, 18 Aug 2026 09:44:13 +0000

Successfully onboarding an automated test suite to Harness CI requires configuring infrastructure placeholders, secrets, pipelines, and branch protection rules.

Here is a 10-step checklist to help you onboard your end-to-end (E2E) automation pipelines seamlessly.

Step 1: Replace Infrastructure Placeholders

Ensure your pipeline YAML definitions (e.g., .harness/e2e-poc.yaml and .harness/e2e-regression-parallel.yaml) contain your specific environment values:

ORG_ID: Harness Organization Identifier
PROJECT_ID: Harness Project Identifier
GIT_CONNECTOR: Harness Git Connector for GitHub Enterprise access
APP_REPO_NAME: Target repository in owner/repo format
K8S_CONNECTOR: Kubernetes connector for build infrastructure
K8S_NAMESPACE: Kubernetes namespace where build pods run

Step 2: Configure Environment Secrets

In Harness, set up the following runtime secrets:

CONNECT_URL
CONNECT_USERNAME
CONNECT_PASSWORD

Step 3: Setup PR Validation Pipeline

Import your short-run pipeline YAML into Harness.
Save it as your PR Validation Pipeline.
Run a manual validation test using runtime overrides:
- TargetEnv = qa
- cucumberTags = @smoke

Step 4: Verify Artifact Generation

Confirm that the initial execution correctly generates and uploads all required outputs:

JUnit Report: reports/junit-report.xml
Test Reports: reports/**
Failure Artifacts: test-results/** (screenshots, traces)

Step 5: Setup Nightly Parallel Pipeline

Import your parallel pipeline YAML into Harness.
Save it as your Nightly Regression Pipeline.
Run a manual validation test with target concurrency parameters:
- TargetEnv = qa
- cucumberTags = @regression
- cucumberParallel = 4

Step 6: Configure Automated Triggers & Branch Protection

PR Trigger: Configured on pull requests with cucumberTags=@smoke.
Nightly Schedule Trigger: Configured on a nightly cron schedule with cucumberTags=@regression and cucumberParallel=4.
GitHub Branch Protection: Enable branch protection on target branches requiring the Harness PR pipeline status check to pass before merging.

Once both the PR and Nightly execution runs pass with green status checks, your pipeline onboarding is complete!

End-to-End Setup Guide: Integrating Playwright + Cucumber with Harness CI

She11 QA — Tue, 18 Aug 2026 09:33:47 +0000

Integrating end-to-end (E2E) automation suites into enterprise CI/CD pipelines requires robust reporting, dynamic execution controls, and seamless artifact management.

Here is a guide on setting up a Node.js + Playwright + Cucumber.js test suite using Harness CI, configured with dual-repository dependencies, parallel execution capabilities, and dashboard-ready reporting.

Key Architectural Setup

Two-Repo Architecture:
- Repository A (Application Automation Repo): Contains application-specific feature files, page objects, and pipeline definitions.
- Repository B (Shared Framework Repo): Hosts core framework utilities, custom assertions, and base drivers consumed as a pinned dependency.
Tech Stack: Node.js, Playwright, Cucumber.js, Allure/JUnit reporting.

Step 1: Configure Harness Connectors & Secrets

Set up these foundational resources within your Harness account:

Connectors:

GIT_CONNECTOR: Grants access to both application and framework GitHub repositories.
K8S_CONNECTOR: Manages the Kubernetes build infrastructure.

Secrets:

CONNECT_URL, CONNECT_USERNAME, and CONNECT_PASSWORD (and proxy settings if required).

Step 2: Configure Pipelines

Import your execution configurations using YAML files inside .harness/:

Standard Run (.harness/e2e-poc.yaml): Used for fast PR checks.
Parallel Regression (.harness/e2e-regression-parallel.yaml): Used for scheduled, high-volume regression runs.

Replace placeholders such as , , and to map to your cluster environment.

Step 3: Define Pipeline Triggers

Set up two primary execution workflows:

Pull Request (PR) Trigger:
- Event: Pull Request to main/POC branch.
- Runtime Variables: cucumberTags=@smoke
Scheduled Nightly Trigger:
- Event: Scheduled Cron.
- Runtime Variables: cucumberTags=@regression, cucumberParallel=4

Step 4: Test Report & Artifact Collection

To ensure test metrics display properly on the Harness dashboard, configure both JUnit parsing and raw artifact archiving.

Generated Outputs:

reports/junit-report.xml (parsed by Harness for test metrics)
reports/cucumber_report.json
reports/allure-results & HTML reports
test-results/screenshots & test-results/traces (captured on failure)

Harness UI Configuration Steps:

Under execution reporting options, enable JUnit Test Report Collection and point it to reports/junit-report.xml.
Enable Artifact Collection with the following paths:
- reports/**
- test-results/**
Optional: Set strictReporting=true to force pipeline failure if report outputs are missing.

Step 5: Best Practices for Headless Environments

Browser Fallbacks: Configure cross-platform handling in your hooks file to run branded browsers (like Edge) locally, while falling back to standard headless containers (e.g., Chromium) in CI Linux pods.
Branch Protection: Enforce GitHub branch rules requiring status checks from Harness to pass prior to merging PRs.
Shared Framework Management: For quick POCs, pin Repository B as a Git dependency in package.json. For production, publish the framework as a versioned private package.

How to Configure Parallel Execution in TestNG vs. Custom Excel Allocator

She11 QA — Tue, 18 Aug 2026 09:23:32 +0000

Optimizing test execution speed is essential for keeping build pipelines lean. Depending on how your framework is structured, you can achieve full parallel execution either natively using TestNG ** or dynamically using a **Custom Excel Allocator.

Here is a step-by-step guide on configuring both approaches, along with a comparison to help you choose the right strategy.

Strategy 1: Native TestNG Parallelization (Recommended for Code-Native Suites)

TestNG natively supports parallel execution at the methods, classes, tests, or instances level using its XML configuration or Maven parameters.

1. Update testng_regression.xml
Modify the tag to set the execution mode and thread pool size:

<suite name="Regression" parallel="methods" thread-count="10">

2. Configure pom.xml for Dynamic Overrides

Allow developers and CI pipelines to override execution settings without altering XML files by adding these lines inside the block of the maven-surefire-plugin:

<parallel>${parallel}</parallel>
<threadCount>${threadCount}</threadCount>

3. Execution Commands

Default Run:

   mvn clean test -P runTestNGTests

Override Thread Count Dynamically:

   mvn clean test -P runTestNGTests -DthreadCount=15

Full Parallel Execution (Match CPU Core Count):

   mvn clean test -P runTestNGTests -Dparallel=methods -DthreadCount=24

Strategy 2: Custom Allocator & Run Manager (For Excel-Driven Suites)

If your framework relies on an Excel-driven Run Manager to parse keyword flows and data sheets dynamically, parallelism is managed via a custom ExecutorService fixed thread pool.

Execution Command

mvn clean test -P runAllocator

How it works: The allocator reads active test rows (Execute=Yes), dynamically assigns thread pools based on target thread properties, and dispatches concurrent runs.

Comparison: Allocator (Run Manager) vs. Native TestNG

Feature	Allocator (Run Manager)	TestNG Native
Entry Point	allocator.Allocator.main() via Maven Exec Plugin	maven-surefire-plugin executing testng.xml
Test Selection	Reads Excel sheets via central properties	Reads testng_regression.xml test classes/methods
Thread Management	Java ExecutorService (FixedThreadPool)	TestNG internal thread pool (parallel + thread-count)
Command	mvn clean test -P runAllocator	mvn clean test -P runTestNGTests
Pros	Pure data-driven control; multi-sheet aggregation; exact instance control	Lighter weight; no Excel dependency; native TestNG integration
Cons	Requires property tuning; multi-sheet parsing requires custom code	Limited to test methods/classes; lacks Excel data loop control

Which Approach Should You Choose?

Choose Native TestNG if you want:
- Simple, code-first test execution.
- Faster execution loops without file parsing overhead.
- Standardized parallel="methods" or parallel="classes" handling.
Choose Allocator + Run Manager if you want:
- Multi-sheet Excel data-driven scheduling.
- Fine-grained control over test iteration instances based on data rows.
- Integration with existing property-driven suite configs.