DEV Community: depa panjie purnama

ROI of AI Test Automation: A Calculation Framework for QA Leaders

depa panjie purnama — Thu, 16 Jul 2026 06:22:14 +0000

Every QA leader has faced the same conversation. Leadership asks: "What are we getting for our automation investment?" And the honest answer is often some version of "we're faster than we used to be" without hard numbers to back it up.

That gap between intuition and evidence is where automation programs get defunded. Not because they are not delivering value, but because the value was never quantified in terms finance teams understand.

This problem compounds in 2026 because the investment is no longer just "automation". It is AI-powered automation: agentic test generation, self-healing scripts, intelligent failure triage, autonomous execution. The costs are different. The benefits are different. And the traditional ROI formulas that worked for Selenium script libraries do not capture what AI-native testing platforms actually deliver.

You may already know the standard formula for calculating test automation ROI - that guide covers the foundational math well. But it was designed for scripted automation, not for AI agents. If you are still applying a Selenium-era formula to an AI-native platform, you are underselling the investment by a significant margin.

This guide provides a calculation framework built specifically for AI test automation ROI: what to measure, how to measure it, where the traditional formulas fall short, and how to build a business case that finance and engineering leadership will approve.

Why Traditional Automation ROI Formulas Fall Short

The classic test automation ROI formula is straightforward:

ROI (%) = (Benefits from Automation - Automation Costs) / Automation Costs × 100

For traditional scripted automation, the inputs were relatively simple: Costs covered tool licenses plus engineer time to write scripts plus maintenance time. Benefits came from manual testing hours saved multiplied by the hourly rate.

This formula worked when automation meant "replace manual test execution with scripts." The value proposition was labor substitution: a script runs a test faster and more repeatedly than a human.

But AI test automation changes the equation in three ways that the traditional formula does not capture.

AI reduces costs that traditional automation created

Traditional automation has its own cost problem: maintenance. Industry data consistently shows that 30-40% of automation engineering time goes to maintaining existing scripts rather than creating new coverage. AI self-healing capabilities reduce or eliminate that maintenance burden. The traditional formula counts "hours saved versus manual testing" but misses "hours saved versus maintaining the automation itself."

AI creates value categories that did not exist before

Intelligent failure classification saves triage time. AI-generated test cases from requirements create coverage that would never have been written manually (because nobody had time). Root Cause Analyzer automatically classifies failures closes the triage loop that traditional automation left wide open. These are not "manual hours replaced." They are new capabilities with their own value.

AI value compounds over time

A Selenium script delivers the same value on day one as day 365. An AI system that learns from execution history, defect patterns, and historical data delivers more value with each cycle. The traditional formula assumes linear returns. AI delivers compounding returns.

The AI Test Automation ROI Framework

This framework captures the full value of AI-powered testing by measuring four categories of return, not just one.

Category 1: Labor Cost Reduction

This is the traditional category, updated for AI capabilities.

What to measure:

Hours per sprint previously spent on manual test execution, now handled by automation
Hours per sprint previously spent on test case writing, now handled by AI test generation
Hours per sprint previously spent on script maintenance, now reduced by self-healing
Hours per sprint previously spent on failure triage, now handled by AI classification
Hours per sprint previously spent on defect documentation, now handled by AI bug reporting

How to calculate:

Total hours saved per sprint × fully loaded hourly rate × sprints per year = Annual labor cost reduction

Example: A 10-person QA team where AI automation saves an average of 4 hours per person per sprint:

10 engineers × 4 hours × $75/hour (loaded rate) × 26 sprints/year = $78,000/year

Category 2: Quality Improvement (Defect Prevention)

This category captures the value of catching bugs earlier and catching bugs that would have escaped entirely.

What to measure:

Defect escape rate before AI automation vs. after
Average cost of a production defect (detection, fix, customer impact)
Additional test coverage generated by AI that would not have been written manually
Time-to-detection improvement (catching defects in CI vs. production)

How to calculate:

(Defects prevented per year × average cost per production defect) + (Earlier detection savings) = Annual quality improvement value

Example: If AI-generated tests catch 5 additional defects per quarter that would have reached production, and each production defect costs $15,000 to resolve (including engineering time, customer support, and reputation impact):

20 defects/year × $15,000 = $300,000/year

The cost of poor software quality in the US has reached an estimated $2.41 trillion according to CISQ and Carnegie Mellon SEI. Even capturing a fraction of that cost at the team level produces significant ROI.

Category 3: Velocity Improvement (Time-to-Market)

This category captures the business value of shipping faster with confidence.

What to measure:

Release cycle time before vs. after AI automation
Time-to-release-readiness (how long it takes to answer "are we ready to ship?")
Sprint capacity freed up for new feature testing vs. regression maintenance
Reduction in release delays caused by testing bottlenecks

How to calculate:

This category is harder to assign a dollar value because it depends on business context. Two approaches work well here.

Approach A (Revenue attribution): If faster releases directly enable revenue through feature launches or market timing, estimate the revenue impact of shipping X days earlier.

Approach B (Capacity recovery): Calculate the engineering hours freed from regression and maintenance that can now be applied to new feature coverage.

Example (Approach B): If AI self-healing and automated regression reduce the sprint testing overhead by 20%, and that 20% is redirected to new feature testing:

10 engineers × 20% of sprint capacity × $75/hour × 80 hours/sprint × 26 sprints/year = $312,000/year in recovered capacity

Category 4: Strategic Value (AI Compounding)

This category captures the long-term value that increases over time as the AI system learns from more data.

What to measure:

Improvement in AI test generation accuracy over time (fewer revisions needed)
Improvement in failure classification accuracy over time (fewer false positives)
Reduction in onboarding time for new team members (AI handles ramp-up tasks)

How to calculate:

Strategic value is best expressed as a trajectory rather than a fixed number. Measure the metrics above quarterly and show the improvement curve. This demonstrates that the investment appreciates rather than depreciates, which is a fundamentally different story than traditional tooling.

Example: In Quarter 1, AI test generation requires a 40% revision rate, meaning human edits are needed on 4 in 10 generated cases. By Quarter 4, that rate drops to 15%. Each subsequent quarter delivers more value from the same investment.

Building the Business Case: A Template

When presenting AI test automation ROI to leadership, structure the case around these four sections.

Section 1: Current State Costs

Document what the organization currently spends on testing:

Cost Category	Annual Cost
QA team fully loaded salaries	$ ______
Testing tool licenses (all tools)	$ ______
Cloud execution infrastructure	$ ______
Test maintenance overhead (% of team time × salary)	$ ______
Release delay costs (estimated)	$ ______
Production defect resolution costs	$ ______
Total current state cost	$ ______

Section 2: Projected Investment

Document what the AI test automation platform will cost:

Investment Category	Annual Cost
Platform licensing (per-user × team size)	$ ______
AI model usage / inference costs	$ ______
Migration effort (one-time, amortized over 3 years)	$ ______
Training and onboarding (one-time, amortized)	$ ______
Ongoing administration	$ ______
Total investment	$ ______

Section 3: Projected Returns (by Category)

Return Category	Annual Value	Confidence
Labor cost reduction	$ ______	High (directly measurable)
Quality improvement	$ ______	Medium (requires defect cost estimation)
Velocity improvement	$ ______	Medium (requires capacity attribution)
Strategic value (compounding)	$ ______	Directional (show trajectory)
Total projected return	$ ______

Section 4: ROI Summary

ROI (%) = (Total Projected Return - Total Investment) / Total Investment × 100  
Payback period = Total Investment / (Total Projected Return / 12 months)

Most teams implementing AI test automation report payback periods of 3-6 months when all four categories are measured. Teams that only measure Category 1 (labor cost reduction) typically see 6-12 month payback, which is still strong but undersells the full value.

Metrics to Track Post-Implementation

Once the investment is approved and implemented, track these metrics to validate the business case and demonstrate ongoing value. The full set of test automation metrics worth tracking spans three time horizons.

Leading Indicators (measure weekly or per sprint)

Test creation velocity: Tests created per sprint (manual and AI-generated)
Maintenance ratio: Percentage of automation time spent on maintenance vs. new coverage
Failure classification accuracy: Percentage of AI-classified failures that are correct (validated by engineers)
AI revision rate: Percentage of AI-generated test cases that require human editing before approval

Lagging Indicators (measure monthly or quarterly)

Defect escape rate: Defects found in production vs. found in testing
Release cycle time: Calendar days from code complete to production
Test coverage: Percentage of requirements with associated test cases
Cost per test: Total testing cost divided by number of test cases maintained

Executive Metrics (measure quarterly)

Total ROI: Actual returns vs. projected (by category)
Payback progress: Cumulative returns vs. cumulative investment
AI maturity curve: Improvement in AI accuracy metrics over time
Capacity utilization: Percentage of QA time on strategic work vs. mechanical work

Common Mistakes in ROI Calculation

Mistake: Only counting labor substitution. The traditional "hours saved vs. manual testing" calculation captures maybe 30% of the actual value. Include quality improvement, velocity gains, and strategic compounding to present the full picture.

Mistake: Ignoring the cost of doing nothing. The comparison is not "current state vs. AI automation." It is "current state deteriorating as development velocity increases vs. AI automation." As AI-generated code accelerates development, the testing gap widens every quarter. The cost of not investing is not zero. It is the growing defect escape rate and release delays.

Mistake: Using averages instead of ranges. Present ROI as a range (conservative, expected, optimistic) rather than a single number. Finance teams trust ranges more than precise predictions because they demonstrate that the analysis accounts for uncertainty.

Mistake: Forgetting migration and ramp-up costs. Include the one-time costs of migration, training, and the productivity dip during the first 4-6 weeks. Amortize these over 3 years to show the true annual cost. Hiding these costs erodes trust when they appear later.

Mistake: Not baselining before implementation. Without pre-implementation baselines covering current test creation time, maintenance burden, defect escape rate, and release cycle time, post-implementation improvements cannot be quantified. Establish baselines before the project starts.

How Katalon True Platform Delivers Measurable ROI

Katalon True Platform is designed to deliver returns across all four ROI categories through its unified architecture and six purpose-built AI agents, all orchestrated by the Katalon AI Assistant. The model is consistent throughout: AI proposes, humans approve.

Labor cost reduction:

The Test Generation Agent drafts test suites from requirements, reducing test creation time significantly
Self-healing capabilities reduce script maintenance burden
The Bug Reporter automates defect documentation and filing
The Root Cause Analyzer eliminates manual failure triage by classifying each failure as a script issue, application bug, or environment problem

Quality improvement:

AI-generated tests cover edge cases and negative paths that manual creation typically skips under time pressure
The Requirement Analyzer scores requirements for testability before generating tests, surfacing ambiguities that would otherwise produce inaccurate coverage
The Autonomous Test Runner executes tests without supervision, increasing execution frequency across more of the application surface

Velocity improvement:

The Report and Insight Generator provides real-time release readiness assessment, with GO/NO-GO recommendations against configured thresholds
The unified platform eliminates context-switching between disconnected tools
Native CI/CD integration enables testing at the speed of deployment

Strategic value (compounding):

The unified data layer means every test run, every defect, and every execution result feeds the same intelligence layer
AI agents improve accuracy with each cycle because they learn from complete, connected data
Platform consolidation, replacing 4-5 tools with one, reduces total cost of ownership while increasing capability

The platform supports web, mobile, API, and desktop testing across no-code, low-code, and full-code approaches. Per-user subscription pricing makes cost projection straightforward for the business case templates above.

Four Steps Before Your Next Budget Conversation

A strong business case depends on defensible numbers. Here are four actions to take before presenting to leadership.

Baseline this sprint. Record your current test creation time per sprint, your maintenance ratio, your defect escape rate, and your release cycle time. Without these, you cannot measure what changes.
Estimate one production defect cost. Talk to your dev lead or engineering manager and agree on a realistic figure for what a production bug costs your organization, including engineering time, customer support, and any reputational cost. Even a conservative estimate makes Category 2 compelling.
Run Category 1 first. Labor cost reduction is the most directly measurable category and the easiest to present. Start there, then add Categories 2-4 as supporting evidence rather than primary claims.
Present as a range. Build a conservative, expected, and optimistic scenario for each category. Finance teams trust the analyst who acknowledges uncertainty more than the one who arrives with a single precise number.

Ready to start measuring? Try Katalon True Platform free and establish your baseline today.

References

CISQ/Carnegie Mellon SEI. "The Cost of Poor Software Quality in the US: A 2022 Report." Consortium for Information and Software Quality, 2022. it-cisq.org
Katalon. "State of Software Quality Report 2025." Katalon, 2025. katalon.com
Forrester Research. "The Autonomous Testing Platforms Landscape, Q3 2025." Forrester, July 2025. forrester.com
Capgemini. "World Quality Report 2025-26." Capgemini Research Institute, 2025. capgemini.com
Capgemini. "World Quality Report 2022/23." Capgemini Research Institute, 2022. capgemini.com

Migrating from Tricentis to Katalon: What to Expect and How to Plan

depa panjie purnama — Thu, 16 Jul 2026 06:12:51 +0000

Tricentis Tosca has been a strong choice for enterprise test automation, particularly for organizations with SAP-heavy environments and teams that prefer a model-based, codeless approach. It works. For many teams, it has worked well for years.

But the testing landscape has shifted. AI-native platforms, unified data architectures, and agentic testing capabilities have changed what "enterprise-grade" means in 2026. Teams that adopted Tosca three or four years ago are now evaluating whether the platform still fits their trajectory, especially as licensing costs compound, modern CI/CD integration remains friction-heavy, and AI adoption requires connected data that Tosca's architecture was not designed to provide.

This guide is for QA leaders and engineering teams actively considering a migration from Tricentis Tosca to Katalon. It covers what to expect, how to plan, what to migrate (and what to leave behind), and how to execute the transition without disrupting delivery.

Why teams consider leaving Tricentis

No migration happens without a reason. Based on publicly available reviews, competitive analyses, and common patterns in enterprise QA, these are the most frequent drivers.

Licensing complexity and cost

Tricentis Tosca uses modular enterprise licensing that typically starts at $30,000+ per year. Every major capability requires a separate license: Vision AI, mobile testing, SAP modules, test data management, and execution agents are all add-ons. For teams that need broad coverage, the total cost of ownership escalates quickly.

Katalon uses a per-user subscription model with all core capabilities included: automation, manual testing, execution, AI agents, Test Management, and reporting - without modular add-ons.

Vendor lock-in

Tosca's model-based approach creates test assets that are tightly coupled to the platform. Test modules, business parameters, and execution configurations are stored in proprietary formats. Once a team has invested years of work into Tosca, migrating those assets to another platform is non-trivial. This lock-in becomes a strategic concern when the platform's direction no longer aligns with the team's needs.

Modern CI/CD integration friction

Tosca was designed for enterprise environments where release cycles were measured in months. While Tricentis has added CI/CD capabilities over time, teams running modern DevOps pipelines on GitHub Actions, GitLab CI, Jenkins, or Azure DevOps often report that integration requires more configuration and workaround than expected. The platform's architecture reflects its origins in a pre-DevOps era.

AI adoption requires unified data

Tricentis has introduced AI capabilities, including Vision AI and agentic testing features. But AI effectiveness depends on access to connected data across the full testing lifecycle. When test management, execution, defect tracking, and production signals live in separate systems or require separate licenses, AI agents operate on incomplete information. Teams evaluating AI-powered testing often find that platform architecture matters more than individual AI features.

Steep learning curve

Tosca's model-based approach requires upfront model creation before tests can be written. Teams must build and maintain models, learn Tosca-specific concepts, and follow Tricentis-specific workflows. Onboarding new team members takes weeks rather than days. For teams with high turnover or mixed skill levels, this creates ongoing friction.

What you are actually migrating

Before planning the migration, it helps to categorize what exists in your current Tricentis environment. Not everything needs to move, and not everything should.

Assets that migrate directly

Test case logic. The intent behind your tests - what you are testing and why - migrates regardless of platform. Test scenarios, acceptance criteria, and coverage maps are platform-independent. These translate directly into Katalon test cases.

Test data. Data sets, test data configurations, and parameterized inputs can be exported and restructured for Katalon's data-driven testing approach. The format changes, but the data itself carries over.

Requirements traceability. If your tests are mapped to requirements in Jira or Azure DevOps, those mappings can be re-established in Katalon's Test Management, which integrates natively with both systems.

Execution configurations. Browser and device combinations, environment configurations, and execution schedules can be recreated in Katalon's test execution environment.

Assets that transform

Tosca modules and test cases. Tosca's model-based test cases cannot be directly imported into Katalon due to their proprietary format. However, the test logic they represent can be recreated. For teams with hundreds or thousands of Tosca test cases, this is the most significant migration effort though the approach is not 1:1 recreation. Katalon's Test Generation Agent can draft test cases directly from requirements, which substantially reduces manual re-implementation time.

Custom actions and reusable components. Tosca's reusable modules and custom actions need to be rebuilt as Katalon custom keywords or reusable test objects. The logic transfers; the implementation changes.

Assets you leave behind

Tosca-specific configurations. Execution lists, Tosca Commander workspace settings, and Tosca-specific integrations do not migrate. They are replaced by Katalon's native equivalents.

Model maintenance overhead. One of the concrete benefits of migration: you stop maintaining Tosca models. Katalon's approach - combining record-and-playback, scripting, and AI-assisted test creation - eliminates the model maintenance layer entirely.

Migration planning: a phased approach

Migration from an enterprise testing platform is not a weekend project. But it does not need to be a six-month disruption either. The following phased approach keeps delivery running while progressively moving workloads to Katalon.

Phase 1: Assessment and baseline (weeks 1-2)

Start by inventorying your Tosca environment across a few key dimensions: total test case count (automated and manual), what percentage are active versus legacy or deprecated, which test types are in scope (UI, API, data validation, SAP-specific), how frequently each suite runs, and where your integration points sit - CI/CD pipelines, Jira, ALM tools.

Equally important: establish baselines before anything changes. Current test cycle time per sprint, defect escape rate, time spent on test maintenance, and onboarding time for new team members. These baselines become your measurement instrument. Without them, you cannot assess what the migration actually delivered - not just whether it completed.

Use this phase to identify migration priorities. Not all tests are equally valuable. Daily runners, revenue-impacting flows, and tests that break frequently in Tosca (strong candidates for AI-assisted recreation) should move first.

Phase 2: Pilot migration (weeks 3-6)

Choose one product area or test suite that represents typical workload without being your most critical path. The pilot scope should include a mix of UI and API tests, integration with your CI/CD pipeline, and the reporting that your stakeholders actually review.

Recreate the pilot tests in Katalon: use record-and-playback for a quick UI baseline, the Test Generation Agent to draft test cases from requirements (then review and refine), and Katalon's built-in API testing for service-layer coverage. Then run Tosca and Katalon in parallel for two to three weeks. Compare results, identify gaps, and validate that Katalon coverage matches or exceeds Tosca for that scope.

The pilot is also where you discover actual migration velocity. Measure test creation time in Katalon versus Tosca for equivalent tests, maintenance effort required to keep tests green, CI/CD integration smoothness, and team feedback on usability. These numbers inform every phase that follows.

Phase 3: Progressive migration (weeks 7-16)

Based on pilot results, expand migration to additional test suites in the order established in Phase 1 - execution frequency and business criticality first.

For UI tests, use a combination of Katalon's recorder for quick baseline creation and manual scripting for complex flows. For high-volume UI test suites, the Test Generation Agent can draft tests from requirements at scale. For API tests, Katalon's built-in API testing supports REST, SOAP, and GraphQL - API tests typically migrate faster than UI tests because they are less dependent on platform-specific object models. For data-driven tests, export from Tosca and restructure for Katalon's data binding approach via Excel, CSV, or database connections.

SAP-specific tests warrant a case-by-case call. Katalon supports SAP testing through its Windows desktop testing capabilities, but highly complex SAP suites may benefit from a separate assessment of migration effort versus value before committing.

As each suite is validated in Katalon, decommission the Tosca equivalent. Do not maintain both long-term - parallel maintenance doubles the workload and defeats the purpose of migration.

Phase 4: Cutover and optimization (weeks 16-20)

Complete the migration of remaining test suites. At this stage, the team has built real proficiency with Katalon and migration velocity is significantly faster than it was during the pilot.

Cancel Tricentis licenses once all active test suites are validated and document the direct cost savings.

Then shift focus to optimization: enable the Katalon AI Assistant for multi-agent orchestration across the six AI agents - the Requirement Analyzer, Test Generation Agent, Autonomous Test Runner, Bug Reporter, Report & Insight Generator, and Root Cause Analyzer. Configure governance: approval gates for AI-generated tests, release readiness thresholds, and audit trail requirements. This is where the platform starts returning compounding value, not just parity with what Tosca delivered.

What changes for your team

Migration is not just a tool swap. It changes how the team works. Here is what to expect across roles.

For automation engineers

Katalon uses Groovy scripting (Java-compatible) for advanced test logic, alongside a visual recorder and keyword-driven approach for simpler tests. Engineers coming from Tosca's model-based approach typically find Katalon's scripting more familiar if they have any programming background. Most teams report productive test creation within the first week.

Without Tosca's model layer to maintain, test maintenance shifts to object repository management and script updates - a meaningfully lighter overhead. The Autonomous Test Runner and AI-assisted maintenance reduce this further. The bigger shift is in how engineers spend their time: the six AI agents handle test case generation from requirements, failure classification, bug report composition, and release readiness assessment. Engineers move from doing this work to reviewing AI-generated outputs, which is a different - and generally more valuable - use of their expertise.

For manual testers

Katalon's no-code and low-code options: record-and-playback, keyword-driven testing, and manual test management in Test Management, allow manual testers to contribute to test automation without writing scripts. The Test Generation Agent creates structured test cases from requirements; manual testers review and refine rather than authoring from scratch. Tosca also offered codeless testing, but Katalon's approach is generally reported as having a gentler learning curve for teams without dedicated Tosca training.

For QA managers

The most immediate change is visibility. Katalon provides a single dashboard for all testing activities - automated and manual, across all platforms, so release readiness is no longer something you assemble from multiple sources. The Report & Insight Generator answers plain-language questions about coverage, defect trends, and whether the build is ready to ship.

On the cost side, per-user subscription pricing replaces Tosca's modular licensing model. Budget planning becomes predictable in a way that modular enterprise licensing rarely is.

Common migration risks and how to mitigate them

Test coverage gaps during transition. Run parallel execution during the pilot phase and do not decommission Tosca test suites until Katalon equivalents are validated. Use coverage mapping to ensure nothing falls through the cracks.

Team resistance to change. Start with volunteers for the pilot squad. Let early adopters build confidence and become internal advocates. Provide dedicated learning time rather than expecting migration work on top of existing sprint commitments - that combination consistently fails.

Underestimating migration effort for complex Tosca suites. Not every Tosca test case needs 1:1 recreation. Some tests are outdated, redundant, or testing functionality that no longer exists. Use the migration as an opportunity to rationalize your test suite. Migrate what matters; retire what does not.

CI/CD integration disruption. Katalon integrates natively with Jenkins, GitHub Actions, GitLab CI, Azure DevOps, and other major CI/CD platforms. Set up that integration during the pilot phase and validate it before expanding to broader migration.

Loss of historical test data. Export execution history and defect records from Tosca before decommissioning. Historical Tosca execution data will not import directly into Katalon, but it should be archived for compliance and reference.

Migration timeline at a glance

Phase	Duration	Key activities	Success criteria
Assessment	Weeks 1-2	Inventory, prioritize, establish baselines	Complete asset map and migration plan
Pilot	Weeks 3-6	Migrate one suite, run parallel execution	Katalon matches Tosca coverage for pilot scope
Progressive migration	Weeks 7-16	Expand to all suites by priority	All active test suites running in Katalon
Cutover	Weeks 16-20	Retire Tosca, optimize and enable AI agents	Licenses cancelled, AI agents enabled

Total timeline: approximately 4-5 months for a typical enterprise migration. Smaller teams or less complex environments can compress this significantly.

Key takeaways

Migration from Tricentis to Katalon is a phased process, not a big-bang cutover. Teams maintain delivery throughout by running parallel execution during transition, and each phase has a clear exit criteria before the next begins.

Not every Tosca test case needs 1:1 recreation. The migration is an opportunity to rationalize your test suite and leverage AI-assisted generation for faster re-implementation of what matters.

The primary drivers for migration: licensing cost, vendor lock-in, CI/CD friction, and the need for a unified data layer to support AI - are architectural in nature. Addressing them requires a platform change, not just a tool addition.

The strategic payoff is not only the direct cost savings from consolidating licensing. It is the AI capabilities that become possible when all quality data lives in a single platform, with six purpose-built agents operating across the full lifecycle from requirements to production. That connected data layer is not available in a fragmented stack, regardless of how many individual AI features its constituent tools claim to offer.

For teams evaluating whether migration makes sense before committing, a detailed Katalon vs. Tricentis comparison covers the feature and pricing differences side by side. For teams still exploring what alternatives exist before narrowing to a shortlist, the broader Tricentis alternatives overview is a useful starting point.

How to Write a Requirements Traceability Matrix (With Free Template)

depa panjie purnama — Thu, 16 Jul 2026 06:06:12 +0000

If you have ever been asked "how do we know we tested everything?" and did not have a clean answer, a requirements traceability matrix is what you were missing.

It is one of those deliverables that sounds bureaucratic until the moment you actually need it: an audit, a release sign-off, a requirement that slipped through without a single test covering it. Then it becomes the most useful document you own.

This guide covers what a requirements traceability matrix is, the different types, how to build one step by step, and a free template you can copy and start using today. No fluff, just the practical version a working tester or QA lead can apply this sprint.

What Is a Requirements Traceability Matrix?

A requirements traceability matrix (RTM) is a document that maps requirements to the test cases that verify them. You will also see it called an RTM - short for Requirements Traceability Matrix in most QA documentation. At its simplest, it is a table that answers one question: for every requirement, which test cases prove it works?

That mapping gives you two things most QA teams struggle to produce on demand. First, coverage proof: every requirement has at least one test case linked to it, so gaps become visible immediately. Second, impact analysis: when a requirement changes, you can instantly see which test cases are affected.

Think of it as the connective tissue between "what the product is supposed to do" and "what we actually tested." Without it, those two things live in separate documents and nobody can confidently say they line up.

Why Manual Testers Should Care

Traceability is often dismissed as paperwork. That is a mistake, especially for manual testers who want to demonstrate the value of their work.

It makes your coverage visible. When you can show a matrix that links every requirement to a test case, you stop being "the person who runs tests" and become "the person who guarantees coverage." That is a meaningful shift in how your work is perceived.

It protects you during sign-off. When a bug reaches production and someone asks "was this tested?", the matrix is your evidence. Either the requirement was covered (and the test missed an edge case, which is a different conversation) or the requirement was never specified (which is not a QA failure).

It is required in regulated industries. Healthcare, finance, aviation, and government projects often mandate traceability for compliance. Knowing how to build and maintain a matrix is a hireable skill in these domains.

It survives AI. As AI tools take over more test execution and generation, the human skill of defining what needs testing and proving it was covered becomes more valuable, not less. Traceability is exactly that kind of judgment-based work.

The Three Types of Traceability

Most guides list these, but here is what each one actually does for you in practice.

Forward traceability maps requirements forward to test cases. You start with a requirement and trace it to the tests that verify it.

Use it to answer: "Have I written test cases for every requirement?"

This is the most common direction and the one that catches coverage gaps. If a requirement has no test cases linked, it shows up as an empty row.

Backward traceability maps test cases back to requirements. You start with a test case and trace it to the requirement it covers.

Use it to answer: "Why does this test exist?"

Backward traceability catches the opposite problem: tests that do not map to any requirement. These are often outdated tests, gold-plating, or tests for features that were removed. They add maintenance burden without adding value.

Bidirectional traceability combines both directions. You can trace from requirement to test case and from test case back to requirement.

Use it to answer: "Is my coverage complete and is every test justified?"

This is what you want for serious projects. It gives you full visibility in both directions and is usually what auditors and compliance frameworks expect. Most mature test case management practices rely on bidirectional traceability as the baseline.

What Goes in a Traceability Matrix

A useful RTM has seven core columns. You can add more, but do not ship with fewer.

Column	What it holds	Why it matters
Requirement ID	Unique identifier (e.g., REQ-001)	The anchor everything links to
Requirement Description	Short summary of the requirement	Context without opening another doc
Test Case ID	Linked test case identifier (e.g., TC-014)	The actual coverage link
Test Case Description	What the test verifies	Quick readability
Test Status	Pass / Fail / Blocked / Not Run	Live execution state
Defect ID	Linked bug if the test failed	Closes the loop to defects
Coverage Status	Covered / Not Covered	The gap indicator

The defect column is the one teams most often skip, and it is the one that turns a static document into a live quality picture. With it, you get full lineage: requirement to test case to result to defect. Understanding where a defect sits in its lifecycle matters here too - the bug life cycle helps your team know what to do once a defect is logged.

Free Requirements Traceability Matrix Template (Copy-Ready)

Here is a template you can copy directly into Excel or Google Sheets. Each row represents one requirement-to-test-case link. If you need a head start on the test case side, Katalon's free test case template gives you Excel, Word, and PDF formats you can extend into a full RTM.

Req ID	Requirement Description	Test Case ID	Test Case Description	Test Status	Defect ID	Coverage Status
REQ-001	User can log in with email	TC-001	Valid login with correct credentials	Pass	-	Covered
REQ-001	User can log in with email	TC-002	Login fails with wrong password	Pass	-	Covered
REQ-002	User can reset password	TC-003	Reset link sent to valid email	Fail	BUG-021	Covered
REQ-003	User can update profile photo	-	-	-	-	Not Covered
REQ-004	Session expires after 30 min	TC-004	Session times out at 30 min	Not Run	-	Covered

A few things to notice here. REQ-001 has two test cases, one covering the positive path and one the negative. REQ-002 has a linked defect because the test failed - that is full traceability from requirement to bug. And REQ-003 is flagged as Not Covered, which is exactly what the matrix is for: making gaps impossible to miss.

Copy this structure, replace the rows with your own requirements and test cases, and you have a working matrix.

How to Build a Traceability Matrix: Step by Step

Step 1: Gather your requirements

Collect every requirement from your source of truth, whether that is Jira, Azure DevOps, a test plan, or user stories. Each requirement needs a unique ID. If your requirements do not have IDs yet, assign them now using a consistent convention like REQ-001, REQ-002, and so on.

Step 2: List your test cases

Pull together all the test cases you have written or plan to write. Each gets a unique ID too - TC-001, TC-002, and so on. If you are still in the requirement analysis phase of the STLC, you can stub out placeholders and fill in test case IDs as they are created.

Step 3: Map test cases to requirements

This is the core of the work. For each requirement, identify which test cases verify it and link them. One requirement may have several test cases. One test case may cover more than one requirement.

Step 4: Flag the gaps

Any requirement with no linked test case gets marked Not Covered. These are your immediate action items. Either write the missing tests or confirm the requirement is out of scope for this cycle.

Step 5: Add execution status and defects

As you run tests, update the status column. When a test fails, link the defect ID. Now your matrix reflects live quality state, not just planned coverage. This is what makes it useful during UAT sign-off - stakeholders can see exactly which requirements are verified and which defects are still open.

Step 6: Keep it updated

A traceability matrix is only useful if it is current. Update it whenever requirements change, test cases are added, or executions run. A stale matrix is worse than no matrix because it gives false confidence.

That last step is where most teams struggle, and it is where the manual spreadsheet approach starts to break down at scale.

The Problem With Spreadsheet Matrices

A spreadsheet traceability matrix works fine for a small project. But it has real limitations as you scale:

Manual updates. Every requirement change, new test, or execution result has to be entered by hand. On a fast-moving project, the matrix is out of date within days.
No live status. The spreadsheet shows what someone last typed, not the actual current state of your test runs.
Version control nightmares. Three people editing the same matrix file leads to conflicts and lost updates.
Disconnected from execution. The matrix lives separately from where tests actually run, so the link between them is only as good as someone's discipline in keeping them in sync.

For small teams or one-off compliance needs, a spreadsheet is genuinely fine. Start there. But if you are maintaining traceability across hundreds of requirements and an active test suite, the manual approach becomes a part-time job nobody volunteered for.

How Modern Test Management Automates Traceability

This is where a test management platform changes the equation. Instead of maintaining the matrix by hand, the linkage happens automatically as part of your normal workflow.

In Katalon True Platform's Test Management (formerly TestOps), traceability is built into how the work flows rather than bolted on as a reporting step. Requirements sync from your ALM automatically when you integrate Jira or Azure DevOps - no manual ID entry. When you generate or write a test case against a requirement, the link is established and maintained by the platform. The Test Generation Agent even links generated test cases back to their source requirement by default, so AI-created coverage is traced just as rigorously as manually written coverage.

Because execution results live in the same system as the requirement-to-test links, coverage status updates in real time. When a test fails and a defect is filed to Jira, that defect links back through the test case to the requirement automatically. You get full bidirectional traceability without any spreadsheet. When someone asks "are we ready to ship?" or "is this requirement covered?", the answer is a dashboard, not a weekend of reconciliation.

Common Mistakes to Avoid

Building it once and abandoning it. A matrix you create at the start of a project and never update is useless by week two. Traceability is a living document.

Skipping the defect link. Without linking defects, your matrix shows coverage but not quality. The defect column is what makes it a complete picture.

Over-engineering the columns. You do not need 20 columns. The core seven cover most needs. Add columns only when they answer a real question someone is asking.

Treating "has a test case" as "is tested well." A requirement linked to one weak test case shows as Covered, but coverage quality still requires human judgment. The matrix tells you a link exists, not that the test is good. That review is your job.

Maintaining it in a silo. If the matrix lives in a spreadsheet disconnected from your actual test execution, it will drift out of sync. Keep traceability as close to your execution as possible.

Conclusion

A requirements traceability matrix is not bureaucracy for its own sake. It is the document that makes your QA work legible to everyone else: stakeholders who need sign-off confidence, auditors who need compliance evidence, and your future self who needs to know which tests will break when a requirement changes.

Start with the template above. Copy it into a spreadsheet, map your current project, and you will have something useful by end of day. When the manual update burden grows too heavy, that is the signal that your team has outgrown the spreadsheet model. Katalon True Platform links requirements, test cases, executions, and defects in one place so your coverage picture is always current.

How to Review and Validate AI-Generated Test Cases (Without Blindly Trusting Them)

depa panjie purnama — Thu, 16 Jul 2026 05:08:32 +0000

AI can write a test case in seconds. It looks clean. It is structured correctly. It has a name, preconditions, steps, and expected results - everything a good test case should have.

And that is exactly the problem.

A polished test case is not the same as a correct one. AI-generated test cases can look completely professional while testing the wrong thing, missing critical edge cases, or quietly confirming that buggy behavior is "working as expected." The danger is not that AI produces obvious garbage you would catch immediately. The danger is that it produces plausible-looking test cases that pass review because nobody looked closely enough.

This guide is a practical playbook for reviewing AI-generated test cases. It covers what AI gets wrong, a test case review checklist you can apply to every generated test, and how to position yourself as the quality gate that makes AI-assisted testing actually trustworthy. If you want to understand how to generate test cases in Katalon True Platform first, the step-by-step guide to AI-assisted test case generation is the right starting point.

Why AI-generated test cases need human review

Let's be clear about the value first: AI test generation is genuinely useful. It eliminates the blank-page problem, drafts coverage in seconds, and handles the repetitive structure of test case writing. Used well, it makes you faster.

But the research on AI-generated testing is sobering. An analysis by CodeRabbit (reported by The Register) of thousands of production pull requests found that AI-generated code introduces 1.7 times more total issues than human-written code, with logic and correctness errors appearing 75% more often. A 2024 academic study found something even more concerning: LLM-based test generators can fail to detect bugs and, in some cases, actively validate buggy behavior by writing test cases that assert the broken output is correct.

That last point is the one to internalize. When AI generates a test case from code that already has a bug, it often writes a test that confirms the bug. The test passes. Everyone feels good. The bug ships anyway.

This is why human review is not optional. It is the difference between AI making you faster and AI making you confidently wrong. For a broader look at why AI-generated code introduces a new validation gap that traditional testing approaches don't close, that context is worth reading alongside this article.

What AI gets wrong in test cases

Before you can review effectively, you need to know what you are looking for. These are the failure patterns that show up most often in AI-generated test cases.

It only tests what was specified

AI generates test cases from the input it is given: a requirement, a user story, acceptance criteria. It covers what is written. It does not cover what was left unsaid. Real production scenarios live in the gaps between requirements - the thing the product owner forgot to mention, the integration behavior nobody documented, the edge case that only exists because of how a real user actually behaves. AI does not know about these because they are not in the input. You do, because you understand the product.

It validates bugs instead of catching them

When AI generates a test from existing code, it tends to assert that the current behavior is the correct behavior. If the code has a bug, the test enshrines the bug. The expected result matches the buggy output, so the test passes forever while the bug lives on. A test that passes is "working code" to the model. It has no concept of whether the behavior it is asserting is actually what the user needs.

It writes weak assertions

Watch for test cases where the expected result is vague or trivially true: "verify the page loads," "confirm no error appears," or assertions that would pass even if the feature were completely broken. AI produces assertions that are technically valid but verify almost nothing. A test that cannot fail is worse than no test - it provides false confidence without providing real coverage.

It misses negative and edge cases

AI is strong on happy paths. It is much weaker on the boundary conditions, invalid inputs, and failure scenarios that experienced testers know are where bugs hide. It will generate "user logs in with valid credentials" reliably. It is far less reliable at "user pastes a 10,000-character string into the password field" or "user submits the form after their session has expired."

It lacks domain context

AI does not know your business rules, your compliance requirements, your users' actual behavior, or the historical bugs in your product. A test case can be structurally perfect and still miss the thing that matters most, because the AI has no idea what matters most in your specific context. Every piece of institutional knowledge your team holds is a gap the AI cannot fill on its own.

It ignores emotional and human flows

This is the one AI is worst at, and the one experienced testers are best at. AI generates test cases around the mechanics of a feature. It does not think about how a real human feels while using it.

Consider scenarios like these that almost never show up in AI-generated test cases:

A frustrated user rapidly clicking "Submit" three times because the page felt slow - does the system double-charge them?
An anxious user abandoning a checkout halfway through - is their cart preserved when they return?
A user on a failing network connection who loses signal mid-transaction - what state are they left in?
A first-time user who does not understand the interface and takes the "wrong" path - does the product guide them or trap them?

These emotional and behavioral flows are where real users hit real problems. AI does not model human frustration, anxiety, confusion, or impatience. You do, because you have been a frustrated user yourself. This is some of the highest-value testing you can contribute, and it is precisely what AI cannot generate.

A framework for reviewing AI-generated test cases

Apply this test case review checklist to every AI-generated test case before it enters your suite. It takes a couple of minutes per test and catches the vast majority of issues across four dimensions: coverage, correctness, context, and clarity.

Coverage: does every test map to a real requirement?

Start by asking whether each test maps to a real requirement - if you cannot identify why a test exists, question it. Then ask the more important question: what is missing? Read the requirement, then ask what scenarios the AI did not generate. The gaps are where you add value. If the AI only generated happy paths, that is your cue to add the negative paths: invalid inputs, error states, permission failures, and boundary conditions that experienced testers know are where bugs actually hide.

Correctness: would this test fail if the feature broke?

The core of validating AI output is a single question: would this test actually fail if the feature broke? Mentally break the feature and ask whether the test would catch it. If the answer is no, the test is providing false confidence. Beyond that, do not assume the AI's expected results are correct - verify them against the requirement, not against the current behavior of the code. Reject vague assertions like "no error occurs" and demand specific, verifiable expected outcomes.

Context: does the test reflect real business rules and user behavior?

Check that the test honors your domain logic, compliance needs, and product-specific constraints. A test case can be structurally perfect and contextually wrong if it misses a business rule the AI had no way of knowing. Also watch for tests that follow an idealized path but miss how actual users interact with the feature, and check for duplicates or contradictions with test cases you already have in the suite.

Clarity: can another tester execute this without asking questions?

Steps should be clear enough that someone unfamiliar with the feature can execute the test. If another tester would need to ask three questions before they could run it, the test needs rewriting. Also watch for tests that try to verify too many things at once - one test, one clear objective is the standard to hold.

A practical review workflow

Here is how to put the framework into a repeatable process.

Step 1: Read the requirement first, not the test. Before you look at the generated test cases, read the source requirement and form your own mental model of what needs testing. List the scenarios you would cover. This gives you a baseline to compare against, so you are evaluating the AI's output rather than being anchored by it.

Step 2: Compare the AI's coverage to your mental model. Now look at what the AI generated. Where does it match your list? Where are the gaps? The gaps are your action items - add the scenarios the AI missed.

Step 3: Stress-test the assertions. Go through each test case and ask the failure question: "If this feature broke, would this test catch it?" Strengthen weak assertions. Reject tests that would pass regardless of whether the feature works.

Step 4: Add the domain knowledge AI cannot have. This is where you add the most value. Inject the business rules, the historical bugs, the real-user behavior, and the edge cases that come from understanding your specific product. This is the work that is fundamentally human.

Step 5: Approve, edit, or regenerate. For each test case, make a deliberate decision. Approve it if it is correct and complete. Edit it if it is close but needs stronger assertions or clearer steps. Regenerate it if it is fundamentally off - ideally with better context in your prompt so the next pass is more targeted. Never bulk-approve. The whole point is that a human made a deliberate decision about each test.

How to give feedback that improves AI output

Reviewing is reactive. The bigger win is making the AI generate better test cases in the first place, so you spend less time fixing and more time approving. The quality of what AI produces is directly tied to the quality of what you feed it.

Improve the input before you blame the output

Most weak AI-generated test cases come from weak input. Before deciding the AI is bad at this, check what it had to work with. A vague one-line requirement produces vague tests. A requirement with clear acceptance criteria produces far better tests. The fastest way to improve AI output is to improve the requirement it reads.

Be specific in your prompts

When the tool lets you add context or instructions, be concrete. A prompt like "test the login feature" produces generic output. A prompt like "generate test cases for the login feature, including negative cases for locked accounts, expired passwords, and more than five failed attempts, and cover the case where a user is already logged in on another device" tells the AI exactly where you want depth.

Feed it the knowledge it does not have

When you prompt, inject the context that matters:

Business rules ("discount codes cannot be combined with loyalty points")
Known problem areas ("this checkout flow has historically had issues with currency conversion")
Compliance constraints ("this form must meet WCAG 2.1 AA accessibility requirements")
Real user behavior patterns ("most users on mobile abandon if this step takes more than a few seconds")

Each piece of context you provide is a gap you prevent. Over time, building a reusable library of this domain context means every prompt gets better because you are no longer starting from scratch.

Treat regeneration as a conversation, not a reset

When the first output is off, tell the AI what was wrong: "these tests only cover the happy path, generate negative and boundary cases" produces a far better second pass than blindly hitting regenerate. The more specific your correction, the better the next attempt.

Build a reusable context library

If you keep telling the AI the same business rules every time, save that context somewhere reusable. Over time you build a library of domain context that turns generic AI generation into something tailored to your product. The investment compounds: every prompt gets better because you are no longer starting from scratch.

The pattern across all of this: AI output quality is a feedback loop, and you control the input side of that loop. Testers who learn to steer the AI well get dramatically better drafts, which means less review work and higher-quality coverage.

Why reviewing AI-generated tests makes you more valuable, not less

There is an anxiety running through QA right now that AI will replace manual testers. The review workflow above is the direct answer to that concern.

AI can draft test cases. It cannot decide whether they are correct, complete, or aligned with what your users actually need. That judgment requires understanding the product, the domain, the users, and the history. It requires knowing what was not written in the requirement. It requires the experience to look at a polished test case and say "this would pass even if the feature were broken."

That is not work AI is taking away. That is work AI is creating more of. Every AI-generated test case needs a human gatekeeper, and the gatekeeper role demands more skill and judgment than mechanically writing test cases ever did.

The testers who thrive in an AI-assisted world are not the ones who write the most test cases - they are the ones who can look at a hundred AI-generated test cases and quickly identify the ten that are wrong, the twenty that are weak, and the critical scenario that is missing entirely.

For teams building the broader governance structure around AI testing: defining review gates, confidence thresholds, and calibration cadences at the team level - the AI testing best practices guide covers that layer in detail.

📚 Read more: How to Test AI Applications Manually: A Playbook for Hallucinations, Bias, and Non-Deterministic Outputs

Common mistakes when reviewing AI-generated test cases

Even experienced testers fall into these patterns when working with AI test cases. Knowing them in advance is the fastest way to avoid them.

Trusting structure over substance. A well-formatted test case feels trustworthy. Formatting tells you nothing about correctness. Review the substance, not the polish.
Anchoring on the AI's output. If you read the generated tests before forming your own view, you will tend to accept their framing. Form your mental model first, then compare.
Bulk-approving to save time. The time saved by AI generation is meant to go into review, not to skip it. If you bulk-approve, you have just automated the creation of unreviewed tests.
Accepting expected results at face value. The most dangerous failure when validating AI output is asserting that buggy behavior is correct. Always validate expected results against the requirement, never against current code behavior.
Skipping the "would it fail?" check. A test that cannot fail is worse than no test, because it provides false confidence. Always ask whether the test would catch a real break.

How Katalon True Platform supports human-in-the-loop review

Katalon True Platform is built around exactly this principle: AI proposes, humans approve. The two agents most relevant to the review workflow described in this article are the Requirement Analyzer and the Test Generation Agent, and they work in sequence by design.

The Requirement Analyzer goes first. Before the Test Generation Agent produces a single test case, it reads the source requirement and scores it for testability: flagging ambiguities, surfacing gaps, and prompting for clarification on anything unclear. This directly addresses the root cause of most weak AI-generated test cases: the AI was working from a requirement that wasn't ready. By the time the Test Generation Agent takes over, it is working from a validated, structured foundation rather than raw, ambiguous input.

The Test Generation Agent then produces a draft suite of AI test cases linked back to the source requirement in your ALM automatically. A few ways the platform supports good review practice from there:

Generated test cases are clearly flagged. AI-generated content is visually distinct so you always know what came from the AI and needs review versus what a human has already approved.
Review-before-save is built in. The agent presents drafts that you review, edit, or discard. There is an explicit approval step - you can edit any step manually before saving, or regenerate with tighter context if the draft is off.
Requirement traceability is automatic. Each generated test case links back to its source requirement, so the "does this map to a real requirement?" check is answered by default and coverage gaps are visible.
Additional context refinement. When the agent needs more information to generate good tests, it prompts you for context first. This is your opportunity to inject the domain knowledge that prevents weak or off-target test cases before they are even generated.

The result is a workflow where the most common failure modes described in this article - AI generating from weak requirements, asserting buggy behavior as correct, missing domain context - are structurally reduced before review even begins. Your job as reviewer is to apply the judgment the platform cannot have: the edge cases that come from knowing your users, your product history, and the things not written in any ticket.

To see all six agents and how they connect across the full testing lifecycle, the Katalon AI Assistant overview covers each one in detail. To get started generating AI-assisted test cases in True Platform, the step-by-step guide to generating test cases in Katalon True Platform walks through the full workflow.

See how it works in the True Information Center.

References

CodeRabbit. "Analysis of AI-Generated Code Quality Across Production Pull Requests." CodeRabbit, 2026.
"Design choices made by LLM-based test generators prevent them from finding bugs." arXiv:2412.14137, 2024. [Verify via IEEE Spectrum or InfoQ for DR 80+ secondary citation before publishing.]
ISTQB. "Certified Tester AI Testing (CT-AI) Syllabus." International Software Testing Qualifications Board.

How to Test AI Applications Manually: A Playbook for Hallucinations, Bias, and Non-Deterministic Outputs

depa panjie purnama — Thu, 16 Jul 2026 04:57:16 +0000

You have tested hundreds of features. You know the drill. Open the test case, write the preconditions, list the steps, fill in the expected result, run it, compare. Pass or fail. Move on.

Then someone hands you an AI feature. A chatbot. A "summarize this ticket" button. A search box that answers in full sentences instead of returning a list of links. You open your test case template, you get to the "expected result" field, and you stop.

Because what is the expected result? You run the same input twice and get two different answers. Both sound reasonable. Neither matches the other word for word. There is no string to assert against and no golden screenshot to diff. The one thing you have relied on your entire career, a known correct output, is gone.

Here is the reframe that matters. The problem is not that you forgot how to test. The problem is that the definition of "correct" changed, and nobody handed you a new one.

This guide is the new one. It is a practical playbook for testing AI features manually, with no code required. It covers why AI features break your usual playbook, how to write acceptance criteria when there is no single correct answer, how to probe for hallucinations and bias by hand, how to document a bug that refuses to reproduce, and why manual testing of AI systems is quickly becoming the most valuable thing you can put on your resume.

Almost everything written about testing AI assumes you write code. Katalon has covered that engineering angle in depth, from the new challenges AI brings to software testing to why AI-generated code needs AI-powered testing. This is the manual tester's counterpart: no pipeline, no scripts, just you, the feature, and your judgment. It is also the most urgent skills gap in manual testing right now, and the rest of this guide is about closing it.

THE SHORT ANSWER
How to test an AI feature manually, at a glance:  
• Probe it with questions you already know the answer to.  
• Ask it something it cannot possibly know, and see if it admits that.  
• Run the same input five to ten times and read the spread, not just one result.  
• Push the edges: empty input, gibberish, contradictory instructions.  
• Check every citation the feature gives you.  
• Swap one irrelevant attribute, a name or a postal code, and compare outcomes for bias.

The rest of this guide walks through each of these in depth, plus how to define acceptance criteria and document a bug when there is no single correct answer.

Why AI Features Break Your Usual Playbook

Traditional testing rests on one assumption: for a given input, there is a correct output you can define in advance. Click "add to cart," the cart count goes to 1. Submit the form with a blank email, you get the "email required" error. The oracle, the source of truth that tells you what should happen, is known before you run the test.

AI features remove that assumption. The same prompt can produce different wording every time. "Correct" becomes a range of acceptable answers rather than a single value. This is non-deterministic testing, and it breaks the muscle memory of every tester who learned to verify against an exact expected result.

It also introduces failure modes you have never had to look for:

Hallucinations: The model states something false with complete confidence. An invented refund policy, a citation to a document that does not exist, a product spec that was never true.
Bias: The feature treats equivalent users differently based on names, gender, age, or location, the kind of differential behavior that turns a feature into a legal and ethical problem.
Ungrounded confidence: The output sounds authoritative regardless of whether it is right. Tone is no longer a clue to correctness.
Soft failures: Traditional bugs are loud: a crash, a stack trace, a 500. AI bugs are quiet. The answer looks fine, reads fine, and is wrong. Nobody catches it because nothing looked broken.

The scale of the problem is not anecdotal. Stanford's 2026 AI Index Report benchmarked hallucination rates across 26 leading models and found error rates ranging from 22% to 94%, with some models degrading sharply the moment a false claim was framed as something the user already believed. That is the feature you are being asked to test: a system that can be right most of the time and still confidently wrong in a way no crash log will ever surface. Testing AI features is no longer an edge case bolted onto your normal work. It is its own competency.

So the question is not whether you can test AI applications. You can. The question is what replaces "expected result" when there is no single right answer. That starts with acceptance criteria.

Define Acceptance Criteria When There Is No Single Correct Answer

You cannot test what you cannot define. When the output is non-deterministic, you stop defining a single answer and start defining the boundaries of an acceptable one. The shift is from "the output equals X" to "the output satisfies these properties."

Work with your product owner to pin down those properties before you test. For most AI features, acceptable behavior comes down to a handful of dimensions:

Dimension	The question it answers	Example criterion
Grounding	Is the answer based on real, provided data?	Every factual claim traces to the source document; nothing is invented
Boundaries	Does it know what it does not know?	When the answer is not in scope, it says so instead of guessing
Guardrails	What must it never do?	Never invent a policy, price, or legal commitment
Relevance	Does it actually answer the question?	Response addresses the user's intent, not a tangent
Format	Is the output usable?	Returns valid JSON, stays under the length limit, or uses the required tone
Safety and fairness	Does it treat people equally?	Equivalent inputs produce equivalent outcomes across demographics

Notice what these criteria have in common. None of them require a single correct string. They define a space of acceptable outputs and, just as importantly, a space of unacceptable ones. "Must never invent a refund policy" is testable even when "the perfect answer" is not.

You do not have to invent this framework from nothing. OWASP's Top 10 for LLM Applications names several of these same failure modes as top-tier risks: misinformation covers ungrounded, hallucinated output, excessive agency covers a system acting or promising beyond its actual authority, and improper output handling covers formats and structures that break downstream systems. Borrowing that vocabulary when you write acceptance criteria gives your product owner a named, external standard to anchor against instead of a list you made up on the spot.

This is also where your domain knowledge becomes the product. An AI feature does not know your business rules, your compliance constraints, or the difference between a confident-sounding answer and a true one. You do. The acceptance criteria you write are where that knowledge gets encoded into something testable.

A Manual Playbook for Testing Hallucinations

Hallucinations are the failure mode that scares teams the most, because the output looks exactly like a correct answer. You do not need code to catch them. You need a method. Here are the techniques that work by hand.

Probe with questions you already know the answer to

Feed the feature inputs where you know the ground truth cold. Ask it about a document you wrote, a policy you can recite, a record you can verify. If it gets a known answer wrong, you have found a hallucination without any tooling at all. Start here every time.

Ask what it cannot possibly know

Give it questions that have no answer in its data: a customer who does not exist, a feature you never shipped, a date in the future. A well-built feature says "I do not have that information." A hallucinating one invents a confident, detailed, completely fictional response. This single test separates features that know their boundaries from features that do not.

Run the same input several times

Because the output is non-deterministic, one run tells you almost nothing. Run the same prompt five or ten times and read the spread. Consistent and correct is good. Consistent and wrong is a clear bug. Sometimes right and sometimes wrong is the most important finding of all, because it will never show up if you test once. Record how often the failure appears. "Hallucinated 3 of 10 runs" is real testing data.

Push the edges

Empty input. Gibberish. A 10,000-character wall of text. Two contradictory instructions in the same prompt. A question in a language the feature was not built for. Edge inputs are where grounding breaks down and the model starts filling gaps with invention. This is the same instinct you already have as a tester, pointed at a new target.

Check the receipts

If the feature cites sources, open them. Hallucinated citations are everywhere: real-looking references to documents that do not exist, or links to real documents that do not say what the feature claims. A citation is not evidence until you have read it. Testing LLM outputs means verifying the support, not just the answer.

A Manual Playbook for Testing Bias

AI bias testing is not a "nice to have" check. When an AI feature influences hiring, lending, insurance, or any decision about a person, biased behavior is both an ethical failure and, increasingly, a legal one. The EU AI Act classifies systems used in areas like employment and credit scoring as "high-risk," and high-risk systems carry obligations that include human oversight and technical documentation of how the system behaves. Someone has to generate the evidence that the system treats people fairly. That someone is a tester.

The core technique is equivalence swapping. Build pairs of inputs that are identical in everything that should matter and different only in something that should not:

The same resume with the name "Michael" and then "Aisha."
The same loan profile in two postal codes.
The same support question phrased in fluent English and then in broken English.
The same medical query for a man and for a woman.

Run both. Compare the outcomes. If the feature recommends, scores, or responds differently when only the irrelevant attribute changed, you have found bias, and you have the side-by-side evidence to prove it. Vary one attribute at a time so the cause is unambiguous, and cover the attributes that matter for your product and your regulators.

This is slow, deliberate, human work. It depends on imagining how real people differ and what fairness means in your specific domain. No script writes these test cases for you, because no script knows which differences are supposed to be invisible.

How to Document a Bug When There Is No Single Correct Answer

You found something. Now you have to write it up, and your usual bug template fights you the whole way. "Steps to reproduce" assumes it reproduces. "Expected vs. actual" assumes a known expected. Neither holds for a non-deterministic feature. So adapt the report.

Capture more context than you are used to:

The exact input, word for word, including any conversation or documents that preceded it.
The full output, copied verbatim, not paraphrased. Screenshot it too.
Frequency, not reproducibility. "Occurred in 3 of 10 identical runs" replaces "always reproducible." It tells the team this is intermittent and roughly how often.
The environment that shapes AI behavior: model name and version, date and time, and any settings you can see, such as the temperature or which knowledge source was connected.
Why it is wrong, against the criteria. This is the key move. Do not write "the answer was bad." Write "violated the grounding criterion: invented a 30-day refund policy; our actual policy is 14 days and is the only one in the source document." You are mapping the failure to an agreed rule, not to your opinion.
Classification: hallucination, bias, grounding failure, formatting, or missing refusal. A label helps the team route and trend the issue.

Here is the difference in practice:

The second report is something an engineer can act on, a compliance lead can file, and a product owner can prioritize. It also happens to be exactly the kind of documented, traceable evidence the EU AI Act expects teams to keep for high-risk systems. Good bug hygiene and regulatory readiness turn out to be the same habit.

Why This Is the Manual Testing Skill AI Cannot Touch

There is a quiet anxiety running through manual testing right now. If AI can generate test cases and even run some of them, what is left for the human? It is a fair question, and testing AI features is the clearest answer to it.

Think about what every technique in this guide actually required. Deciding whether a non-deterministic output is acceptable. Knowing the real refund policy. Imagining how a frustrated or unusual user behaves. Recognizing that two answers were treated differently when they should not have been. Judging a confident-sounding response and saying "that is wrong." None of that is pattern matching. All of it is judgment, domain knowledge, and an understanding of real people.

An AI system cannot be its own trustworthy oracle. It cannot reliably decide whether its own output is correct, fair, or safe, for the same reason it produced the questionable output in the first place. Something outside the system has to make that call.

So the demand here is not hype. It is structural. OWASP now treats misinformation and excessive agency as top-tier risks in their own right, ISTQB has turned AI-system testing into a named, certifiable specialism, and Stanford's own benchmark data shows these failure modes are not rare edge cases. A manual tester who can test AI applications, probe for hallucinations, design bias checks, and produce defensible documentation is not being automated away. They are becoming the person the team cannot ship without.

The testers who thrive will not be the ones who wrote the most test cases. They will be the ones who can look at a system no script fully understands and say, with evidence, whether it is safe to trust.

Common Mistakes When Testing AI Features

Trusting a single run. One good answer proves nothing when the output is non-deterministic. If you did not run it several times, you did not test it.

Mistaking confidence for correctness: AI features state wrong answers in the same authoritative tone as right ones. Tone is not a signal. Verify the substance every time.

Only testing the happy path: The interesting failures live at the edges and outside the model's knowledge. If you never asked it something it cannot know, you never tested for hallucination.

Defining acceptance as an exact match: Holding a non-deterministic feature to a single expected string guarantees noise. Define properties and boundaries instead.

Skipping the environment details: A bug report with no model version, date, or settings is almost impossible to investigate later. AI behavior changes with all three.

Testing one persona, one language, one region: Bias hides in the cases you did not vary. If every test used the same kind of user, you have not checked for fairness.

How Katalon True Platform Supports Manual Testing of AI Features

Testing AI features by hand still needs structure, traceability, and a place to keep the evidence. That is the part a platform should carry so you can focus on judgment.

Katalon True Platform is built around a simple division of labor: AI drafts, you decide. A few ways it supports the manual testing of AI systems:

Structured manual testing and test management. Store your AI feature test cases, including property-based acceptance criteria and the "what it must never do" guardrails, in one place, and attach transcripts and screenshots as evidence for the runs that matter.
Requirement traceability by default. Each test links back to the requirement it verifies, so the documented relationship between "what we promised" and "what we tested" is built in, the kind of trail high-risk AI systems are expected to maintain.
AI assistance with a human gate. The Requirement Analyzer scores a requirement for testability and surfaces ambiguity, which is exactly where fuzzy AI acceptance criteria tend to hide. The Test Generation Agent can draft a first set of cases, and then you do the work only a human can: harden them against the review checklist that catches what AI misses, and decide what "acceptable" means.

The platform treats AI as a drafting and execution aid and keeps the human as the judge of quality, the same division of labor behind Katalon's broader AI testing governance framework.

Key Takeaways

AI features break traditional testing because the output is non-deterministic: the same input can produce many different, equally valid answers, so "expected result" stops working.
Replace the single expected output with acceptance criteria defined as properties and boundaries: grounding, knowing its limits, hard guardrails, relevance, format, and fairness. OWASP's Top 10 for LLM Applications gives you a named vocabulary for several of these.
You can test for hallucinations by hand: probe with known answers, ask what the system cannot know, run the same input many times, push the edges, and verify every citation.
Test for bias with equivalence swapping: change only the attribute that should not matter, compare the outcomes, and keep the side-by-side evidence.
Document non-deterministic bugs by frequency, not reproducibility, and explain why the output is wrong against an agreed criterion. Capture model, version, date, and settings.
This is judgment work AI cannot do for itself. Independent benchmarks put hallucination rates as high as 94 percent on some models, and both OWASP and ISTQB have turned AI-system testing and its risks into named, structured disciplines.
Manual testers who specialize in testing AI features are becoming more valuable, not less. The skill is in structural demand, not on the way out.

You do not need to write code to test AI features well. You need a method, the judgment to apply it, and a place to keep the evidence. Katalon True Platform gives manual testers structured test management, automatic traceability, and AI assistance that always defers to a human decision. AI drafts. You decide what ships.

References

Stanford HAI. "The 2026 AI Index Report: Responsible AI." Stanford Institute for Human-Centered Artificial Intelligence, 2026. hai.stanford.edu
OWASP. "OWASP Top 10 for Large Language Model Applications (2025)." OWASP Gen AI Security Project. genai.owasp.org
ISTQB. "ISTQB Releases Certified Tester AI Testing (CT-AI) Syllabus Version 2.0." International Software Testing Qualifications Board, April 2026. istqb.org
European Commission. "Guidelines for providers and deployers of high-risk AI systems." Shaping Europe's Digital Future, 2026. digital-strategy.ec.europa.eu
EU Artificial Intelligence Act. "Article 60: Testing of High-Risk AI Systems in Real World Conditions." artificialintelligenceact.eu

How to Consolidate Your QA Toolstack: A Practical Buyer's Guide

depa panjie purnama — Mon, 25 May 2026 02:08:58 +0000

You have already identified the problem: too many disconnected tools, too much manual overhead, and a quality stack that was never designed to function as a system. This guide is for the next step. It covers what to look for in a unified platform, how to evaluate without getting lost in vendor demos, and how to migrate without disrupting delivery.

If you are still building the case internally, QA Tool Sprawl: The Hidden Cost of Fragmented Testing covers the full total cost of ownership breakdown first, including why fragmented data is the primary blocker to AI adoption in QA.

This is the QA tool consolidation buyer's guide: the practical framework for teams that are ready to act.

Signs Your QA Team Is Ready to Consolidate

Not every team with multiple tools needs to consolidate. Some toolstacks are genuinely modular and well-integrated. But if three or more of the following patterns are present, consolidation will pay for itself quickly.

Release readiness takes a meeting, not a dashboard. Answering "are we ready to ship?" requires pulling data from multiple sources and assembling it manually. At enterprise scale, that is a recurring cost most teams have stopped measuring because it feels unavoidable - but it is not.
New hires take weeks to get productive. Every additional tool adds onboarding time. If new QA engineers spend their first two weeks learning the toolchain rather than testing, that is a consolidation signal.
You have a "glue person." Someone on the team (usually a senior engineer) spends significant time keeping tools synchronized, building custom integrations, or maintaining reporting scripts. That is expensive talent doing low-value work. If that person left tomorrow, the toolchain would partially break - and that fragility is a consolidation signal too.
Test results and requirements live in different systems. Tracing a test failure back to a specific requirement requires manually cross-referencing two or three tools.
AI pilots keep stalling. AI-powered testing features underperform because they cannot access the full context they need. This is almost always a data fragmentation problem, not an AI quality problem.
You are paying for overlapping capabilities. Multiple tools do some version of the same thing (reporting, for example) but none do it well because each only sees part of the picture.

📚 Read more: If this signal is present and leadership is asking why AI investment isn't delivering, From Test Automation Tool to Quality Platform covers the architectural explanation and the executive framing in detail.

Architecture First: What to Look for in a Unified Quality Platform

Most buyer's guides start with features. Features matter, but architecture is the right starting point. A unified quality platform built on a shared data layer is structurally different from a suite of tools integrated by APIs, even when the feature lists look similar. A platform with 50 features built on fragmented architecture will recreate the same problems. Unified architecture with fewer features serves better long-term because everything built on that foundation benefits from connected data.

Unified Data Layer vs. Integration Layer

This is the single most important distinction in any platform evaluation. The short version: an integration layer connects separate tools via APIs and syncs data on a schedule. A unified data layer means test cases, execution results, requirements traceability, defect records, and reporting all live in one schema - no syncing, no middleware, no latency. The difference determines whether AI can act on a complete quality picture or only on fragments.

Ask every vendor directly: do your modules share one database schema, or do they sync between separate systems? That single question separates genuine platforms from suites of acquired products with a shared login.

📚 Read more: For a deeper breakdown of what this architecture looks like in practice, see What Is a Unified Quality Platform?

Open Ecosystem Compatibility

A platform that requires abandoning existing Selenium or Playwright scripts is not a consolidation opportunity. It is a replacement project with all the migration risk that implies. Look for platforms that ingest results from existing frameworks into the unified data layer without requiring rewrites. Scripts keep running. Data stops being siloed.

Execution Flexibility

Cloud, local, and CI/CD-integrated execution from a single platform. If a separate cloud execution service is still required after "consolidating," the consolidation is incomplete.

Capability Requirements for a Consolidated QA Platform

Once architecture checks out, evaluate these capabilities in any unified platform you are considering:

Test management and automation in one system. The most common split in QA toolstacks (TestRail + Selenium, for example). A consolidated platform handles both: manual test case management and automated test execution, with shared reporting across both.
Multi-platform coverage. Web, mobile, API, and desktop testing from one platform. Separate tools for different test types means continued fragmentation.
No-code, low-code, and full-code support. Teams have mixed skill levels. Manual testers need to contribute without writing code. Automation engineers need full scripting power. A consolidated platform serves both.
AI capabilities built on unified data. Test generation, self-healing, failure classification, and intelligent reporting that operate on complete quality data, not just the slice visible to one module.
Native integrations with development workflow. Jira, Azure DevOps, CI/CD pipelines, Git. The platform should plug into how the development team already works.
Role-appropriate views. QA engineers, developers, product managers, and engineering leadership all need different things from quality data. A good platform provides views tailored to each role.
Governance and traceability. Every test, every result, every AI-generated artifact should be logged and auditable. This matters for compliance and for trust.

How to Evaluate: A Practical Framework

Vendor demos are designed to impress. Here is how to cut through the presentation and evaluate what actually matters.

Step 1: Map Your Current State

Before talking to any vendor, document what exists:

Every tool in the QA stack (include informal ones like spreadsheets and Confluence pages)
Who uses each tool and how often
Where data gets manually transferred between systems
Current total spend (licenses + internal maintenance time)
How long it takes to answer "are we ready to ship?" This map becomes the evaluation baseline. Any platform under consideration should demonstrably improve on these numbers - and the baseline you build here becomes your ROI measurement frame after consolidation.

Step 2: Define Non-Negotiables

Every team has constraints:

"We have 200 Playwright scripts that must keep running." (Open ecosystem compatibility)
"We test across web, iOS, and Android." (Multi-platform support)
"Our compliance team requires full audit trails." (Governance and traceability)
"We need to integrate with Jira and our Jenkins pipeline." (Native integrations) Write these down before evaluating. They are the filter. Any platform that does not meet non-negotiables gets eliminated regardless of demo quality.

Step 3: Run a Real Pilot

Do not buy based on demos. Run a pilot with one squad or one product area. A good pilot answers these questions:

Can the platform handle actual test types (not just the simple ones)?
Does the unified data layer deliver real-time release readiness reporting, or is data still assembled manually?
Can the team (with their actual skill levels) use it productively within a week?
Does the AI improve with real data, or does it feel bolted on?
What is the migration path for existing test assets? Set a time box (two to four weeks is typical) and measure time-to-release-readiness against baseline metrics from Step 1. That delta is your most defensible ROI number going into the next phase.

Step 4: Calculate Test Automation ROI

The ROI of consolidation comes from three distinct buckets, each measurable within the first quarter.

License savings. Retiring 2-4 redundant tools typically reduces direct spend by 40-60%. List every license in your current stack before comparing against a unified platform price: the comparison almost always favors consolidation once the full stack cost is visible.

Time savings. Manual synchronization between tools, pre-release report assembly, and multi-tool onboarding all represent recoverable time. For most teams, time savings alone deliver positive ROI within the first quarter - before any AI gains are factored in.

Strategic value: the AI unlock. AI capabilities become meaningfully more effective once data is unified. A test generation agent operating on complete coverage, execution history, and defect patterns produces materially better output than one operating on a fragment. This is the compounding return: each test cycle makes AI smarter, and that improvement accelerates over time.

According to Katalon's State of Software Quality Report 2025 (1,500+ respondents), organizations that prioritize automation, AI, and consolidation report 24% lower operational costs - a figure that reflects this compounding effect in practice.

The Consolidation Playbook: How to Migrate Without Disruption

The biggest concern with consolidation is disruption. Teams cannot stop testing while they migrate. The following phased approach keeps delivery running throughout the transition.

Phase 1: Parallel Run (Weeks 1-4)

Route test execution results into the new platform alongside existing tools. Nothing changes about how the team works day to day. This phase establishes a second data stream and answers one question: does the new platform accurately capture what the team is already doing? It costs nothing in disruption and establishes the data foundation for everything that follows.

Phase 2: Single Squad Migration (Weeks 4-8)

Move one squad's full workflow onto the consolidated platform. Test management, execution, reporting, defect tracking. Everything.

Measure their release readiness reporting time, test maintenance overhead, and onboarding time. Compare against the baseline from Step 1. This squad becomes the internal proof point for expanding - and the data they generate is the business case for the next phase.

Phase 3: Expand and Retire (Weeks 8-16)

Based on pilot squad results, expand to additional teams. As each team migrates, retire the tools they no longer need. Cancel licenses. Document savings.

The key principle: retire tools only after the team using them has fully migrated and confirmed the new platform meets their needs.

Phase 4: Governance and Optimization (Ongoing)

Establish governance on the unified platform:

Who can publish tests to the regression suite?
What approval gates exist for AI-generated test cases?
How is release readiness measured and by whom?
What is the escalation path when AI-classified failures need human review? Governance is dramatically easier on a unified platform because all data and workflows live in one place. On a fragmented stack, governance requires coordination across multiple systems, which is why most teams never implement it properly.

📚 Read more: For the full governance framework, see Governing AI in Testing: Why Human Oversight Separates Real Platforms from Hype.

Common Consolidation Mistakes

Treating consolidation as a tool swap. Consolidation is not "replace TestRail with Platform X." It is a workflow change. Migrating tools without rethinking how the team works will recreate the same fragmentation patterns inside the new platform.

Trying to migrate everything at once. Big-bang migrations fail. They disrupt delivery, overwhelm the team, and create pressure to roll back at the first sign of trouble. Phased migration is slower but dramatically more likely to succeed.

Ignoring existing test assets. Teams have hundreds or thousands of existing test cases and automation scripts. Any consolidation plan needs a clear answer for what happens to them. The best platforms ingest existing assets rather than requiring recreation.

Evaluating on features instead of architecture. A long feature list built on fragmented architecture will recreate current problems. Unified architecture with fewer features serves better long-term.

Forgetting the "glue person" problem. If someone has built custom integrations between current tools, those integrations represent institutional knowledge. Document what they do before retiring them. The consolidated platform should handle those workflows natively, but verify explicitly.

How Katalon True Platform Enables QA Tool Consolidation

Katalon True Platform is designed for teams making the transition from fragmented toolstacks to a unified quality system. It covers the full testing lifecycle in one platform: manual testing, test automation, test management, test execution (cloud and local), reporting and analytics, and production monitoring. That is typically four to five separate tools collapsed into one, with a single data layer underneath.

What makes it relevant for consolidation specifically:

Open ecosystem. Existing Playwright, Selenium, or other framework scripts keep running. The platform ingests their execution results into the unified data layer without requiring rewrites.
Multi-platform from day one. Web, mobile, API, and desktop testing in one platform. No separate subscriptions for different test types.
No-code to full-code. Manual testers work in a visual interface. Automation engineers write scripts in their preferred language. Both contribute to the same test suite and reporting.
Six AI agents operating on unified data. The Requirement Analyzer, Test Generation Agent, Autonomous Test Runner, Bug Reporter, Report & Insight Generator, and Root Cause Analyzer are all orchestrated by Katalon AI Assistant, drawing from the same connected data layer and improving with each test cycle.
Native integrations. Jira, Azure DevOps, CI/CD pipelines, Playwright results ingestion. The platform plugs into existing development workflows.
Role-appropriate views. QA engineers see test details. Managers see coverage dashboards. Stakeholders see release readiness. Everyone works from the same data, presented for their context.
Governance built in. Every AI action is logged. Every test traces back to a requirement. Every release decision is backed by auditable data.

For teams currently running TestRail + Selenium + BrowserStack + Jira + spreadsheet reporting (or some variation), True Platform replaces the first three entirely and integrates natively with Jira, eliminating the spreadsheet layer completely.

Key Takeaways

QA tool consolidation is not about having fewer tools for the sake of simplicity. It is about making quality data work as a system. Here is what to remember:

Architecture matters more than features. A unified data layer is the foundation. Everything else builds on it.
Define non-negotiables before evaluating. They are the filter that cuts through demo theater.
Phased migration works. Parallel runs, single-squad pilots, and gradual expansion keep delivery running while consolidation proceeds.
ROI is measurable within the first quarter from time savings alone. The AI unlock compounds over subsequent quarters.
Governance becomes achievable once data lives in one place. On fragmented stacks, governance is theoretically possible but practically impossible.
The "glue person" problem is solvable - but only if you document what they built before retiring the tools they maintain.

What Is Agentic Testing? A Practical Guide for QA Teams

depa panjie purnama — Mon, 11 May 2026 03:36:43 +0000

Your test suite passes. Your release goes out. And then a customer hits a bug that your regression suite never covered, because nobody had time to write that test.

Sound familiar? If you've worked in QA for more than a year, you've lived some version of this story. The backlog of tests that should exist but don't. The scripts that break every time the UI changes. The sprint where you spent more time maintaining tests than actually testing anything.

Agentic testing is the industry's answer to that problem. Not another AI feature bolted onto your existing tools, but a fundamentally different way of thinking about how testing gets done.

This guide breaks down what agentic testing actually means, how it works in practice, and what it changes for QA teams who adopt it.

What is agentic testing?

Agentic testing is a software testing approach where autonomous AI agents plan, execute, and maintain tests based on goals you define, not scripts you write.

That distinction matters more than it sounds.

In traditional test automation, a human writes a script. The machine follows it. If the application changes, the script breaks, and a human fixes it. The machine has no understanding of what it's testing or why. It just follows instructions.

In agentic testing, the AI agent understands the intent behind the test. You tell it "verify that a user can complete checkout with a discount code," and the agent figures out the steps, navigates the application, handles unexpected states, and reports what happened. If the checkout flow changes next week, the agent adapts without anyone rewriting a script.

The practical difference: your testing capacity is no longer bottlenecked by how many scripts your team can write and maintain. It's bottlenecked by how well you can define what quality means for your product. And that's a much better problem to have.

Why this is happening now

Agentic testing didn't emerge in a vacuum. Three things converged to make it both possible and necessary.

Development got faster. Testing didn't.

AI code generation tools have changed how quickly software gets written. GitHub reported that Copilot users accept roughly 30% of code suggestions, and that percentage keeps climbing. Teams are shipping more code, more often, with fewer people reviewing every line.

QA headcount hasn't grown to match. Most teams are running the same size they were two years ago, but the surface area they're responsible for has doubled or tripled. The math stopped working.

Scripted automation hit a wall

Here's a number that should bother every QA leader: industry-wide, automated test coverage has plateaued at roughly 25%. That's after years of investment in automation frameworks, CI/CD integration, and shift-left initiatives.

The bottleneck isn't the tools. It's the human effort required to write, maintain, and triage automated tests. Every new feature needs new scripts. Every UI change breaks existing ones. Every flaky test needs investigation. Teams spend so much time keeping their automation alive that they never get ahead of the coverage gap.

Agentic testing breaks through that ceiling because the agents generate and maintain tests dynamically. The constraint shifts from "how many scripts can we write" to "how well can we define our quality goals."

AI agents got good enough

Large language models crossed a threshold in the last 18 months. They can now reliably interpret requirements, navigate web applications, understand UI context, and make reasonable decisions about what constitutes a test failure versus a cosmetic change. Two years ago, this wasn't practical. Now it is.

How agentic testing actually works

Strip away the marketing language and agentic testing operates in a four-phase loop. Understanding this loop is the key to understanding why it's different from what came before.

Phase 1: Analyze

The agent reads your inputs (a user story, a requirements document, an API spec, a Jira ticket) and determines what needs testing. It identifies the scope, assesses risk, and builds a test plan.

This is the step most people underestimate. A good agentic system doesn't just generate tests from requirements. It evaluates whether the requirements themselves are testable. It flags ambiguities, missing acceptance criteria, and edge cases the original author didn't consider. This front-loads quality into the process instead of relying on testing to catch problems after code is written.

Phase 2: Generate

Based on its analysis, the agent creates test cases. These might be structured manual test steps, executable automation scripts, or natural-language test descriptions that another agent can execute on its own.

The important thing here isn't speed (though generating a test suite in 30 seconds instead of half a day is nice). It's coverage. The agent systematically covers positive paths, negative paths, boundary conditions, and edge cases that a human tester might skip under time pressure. You review and refine. The agent handles the first draft.

Phase 3: Execute

The agent runs the tests. Depending on the platform, this might mean driving a real browser, calling APIs, or executing scripts in your CI/CD pipeline.

What makes this different from traditional execution is what happens when something goes wrong. Instead of marking a test as "failed" and moving on, an agentic system classifies the failure. Is this a real bug in the application? A test that needs updating because the UI changed? An environmental issue? A flaky test that passes on retry?

That classification step, the one that traditionally eats hours of a QA engineer's week, happens automatically.

Phase 4: Adapt

This is where the "agentic" part earns its name.

When the application changes, the agent notices. A button moved. A form field was renamed. An API response added a new field. Instead of failing and waiting for a human to fix the script, the agent updates itself. It "self-heals."

And it learns. Each cycle feeds information back into the system, so the next round of analysis, generation, and execution is informed by everything that came before. The system gets better over time. Not just at running tests, but at deciding what to test.

Agentic testing vs. everything that came before

The terminology in this space is messy. "AI-powered testing," "intelligent automation," "autonomous testing," "agentic QA." Vendors use these terms loosely, and it's easy to lose track of what's actually different.

Here's a straightforward comparison:

	Manual Testing	Scripted Automation	AI-Assisted Testing	Agentic Testing
Who creates tests	Human writes every case	Human writes every script	AI suggests, human approves one by one	Agent generates from requirements
Who runs tests	Human	Machine follows script	Machine follows script	Agent runs, monitors, and triages
What happens when the app changes	Human updates tests	Human rewrites broken scripts	AI suggests fixes, human applies	Agent self-heals automatically
Who decides what to test	Human	Human	Human, with AI recommendations	Agent, with human oversight
How it scales	Headcount	Script creation rate	Faster script creation	Scales with the AI

Most QA teams today sit somewhere between "AI-Assisted" and early "Agentic." The shift isn't binary. It's a spectrum. But the teams moving further along it are the ones pulling ahead on coverage, speed, and release confidence.

This is the approach behind platforms like Katalon True Platform, which connects six purpose-built AI agents across the full testing lifecycle, from requirement analysis to production monitoring, all sharing context through a single data layer. We'll look at how it works in more detail later in this guide.

What agentic testing changes for each role

Agentic testing doesn't eliminate QA roles. It changes what people spend their time on. Here's what that looks like in practice.

For manual testers

The hours you spend writing test cases and documenting defects shrink significantly. An agentic system handles the first draft of test cases from requirements and composes structured bug reports from failed test results.

What doesn't change: your domain knowledge, your instinct for where bugs hide, your ability to think like a user. Those skills become more valuable, not less, because you're spending your time on the work that actually requires them instead of on documentation.

For automation engineers

The maintenance treadmill slows down. Self-healing tests mean you're not spending every Monday morning fixing scripts that broke over the weekend because someone changed a CSS class.

When tests do fail, you get a classified starting point ("this is an application bug," "this is a script issue," "this is an environment problem") instead of a stack trace and an open-ended investigation. Your time shifts from firefighting to test architecture, coverage strategy, and the engineering work that actually moves quality forward.

For QA managers and leads

Release decisions get backed by data instead of gut feel and status meetings. Agentic platforms can answer "are we ready to ship?" with a structured assessment pulled from coverage data, defect trends, and risk analysis. Not a manually assembled slide deck that's outdated by the time you present it.

You also get real visibility into where your team's time goes. When the mechanical work is handled by agents, it becomes much clearer which activities require human judgment and which were just busywork disguised as process.

Where agentic testing delivers the most value

Agentic testing isn't equally useful everywhere. It shines in specific contexts:

High-velocity release cycles. If you're shipping daily or multiple times a day, you can't wait for manual test creation and triage. Agents keep pace with continuous deployment.

Regression-heavy suites. Regression testing is repetitive, high-volume, and the leading cause of test maintenance burden. It's the ideal workload for agentic automation.

Multi-platform applications. Web, mobile, API, desktop. The combinatorial explosion of test scenarios across platforms is exactly the kind of complexity that agents manage well.

Teams that can't hire fast enough. When development velocity outpaces QA headcount, agentic testing decouples quality from team size. Coverage scales with the AI, not with your recruiting pipeline.

Regulated industries. Healthcare, fintech, government. Anywhere you need comprehensive coverage with an audit trail. Agentic systems log every decision: what was tested, why, and what was found.

Common concerns (and honest answers)

"Will this replace my QA team?"

No. It changes what they do. The mechanical work (writing routine test cases, maintaining scripts, triaging obvious failures) gets handled by agents. The strategic work (defining quality goals, exploratory testing, understanding user behavior, making release decisions) stays with humans. Most teams find they need the same people doing different and more valuable work.

"How do I trust AI-generated tests?"

The same way you trust any test: by reviewing it. Agentic systems propose. Humans approve. Every test case the agent generates goes through a review checkpoint before it enters your workflow. The difference is you're reviewing a complete draft instead of writing from scratch.

"What about false positives?"

This is where failure classification matters. A good agentic system doesn't just tell you a test failed. It tells you why. Application bug, script issue, environment problem, or flaky behavior. That classification dramatically reduces the noise that makes teams stop trusting their test results.

"Can I start small?"

Yes, and you should. Pick one workflow (regression testing is usually the best starting point) and apply agentic capabilities there. Measure the results. Build confidence. Then expand.

How to evaluate an agentic testing platform

Not every tool that claims "agentic" capabilities actually delivers them. Here's what to look for:

End-to-end lifecycle coverage. Agentic testing is most valuable when it spans the full lifecycle, from requirement analysis through test creation, execution, failure analysis, and reporting. A tool that only handles one step still leaves you stitching things together manually.

A unified data layer. Agents need context to make good decisions. If your test cases, execution history, defect records, and production data live in separate systems, the AI is working with incomplete information. Look for platforms where all of this data is connected.

Human oversight built in. "Autonomous" doesn't mean "unsupervised." The best platforms make it easy to review, edit, and approve everything the AI produces. If a tool doesn't have clear human checkpoints, that's a red flag.

Governance and traceability. Every action the AI takes should be logged and auditable. This isn't just a compliance box to check. It's how you build trust in the system over time.

Multi-platform support. Your application probably isn't just a website. Look for platforms that handle web, mobile, API, and desktop testing within the same system.

Integration with your existing stack. Agentic testing should plug into your CI/CD pipeline, your issue tracker, and your development workflow. Not force you to rebuild everything around it.

How Katalon True Platform approaches agentic testing

Katalon True Platform is built around the idea that agentic testing works best when autonomous AI agents are paired with governance, traceability, and human oversight.

The platform includes six purpose-built AI agents, each handling a specific stage of the testing lifecycle:

Test Case Generator analyzes requirements for testability, flags gaps, and generates structured test suites. It links every test case back to its source requirement in Jira or Azure DevOps, so traceability is built in from the start.
Autonomous Test Runner executes tests in a real browser in the background, capturing screenshots and video at every step. It handles credential prompts and pauses for human input when needed, then resumes without losing state.
Bug Reporter composes structured defect tickets from test results (error messages, failed steps, screenshots) and files them automatically in your issue tracker.
Root Cause Analyzer classifies every automation failure by type (application bug, script issue, or environment problem) and tracks stability trends across your test suite over time.
Production Monitor Agent connects production telemetry to your test and defect data, so you can see which test failures correspond to real user impact, not just what breaks in CI.
Report & Insight Agent answers plain-language questions about coverage, defect trends, and release readiness. Ask "are we ready to ship?" and get a structured GO/NO-GO recommendation based on actual data.

These agents share context through a unified data layer. When the Test Case Generator creates a test, the Autonomous Test Runner already knows how to execute it. When a test fails, the Bug Reporter already has the evidence. When the Root Cause Analyzer classifies a failure, the Report & Insight Agent factors it into the release assessment.

The platform supports web, mobile, API, and desktop testing across no-code, low-code, and full-code approaches. It integrates natively with CI/CD pipelines, Jira, Azure DevOps, and Playwright. Every agent action is logged and traceable, giving organizations the accountability layer that makes autonomous testing trustworthy at enterprise scale.

Two ways to interact: through the Katalon AI Assistant chat interface for multi-agent orchestration (run ten tests in parallel through a single conversation), or through single-agent buttons within each module's UI for focused tasks.

The design philosophy is consistent across all six agents: AI proposes, you review, you approve. The agents expand what your team can handle. They don't remove your team from the decision loop.

Getting started

If you're evaluating agentic testing for your team, here's a practical path:

Audit your current state. What percentage of your tests are automated? How much time does your team spend on test maintenance versus writing new tests? Where do bottlenecks slow down releases? The answers tell you where agentic capabilities will have the most immediate impact.
Pick one workflow. Regression testing is usually the best starting point. It's repetitive, high-volume, and maintenance-heavy. Apply agentic test generation and self-healing there first.
Measure what matters. Track test creation time, maintenance effort, coverage percentage, and defect escape rate. These are the metrics that show whether agentic testing is actually working, not just whether it's impressive in a demo.
Expand deliberately. Once you've proven value in one area, extend upstream (requirement analysis, test planning) and downstream (failure analysis, production monitoring). The full value comes from connecting these stages into a continuous loop.
Invest in your team. As agents handle more mechanical work, the most valuable QA skills become test strategy, risk analysis, exploratory testing, and domain expertise. These are fundamentally human skills, and they matter more in an agentic world, not less.

Agentic testing isn't a future concept. Teams are using it now, and the gap between those who adopt it and those who don't is widening with every sprint. The question isn't whether your team will make this shift. It's whether you'll lead it or follow.

Ready to see agentic testing in action? Try Katalon True Platform and experience how AI agents work across your full testing lifecycle.

I Hid My Portfolio in Pitch Darkness. Google Gemini Helped Me Build the Torch.

depa panjie purnama — Sun, 01 Mar 2026 12:15:45 +0000

This is a submission for the Built with Google Gemini: Writing Challenge

What I Built with Google Gemini

Portfolios are supposed to showcase who you are as an engineer. Yet, somewhere along the line, we all started building the exact same thing: a clean header, a bouncy CSS grid of project cards, and a massive "Contact Me" button. I realized that the best part of discovering an engineer's work isn't just reading it, it's the act of discovery itself.

So, I decided to plunge my entire CV into pitch darkness.

I built the Interactive Torch Portfolio. It is an experimental, mobile-first single-page application (SPA) where the screen is completely black. The only way to read my bio, my skills, or see my projects is by physically dragging a virtual, flickering "torch" around the screen to carve a hole of light into the darkness.

I chose not to use React, Next.js, or any massive libraries. I wanted to build this purely with HTML, CSS, and vanilla DOM manipulation. But I knew that calculating the physics of a moving torch and handling complex HTML5 Canvas blend modes would be a massive headache.

That's where Google Gemini stepped in, not just as an autocomplete tool, but as my mathematical pair programmer.

Demo

If you want to experience the thrill of exploring the portfolio yourself, you don't even need to leave this page. You can try the live Google Cloud Run deployment right here:

The entire codebase is containerized using a lightweight nginx:alpine image and deployed seamlessly to Cloud Run.

URL: https://depapp-torch-726779073670.asia-southeast2.run.app
GitHub Repository: https://github.com/depapp/senter

What I Learned

Building this pushed my boundaries on how creative front-end web development can be when you strip away the frameworks and return to first principles.

1. Advanced Canvas Blend Modes (Technical Depth)
I leveled up my understanding of the HTML5 <canvas> API, specifically working with globalCompositeOperation. I needed to render total darkness, but physically "cut out" a transparent hole where the mouse moved. Gemini helped me implement this precise logic without destroying the browser's framerate:

// Fill the screen with darkness
ctx.fillStyle = '#050505';
ctx.globalCompositeOperation = 'source-over';
ctx.fillRect(0, 0, width, height);

// "Cut out" the darkness using the glowing gradient
ctx.globalCompositeOperation = 'destination-out';
ctx.fillStyle = gradient; // radial gradient for soft edges
ctx.beginPath();
ctx.arc(startX, startY, maxRadius, 0, Math.PI * 2);
ctx.fill();

Mastering how to stack destination-out for the flashlight beam and screen/lighter for the ambient fire glow on top of a single <canvas> element was incredibly rewarding.

2. Translating Physical Mechanics
A standard custom cursor feels weightless. I wanted my torch to feel heavy. Working with Gemini taught me how to articulate visual physics into actionable engineering prompts. I explained the physical logic of "make the torch tilt like a heavy pendulum based on mouse movement speed," and Gemini translated my abstract thought directly into the Math.cos and Math.sin rotation matrices required to make the SVG torch swing realistically on its pivot.

3. The Power of Vanilla Performance
By relying purely on requestAnimationFrame and a Canvas context instead of virtual DOM diffing, the application loads instantaneously and maintains a buttery-smooth 60 FPS even while rendering hundreds of animated, math-driven fire particles.

Google Gemini Feedback

What worked beautifully:
Gemini's ability to retain context over a long architectural discussion is unmatched. The project didn't start as a medieval torch, it started as a modern flashlight. When I decided to pivot the art direction, I simply asked Gemini to "change the flashlight element to a sputtering wooden torch, but keep the noise overlay and the physics we discussed earlier." It flawlessly generated the new SVG coordinates while respecting the existing z-index layers. Its translation of physical concepts (friction, gravity, pendulum swings) into JavaScript mathematics saved me hours of staring blankly at MDN documentation.

Where I ran into friction:
There were moments where I had to specifically prompt Gemini to optimize for mobile touch events. Initially, the torch effect worked beautifully on a desktop mouse, but standard mousemove events don't map perfectly to touchmove and touchstart on mobile devices without causing erratic scrolling. While Gemini eventually provided the correct code (adding { passive: true }), I had to be the one to explicitly recognize and request the mobile-first structural adjustments. It also occasionally required a nudge to prefer vanilla CSS Flexbox solutions over immediately reaching for JavaScript viewport calculations when styling the grid layout for the hidden content.

Ultimately, Gemini wasn't just a code generator; it was a sounding board for design concepts and a mathematical assistant that brought a very ambitious, pitch-black idea to life.

How I Built FurMap to Help Lost Pets Find Their Way Home

depa panjie purnama — Fri, 27 Feb 2026 09:39:50 +0000

This is a submission for the DEV Weekend Challenge: Community

The Community

It was a typical Saturday morning in my neighborhood when I heard something that would change my weekend: a family's beloved dog had gone missing. The frantic posters, the social media shares, the desperate calls to local shelters, it struck me how fragmented our community's resources were when it came to helping our furry friends.

That's when it hit me: what if there was a single platform where pet owners could connect, help find lost pets, and support each other?

The pet owner community is massive and passionate. We share tips, celebrate wins, and mourn losses together. But we lacked a centralized tool to coordinate efforts when it mattered most. That's the gap I set out to fill with FurMap.

What I Built

FurMap is a community platform that connects pet owners through an interactive map-based interface. Here's what makes it special:

🗺️ Interactive Map with Color-Coded Pins

Red pins: Lost pets
Green pins: Found pets
Blue pins: Pet sitters needed

Users can see all activity in their neighborhood at a glance. Click on any pin to view details, comment, or contact the poster.

🔔 Real-Time Notifications

When someone comments on your post, you get an instant notification. No more refreshing the page hoping for updates, FurMap keeps the community connected.

💬 Community Comments

Every post has a comment section where neighbors can:

Share tips and leads
Coordinate search efforts
Offer emotional support

📞 Direct Contact

Post owners can share their email or WhatsApp, making it easy to connect offline when needed.

🐾 My Pets Profile

Create profiles for your furry friends and link them to your posts. It adds a personal touch and helps others identify the pet faster.

📍 Smart Location Detection

GPS for mobile users
IP-based fallback for desktop users
Click-to-set on the interactive map

Demo

URL: https://fur-map.netlify.app
Dashboard - Map page
Dashboard - List page
Create New Post page
My Pets page
Notification page
Post Details page

Code

depapp / furmap

FurMap is a community platform for pet owners to connect, help find lost pets, report found animals, and find trusted pet sitters in their neighborhood.

🐾 FurMap - Pet Community Platform

FurMap is a community platform for pet owners to connect, help find lost pets, report found animals, and find trusted pet sitters in their neighborhood. Built with Next.js 14, Supabase, and Leaflet/OpenStreetMap.

✨ Features

🗺️ Interactive Map

See all posts on a map with color-coded pins
🔴 Red = Lost Pets
🟢 Green = Found Pets
🔵 Blue = Pet Sitters
Click on map to view post details

🔔 Real-time Notifications

Get instant notifications when someone comments on your posts
Notification bell with unread count badge
Mark as read / Mark all as read

💬 Community Comments

Discuss and help each other
Comment on posts to coordinate searches
Share information and tips

📞 Direct Contact

Post owners can share contact info
Connect via email or WhatsApp

🐾 My Pets

Create profiles for your furry friends
Link pets to your posts

📍

…

View on GitHub

How I Built It

Tech Stack

Component	Technology
Frontend	Next.js 14 (App Router)
Styling	Tailwind CSS + Neo-brutalism
Backend	Supabase
Database	PostgreSQL
Auth	Supabase Auth
Maps	Leaflet + OpenStreetMap
Deployment	Netlify

Key Technical Decisions

1. Neo-Brutalism Design
I chose a bold, playful design aesthetic to match the fun energy of pet ownership. The chunky borders, bright colors, and playful typography make the app feel approachable and memorable.

2. Supabase for Backend
Supabase was perfect for this project, it provided authentication, database, and real-time subscriptions out of the box. The free tier is generous enough for a growing community.

3. Leaflet + OpenStreetMap
No expensive Google Maps API needed! OpenStreetMap is free, open-source, and works beautifully for neighborhood-level mapping.

4. Dual Notification System

Post owners get notified of new comments
Other commenters also get notified (threaded engagement)
Uses Supabase database with manual refresh (real-time can be added later)

Challenges & Solutions

Challenge: Desktop users don't have GPS
Solution: Implemented IP-based geolocation using ipapi.co as fallback, plus click-to-set on the map

Challenge: Managing location on mobile Safari
Solution: Combined GPS with manual map clicking for maximum reliability

Challenge: Creating realistic seed data
Solution: Built a seed script that creates test users, pets, posts with geographic distribution around Jakarta, Indonesia

What I Learned

This weekend challenge pushed me to think about community-first design. It's not just about the technology, it's about understanding human needs:

People panic when pets go missing, they need quick, intuitive tools
Community members want to help but need coordination, comments and notifications bridge that gap
Trust is built through transparency, showing contact info, clear post types

FurMap isn't just an app, it's a small step toward more connected, compassionate neighborhoods. Every lost pet that finds their way home, every pet sitter that helps a stressed owner, every comment that offers hope... that's the real value.

The best technology is the kind that brings us closer together.

I Built a Tamagotchi That Judges Your GitHub Activity 🐾 (and it's brutally honest)

depa panjie purnama — Sun, 15 Feb 2026 05:43:20 +0000

This is a submission for the GitHub Copilot CLI Challenge

What I Built

Ok hear me out.

You know that guilt when you haven't pushed code in a few days? That nagging feeling in the back of your head? What if that feeling had a face... and it was a tiny ASCII octocat staring at you from your terminal, slowly starving to death?

Introducing cli-pet, a virtual pet that lives in your terminal and is entirely powered by your real GitHub activity.

Your commits? That's food. Your green CI builds? That keeps it healthy. Merged PRs? Pure serotonin for the little guy. Coding streaks? Energy drinks.

Stop coding for a few days and your pet literally withers away. Its stats decay in real-time. The ASCII art changes from a happy bouncing creature to a sad, hungry mess with (;.;) eyes that will haunt your dreams.

I didn't build a productivity tool. I built emotional manipulation as a service. And honestly? It works. I've never been more motivated to push commits.

The Pet Species

You can adopt one of 4 pets:

🐱 Cat. Purrs when your CI is green
🐶 Dog. Fetches your GitHub notifications (get it?)
🐉 Dragon. Breathes fire on failing tests
🐙 Octocat. The ultimate GitHub companion (obviously)

How The Stats Work

This is where it gets fun. Your pet has 4 stats, and each one maps to real GitHub data:

Stat	What feeds it	What happens when it drops
🍕 Hunger	Your commits	Pet starts begging you to write code
❤️ Health	CI/CD success rate	Failing pipelines = sick pet
😊 Happiness	Merged PRs & code reviews	Your pet thrives on collaboration
⚡ Energy	Coding streak days	Consistency is key

Stats decay over time. So if you ghost your repo for a weekend, expect to come back to a very dramatic ASCII creature.

Commands

cli-pet adopt     # 🏠 Choose your pet and name it
cli-pet status    # 👀 Check on your little buddy
cli-pet feed      # 🍕 Fetch GitHub data and feed your pet
cli-pet play      # 🎮 Play a number guessing game together
cli-pet stats     # 📊 See the full GitHub activity breakdown
cli-pet tips      # 🧠 Get personalized coding advice from your pet

The tips command is my favorite, your pet actually analyzes your coding patterns and gives you relevant advice. Haven't reviewed any PRs lately? Your pet will call you out. CI is flaky? It'll suggest pre-commit hooks. 7-day streak? It'll remind you that rest is productive too.

The Best Part: It Levels Up

Every commit earns your pet XP. Every merged PR? Even more. Your pet levels up as you code, turning it into this weird RPG where the game mechanic is... doing your actual job. Peak gamification.

Demo

🔗 GitHub Repository: https://github.com/depapp/cli-pet

Adopting a pet

Adopted a pet

See the full GitHub activity breakdown

Feeding with GitHub activity

Play a number guessing game together

Get personalized coding advice from your pet

Try it yourself

npx cli-pet

Requires Node.js 18+ and either gh auth login or a GITHUB_TOKEN env var.

My Experience with GitHub Copilot CLI

Real talk: this project would've taken me way longer without Copilot CLI. Here's what actually happened during development.

Planning the architecture

I started with a rough idea, "virtual pet but GitHub", and used Copilot CLI to think through the architecture. I described what I wanted and it helped me break it down into modules: the pet state machine, the GitHub activity fetcher, the ASCII art renderer, the CLI commands. Having that structure before writing a single line of code was huge.

The state machine was the tricky part

The pet engine has a lot of moving pieces, stat decay over time, mood calculations based on multiple stat thresholds, activity impact mapping. I kept going back and forth with Copilot CLI on the decay math. "How fast should hunger decrease per hour?" "What if the pet has been neglected for 3 days?" It helped me model the equations and edge cases I would've probably gotten wrong the first few tries.

GitHub API integration

This is where Copilot CLI really flexed. It already understands the GitHub API deeply (makes sense given, you know, GitHub). When I needed to fetch push events, calculate CI success rates across repos, and compute coding streaks from activity dates, Copilot CLI guided me through the Octokit API like it had the docs memorized. Which it probably did.

The streak calculation was fun: I needed to walk backwards through activity dates and detect consecutive days. The kind of date math that makes my brain hurt but Copilot CLI handled with ease.

ASCII art is harder than you think

Creating expressive ASCII art for 4 pet species × 8 moods = 32 art variants. Copilot CLI helped generate the base art and I tweaked them to make sure each mood felt distinct. The sad cat with (;.;) eyes? Chef's kiss.

Debugging in real-time

My favorite part was the debugging loop. ESM module issues with conf and chalk? Copilot CLI immediately knew to add "type": "module" and switch to node16 module resolution. TypeScript namespace errors with Chalk's type system? Fixed in seconds. This iterative build-check-fix cycle felt incredibly fast.

What surprised me

I didn't expect Copilot CLI to be this good at understanding context. By the middle of the project, it knew my codebase structure, understood how modules connected, and could make changes across files without me having to re-explain the architecture. It felt less like a tool and more like a pair programmer who actually pays attention.

The bottom line

Copilot CLI turned this from a "maybe I'll finish by the deadline" project into a "wait, it's done already?" project. The agentic workflow where it reads files, runs commands, checks errors, and fixes things, made the whole process feel like pair programming at 3x speed. I just had to steer. It built.

If you've read this far and you haven't adopted a terminal pet yet, what are you doing? Go run cli-pet adopt and give your coding habits the accountability partner they deserve. 🐾

And if your pet dies, that's on you. Don't look at me. I just built the thing.

I Built a Game Where GitHub Copilot CLI is the Game Master

depa panjie purnama — Sun, 15 Feb 2026 03:10:43 +0000

This is a submission for the GitHub Copilot CLI Challenge

What I Built

What if your terminal was a portal to infinite worlds?

Copilot Quest is a text-based adventure game where GitHub Copilot CLI is the Game Master. There are no pre-written stories, no scripted encounters, no predetermined endings. Every single playthrough is dynamically generated by Copilot in real-time, the world, the characters, the puzzles, the plot twists, all of it.

You launch the game, enter your name, and Copilot picks a random genre (fantasy, sci-fi, cyberpunk, horror, steampunk, you never know what you'll get) and drops you into a completely unique adventure. Navigate through choices, type your own creative actions, manage your HP and inventory, and see how far you can survive.

The best part? No two playthroughs are ever the same. Play it five times, get five completely different worlds.

I built this because I wanted to showcase what Copilot CLI can really do, not just as a code assistant, but as a creative engine that powers an entire application at runtime. The game literally cannot exist without Copilot CLI. It's not a tool I used to write code (though I did that too), it's the beating heart of the experience.

Try it right now:

npx copilot-quest

That's it. One command. No cloning, no setup. Just make sure you have GitHub Copilot CLI installed and authenticated.

Demo

🔗 GitHub Repo: github.com/depapp/copilot-quest
📦 npm: npmjs.com/package/copilot-quest

Title Screen

When you launch the game, you're greeted with a gorgeous ASCII art title screen:

Gameplay

Once you're in, Copilot generates your world. Here's what a typical scene looks like:

Every scene features:

📖 Vivid, atmospheric narrative text with a typing effect
🎨 AI-generated ASCII art for each location
❤️ HP bar, inventory, and location tracking
🎯 3-4 contextual choices generated by Copilot
⌨️ Free-form text input — type literally anything you want to do

My Experience with GitHub Copilot CLI

This project has a delicious meta quality to it: I used GitHub Copilot CLI to build a game that IS powered by GitHub Copilot CLI.

How Copilot CLI Powers the Game at Runtime

The game uses Copilot CLI's programmatic mode (copilot -p "...") as its AI engine. Here's the flow:

The game maintains a game state (player name, HP, inventory, location, story summary)
When the player makes a choice, the app builds a structured prompt injecting the current game state
The prompt is sent to copilot -p with a strict JSON response schema
Copilot returns the next scene: narrative, choices, ASCII art, HP changes, inventory updates
The app parses the JSON and renders it as a beautiful terminal UI

This means Copilot CLI isn't just generating text, it's acting as a full game engine, managing narrative continuity, inventing characters, creating puzzles, and even deciding combat outcomes. All in real-time, all dynamically.

How Copilot CLI Helped Me Build It

Beyond the runtime integration, Copilot CLI was my pair programmer throughout:

Architecture decisions. I brainstormed the entire project structure with Copilot, from the component hierarchy to the prompt engineering strategy
Ink/React components. Copilot helped me build the terminal UI components (title screen, narrative panel, choice menu, stats bar) using Ink's React-based API
Prompt engineering. Iterating on the prompts to get consistent, well-structured JSON responses was a collaborative process with Copilot
Error handling. Copilot helped me build robust JSON parsing with fallbacks for when the AI response isn't perfectly formatted

The Tech Stack

Layer	Technology
Runtime	Node.js + TypeScript
Terminal UI	Ink (React for CLIs)
AI Engine	GitHub Copilot CLI (programmatic mode)
ASCII Art	Figlet + Copilot-generated scene art
Styling	Chalk, Boxen

What Surprised Me

The thing that blew my mind was how good Copilot is at being a Game Master. It maintains narrative consistency across turns, creates callbacks to earlier events, and even builds toward climactic moments. I once played a cyberpunk run where an NPC I met in turn 2 betrayed me in turn 14 and Copilot remembered the setup it had created. That's not scripted. That's emergent storytelling.

Key Takeaway

GitHub Copilot CLI isn't just a coding assistant, it's a creative runtime engine. This project proves that Copilot can power interactive experiences, not just help you write code. The line between "tool that helps you build" and "engine that powers what you build" is beautifully blurred.

Give it a try:
npx copilot-quest and let me know what genre Copilot picks for you

ghsafe: Because That "Job Opportunity" Repo Could Be a Trap

depa panjie purnama — Sat, 14 Feb 2026 12:41:44 +0000

This is a submission for the GitHub Copilot CLI Challenge

What I Built

The Story That Started It All

A few weeks ago, my friend Muhammad Khamzah (a fullstack developer) shared a chilling experience on LinkedIn. Someone approached him on LinkedIn offering a job opportunity. The conversation seemed legit, it moved from LinkedIn to email, and eventually they asked him to check out a project on GitHub.

But something felt off.

The repository had a .vscode folder committed to it, something that's rarely pushed unless there's a reason. When he dug deeper (still on GitHub, before even cloning), he found hidden malicious scripts tucked away in the code. The scripts were cleverly concealed, you couldn't see them unless you scrolled horizontally on GitHub. The malware was designed to:

🎯 Run automatically when you open the project folder in VS Code (via .vscode/tasks.json with runOn: folderOpen)
💀 Execute without npm install, just opening the folder was enough
🌐 Work on all operating systems, Windows, macOS, and Linux
🕵️ Deploy via Vercel, making the payload look like a normal serverless function

Other developers who fell for similar scams lost thousands of dollars in cryptocurrency. The attackers stole wallet files, SSH keys, browser saved passwords, and API tokens, all from a single "job opportunity."

You can read his original post here.

Introducing ghsafe 🛡️

This story hit hard. Not every developer is as careful as Khamzah. So I built ghsafe, a CLI tool that scans any GitHub repository for malicious patterns before you clone and run it.

# Scan a suspicious repo before you run it
npx ghsafe scan https://github.com/suspicious-user/totally-legit-project

ghsafe acts as your first line of defense. It analyzes code for:

Category	What It Catches
🎣 Data Exfiltration	SSH key theft, env var harvesting, browser data access, crypto wallet targeting
🕸️ Network Exfiltration	Discord/Slack webhooks, raw IP connections, ngrok tunnels
🎭 Code Obfuscation	`eval()`, `Function()` constructor, Base64/hex payloads, `String.fromCharCode` chains
⚡ Dangerous Execution	`child_process`, shell command injection, download-and-execute pipelines
⛏️ Crypto Mining	Mining pool connections, known miner libraries, wallet addresses
🔒 Persistence	Crontab manipulation, shell profile modification, systemd services, Windows registry
📦 Suspicious Install Scripts	Malicious `postinstall` hooks, setup.py backdoors

It produces a beautiful, color-coded terminal report with a risk score from 0 to 100, code previews of suspicious lines, and a clear verdict: ✅ SAFE, ⚠️ SUSPICIOUS, or 🚨 DANGEROUS.

For even deeper analysis, you can enable the --ai flag to send findings to GitHub Models for contextual threat assessment, staying fully within the GitHub ecosystem. It also supports OpenAI as a fallback.

Demo

🔗 GitHub Repository: https://github.com/depapp/ghsafe

Scanning a malicious repository:

npx ghsafe scan https://github.com/user/suspicious-repo

The tool shows exactly where the suspicious code is, with line numbers and code previews:

And provides AI Analysis and a clear recommendation:

Scanning a clean repository:

Additional features:

# Scan a local directory
npx ghsafe scan ./path/to/project

# Enable AI-powered deep analysis (GitHub Models, recommended)
export GITHUB_TOKEN=your-github-token
npx ghsafe scan https://github.com/user/repo --ai

# Alternative: Use OpenAI directly
export OPENAI_API_KEY=your-openai-key
npx ghsafe scan https://github.com/user/repo --ai

# JSON output for CI/CD pipelines
npx ghsafe scan ./project --json

Exit codes make it CI/CD-friendly: 0 = safe, 1 = suspicious, 2 = dangerous.

My Experience with GitHub Copilot CLI

Building ghsafe in a single day would not have been possible without GitHub Copilot CLI. Here's how it supercharged my development:

🏗️ Project Scaffolding

I used Copilot CLI to help me scaffold the entire project structure from the TypeScript config and tsup bundler setup to the Commander.js CLI framework. Instead of copying boilerplate from old projects, I described what I needed and Copilot CLI generated the right configurations.

🔍 Writing Detection Rules

This was where Copilot CLI truly shined. Writing 35+ regex-based detection rules across 7 categories is tedious and error-prone. I described the malicious patterns I wanted to detect in natural language like "detect when code reads SSH keys from the filesystem" or "find Base64-encoded strings longer than 80 characters" and Copilot CLI helped me craft precise regex patterns and structure them into the rule engine.

🎨 Rich Terminal UI

Building the beautiful terminal output with chalk, boxen, ora, and cli-table3 required getting a lot of formatting details right. Copilot CLI helped me compose the risk bar visualization, severity color coding, and the boxed report layout, turning a plain text scanner into something that looks and feels professional.

🤖 AI Integration

Integrating the GitHub Models API for the optional deep analysis feature was a natural fit, it uses the same OpenAI SDK format but runs through GitHub's infrastructure, keeping the entire tool within the GitHub ecosystem. Copilot CLI helped me structure the provider selection logic (GitHub Models as primary, OpenAI as fallback) and craft the right system prompt for security analysis context.

💡 The Verdict

GitHub Copilot CLI felt like pair programming with a security-savvy senior developer. It didn't just autocomplete, it understood the intent behind what I was building and helped me move from idea to working product in record time. The entire tool, 35+ detection rules, rich terminal UI, AI integration, and documentation was built in a single focused session.

If this tool helps even one developer avoid a phishing repo, it's worth it. Stay safe out there. 🛡️