From a multi-agent testing experiment to a broader discussion about how engineering teams should evolve in the age of AI.
At first, I simply wanted to share an AI-powered automated testing system I had been working on.
The system could already understand requirements, generate test cases, interact with web browsers and mobile apps, identify defects, and connect bug fixing with regression testing. I expected the session to focus on how AI could improve testing efficiency.
But during our internal session on November 5, 2026, the discussion gradually moved beyond testing.
We started talking about how product, development, and QA teams work together. We discussed why requirements so often fall out of sync, and how AI might change the way engineering teams are organized.
By the end of the discussion, the session had become the final push that helped set our company's organizational transformation in motion.
Looking back, I find that quite interesting. We set out to help testers work more efficiently with AI. Instead, we ended up discussing how AI might force the entire engineering organization to rethink the way it works.
1. 🤖 I Wanted AI to Handle the Entire Testing Process
I've been working on a multi-agent automated testing system.
Rather than simply asking AI to generate automation scripts from test cases, I wanted to connect the entire process: understanding requirements, designing tests, executing them, reporting defects, verifying fixes, and running regression tests.
The workflow looks roughly like this:
- AI reads product requirement documents and prototypes to understand business workflows and acceptance criteria.
- It generates test cases, which are then reviewed by multiple agents.
- Based on those test cases, AI controls web browsers and mobile apps, simulating real user interactions to execute the tests.
- When it finds an issue, AI generates a defect report and submits it to our issue-tracking system.
- Each reported defect goes through a second review and human confirmation before being assigned to the appropriate developer.
- After the developer fixes the issue, AI automatically runs regression tests to verify the fix and move the defect through the closure process, with human oversight retained.
The end-to-end workflow, from requirements to regression testing, is now in place and working.
What impressed me most was that AI had started taking over some of the most frustrating parts of traditional test automation.
With UI automation, the hardest part was often not writing the initial scripts. It was maintaining them.
A developer changes a UI component, and testers may have to revisit element locators, page objects, and interaction scripts. The more frequently the interface changes, the higher the maintenance cost. Sometimes, testers spend more time maintaining automation scripts than actually running tests.
Now, AI can observe the current page, understand its elements and interaction states, and perform actions based on the testing objectives. It still has to deal with UI changes and execution failures, but it can already take over a considerable amount of work that previously required repeated manual maintenance.
That made me reconsider the limits of automated testing.
2. 🔍 Automated Testing Should Do More Than Repeat Known Test Cases
For a long time, I held a fairly firm view of automated testing:
Automation is useful for regression testing, but it is not very good at discovering new defects.
The reason is straightforward. Traditional automated testing relies heavily on test cases designed in advance, fixed execution steps, and predefined assertions. It can repeatedly execute known test paths, but it struggles to discover problems that testers never thought to look for.
It behaves more like an executor that follows instructions than a tester capable of exploring software behavior.
But as AI agents emerged that could understand requirements, observe screens, analyze page states, and adapt their actions, I began to change my mind.
When training and refining the agents' testing behavior, I emphasized one principle:
Don't just follow the requirements document. And don't treat the test cases as the limits of your exploration.
Requirements describe what a product is expected to do, but they cannot cover every possible user behavior or system state.
So, after completing the tests defined by the requirements, AI should also explore additional scenarios, including:
- Invalid and unexpected inputs: Empty values, excessively long text, illegal characters, unusual formats, and unexpected combinations of data.
- Boundary conditions: Maximum and minimum values, threshold conditions, duplicate submissions, and unusual sequences of operations.
- Abnormal workflows: Interrupted operations, repeated clicks, page refreshes, network failures, expired sessions, and inconsistent states between consecutive steps.
- Security-related scenarios: Unauthorized access, permission boundaries, sensitive information exposure, and attempts to perform operations that should not be allowed.
- Unexpected interactions: Skipping normal steps, performing rapid consecutive actions, repeatedly switching states, or accessing the same business function through different entry points.
Of course, this does not mean AI should be allowed to operate freely against production systems. The scope of exploration, test environments, account permissions, and operational risks must all be controlled. Security testing must also remain within clearly authorized boundaries.
I don't want AI to simply prove that the requirements have been implemented. I want it to look for problems that might emerge when the system is used in ways that were never anticipated.
This is where my understanding of automated testing has changed.
In the past, automation mainly repeated checks for known issues. Now, AI has the potential to help discover problems we didn't know to look for.
It cannot guarantee that every defect will be found, and it cannot replace human judgment. But automated testing no longer has to be limited to test cases written in advance.
3. 🌙 AI Works at Night. Humans Make the Decisions During the Day.
These days, I usually schedule automated testing tasks before finishing work.
The agents then continue running overnight. When I arrive at work the next morning, I can usually find a batch of defect reports already submitted by AI.
My job is to review those issues one by one, determine whether they are genuine and consistent with the current requirements, confirm them, and assign them to the appropriate developers.
This changes how testers spend their time.
Previously, testers had to spend a considerable amount of time operating systems, executing test cases, observing results, and preparing reports. Now, AI can take over a significant portion of that repetitive work, allowing people to focus more on evaluating issues, analyzing risks, and improving product quality.
However, this process also revealed another problem:
Finding a defect doesn't mean communicating it effectively.
Initially, AI-generated defect reports were often just dense blocks of text. They contained plenty of information, but they weren't particularly easy for developers to read.
If a report cannot help the person receiving it quickly understand the problem, then even a large amount of information does little to improve collaboration.
So I went through several rounds of adjustments to the reporting format. We needed more than a record of everything that happened during execution. The report had to clearly explain what happened, where the problem occurred, how the actual behavior differed from the expected behavior, and how the issue could be reproduced and verified.
After several iterations, the reports became much more readable and useful.
This made me realize that AI output quality should not be measured only by how much work it completes. We also need to consider whether the next person in the workflow can actually use the results.
After each testing cycle, I also ask AI to summarize the problems and lessons from that run. Experiences that have been reviewed, validated, and shown to be reusable are then distilled into general-purpose Skills.
That way, the next testing cycle doesn't have to start from scratch. It can build on what the previous cycle has learned.
In practice, this creates a continuous improvement loop:
Execute → Discover issues → Review → Capture lessons → Turn reusable knowledge into Skills → Start the next cycle.
4. 📊 The Numbers Show the Value, but Also Reveal Problems
In one of our knowledge-base projects, we've accumulated some concrete results:
| Metric | Result |
|---|---|
| Total defects identified | 47 |
| Defects identified by AI | 36 |
| Defects identified manually | 11 |
| Defects ultimately confirmed as valid | 30 |
| Valid defects contributed by AI | 21 |
In other words, AI identified 21 of the 30 defects ultimately confirmed as valid, accounting for 70%.
These figures show that, within the scope of this project and its test coverage, AI is no longer just a tool for executing test cases. It is making a real contribution to defect discovery.
We've also experimented with having agents navigate pages, capture the API calls involved in business workflows, and run scheduled, full-scale API validation every day. Our current test environment covers nearly 300 APIs associated with business workflows.
These experiments show that AI can take on continuous testing tasks at both the UI and API levels.
But the data also reveals some problems.
For example, we currently have only one test environment. When multiple requirements are being developed in parallel, code changes can conflict, and deployments can interrupt other testing tasks. We still need to evaluate adding a second test environment and a dedicated performance testing environment.
Browser compatibility also needs further work, including test coverage for different Windows versions and the 360 browser.
These are infrastructure problems. They can be addressed gradually through additional resources and technical improvements.
What really made me think, however, was another category of problems.
5. 🧩 Automation Exposes Organizational Problems That Traditional Testing Often Hides
At one point, we noticed that some defects reported by AI were being classified as invalid.
Our first suspicion was naturally that AI might have misunderstood the requirements or that there was a problem with the testing logic.
But after looking more closely, we found that AI wasn't entirely to blame.
Some requirements had already been adjusted through discussions between product and development, but the official PRD had not been updated. Product and development might have reached a new understanding, or even made changes to the implementation privately, while QA and AI were still working from the original requirements document.
As a result, AI identified a discrepancy based on the official requirements, but the reported issue was considered invalid.
From a testing perspective, AI hadn't necessarily done anything wrong. From the development team's perspective, based on their current understanding of the implementation, the issue might not have been something they believed needed fixing.
The real problem was that the team was working with multiple versions of the same requirement context.
This led me to an important observation:
Automation can expose organizational problems that traditional manual testing has quietly absorbed and hidden for years.
In the past, testers often compensated for inconsistencies in requirements documents through verbal communication, ad hoc clarification, and their own experience. This work is important, but it can also make an organization believe that its processes are working perfectly well.
AI doesn't automatically know about a private conversation, nor can it understand the latest version of someone's thinking. It executes tasks based on the context it receives.
When the official documentation no longer matches the team's actual understanding, automation exposes that gap.
So the question isn't simply whether AI is smart enough.
We also need to ask: Does the team have a shared version of the requirements that everyone agrees on, that is kept up to date, and that AI can correctly interpret?
6. 🏢 Why Did a Technical Presentation Turn into a Discussion About Organizational Structure?
On November 5, 2026, we held an internal session to share our experience with local AI agents and automated testing. One of the company's founders also joined the discussion.
The original focus was AI testing capabilities, but the conversation gradually expanded to engineering workflows and organizational structure.
I increasingly agree with one observation:
AI is reducing the cost of producing code, but the cost of coordinating people isn't decreasing at the same pace.
In the past, a requirement might be analyzed and documented by a product manager, handed over to frontend and backend developers, and eventually passed to QA for verification. Every handoff meant that context had to be transferred again.
If any part of that chain involved a misunderstanding, rework could follow.
Now, AI can help product managers make certain code changes. It can help testers understand and modify code. It can also allow developers to participate more directly in requirement analysis and product design.
If a small requirement only needs a few code changes, but still has to pass through multiple roles, processes, and rounds of communication, the coordination cost may far exceed the actual coding cost.
This doesn't mean professional specialization has lost its value. Rather, AI is gradually lowering some of the capability barriers between traditional roles.
During our discussion, we proposed a direction: moving away from traditional role-based division of labor toward a more flexible model in which people can work across traditional role boundaries with the help of AI.
When we talk about a full-stack approach across the team, we don't mean that everyone must become a traditional full-stack engineer, or that people should abandon their professional expertise.
What we really mean is this:
People in product, development, and QA should all be able to use AI to cross their previous technical boundaries and help solve the problems they understand.
For example:
- Product managers can use AI to make code changes and verify smaller requirements themselves.
- Developers can participate in requirement analysis earlier and take overall ownership of larger initiatives.
- Testers don't have to stop at submitting bug reports when they discover defects. They can use AI to investigate the issue and even modify code, with developers providing the necessary review and confirmation.
For small requirements, we want to reduce unnecessary handoffs. For larger initiatives, we want to assign a clear owner to coordinate the entire process.
This isn't about eliminating professional roles. It's about reducing the information loss caused by rigid boundaries between them.
7. 🔗 Collaboration Should Start with Requirements, Not After Testing Finds a Problem
To make this way of working possible, we first need to address the problem of shared requirement context.
We discussed several concrete directions.
First, product, development, and QA should participate in requirement alignment from the beginning, rather than waiting until the PRD is finished and development is complete before testers start trying to understand the requirements.
Second, we want to establish a standardized PRD Skill and structured templates so that requirement documents follow a more consistent format.
More importantly, whenever requirements change, the official documentation must be updated promptly. Changes should not exist only in private conversations between a few people.
We also discussed a more ambitious idea: using recordings from smart badges to identify requirement changes during discussions, flag content that might affect the requirements, and assist in updating the PRD. The proposed changes would then be submitted to the relevant people for review and approval.
This is still an idea to be explored, not a feature that has already been deployed.
But the underlying principle is clear:
We should make sure the team's actual shared understanding is captured promptly in a form that everyone can use.
Beyond PRDs, we also need shared development and testing Skills, coding standards, and Specs.
When product, development, and QA work from a reasonably consistent context, AI can become a tool that connects different parts of the workflow instead of creating more isolated information silos between roles.
8. 🚀 From Individual AI Tools to an AI-Driven Engineering System
The discussion eventually moved beyond any particular agent or automated testing tool.
We started looking at the engineering system as a whole.
For example:
- Building shared Skill libraries for development and testing.
- Standardizing coding conventions and Spec management.
- Creating a complete workflow: retrieving defects from Yunxiao, using AI to fix them, running automated regression tests, and retaining human review.
- Exploring automated effort estimation and requirement classification.
- Continuing to improve test environments and browser compatibility.
- Ensuring that requirement changes are communicated promptly to reduce unnecessary discussions and rework.
These initiatives won't be completed automatically after a single meeting. The organizational transformation plan has already been launched, and progress will be reviewed and coordinated through weekly updates.
Of course, having a clear direction and an initiated plan doesn't mean the road ahead will be smooth.
Once implementation begins, we'll inevitably encounter practical obstacles: the inertia of existing processes, established collaboration habits between different roles, limitations in technical infrastructure, and the challenges of fitting new ways of working into existing business rhythms.
These problems can't be solved through one meeting or a few new rules. We'll need to identify issues as they arise, adjust our approach, and gradually develop a working model that actually fits the team.
For me, this is what makes the discussion meaningful. It wasn't just about proposing an idea for organizational change. The plan has already moved into the implementation stage.
An AI-driven engineering system can't be built simply by purchasing a few tools, connecting some APIs, or creating a handful of agents.
It requires technical capabilities, process standards, accumulated knowledge, and organizational collaboration to evolve together.
If information within the team remains inconsistent, even the most capable AI will simply execute the wrong context more efficiently.
9. 💡 In the AI Era, Coding Ability May No Longer Be the Rarest Resource
Looking back, I initially just wanted AI to help me automate more testing work.
But as AI began to understand requirements, interact with interfaces, actively explore abnormal scenarios, submit defects, assist with fixes, and perform regression testing, I realized that I needed to rethink more than just testing. I needed to rethink the entire software development process.
In the past, many efficiency problems were attributed to a lack of capability in a particular role, or to a specific stage of the process not being fast enough.
Now, AI is making more and more tasks easier.
But as writing code becomes easier, what may actually slow down development is the failure to synchronize requirements, unclear ownership, and too many handoffs between people.
I don't believe AI will turn everyone into a programmer, nor do I think software teams will no longer need professional roles such as product, development, and QA.
I believe AI will gradually remove unnecessary barriers between these roles.
People who understand a problem should be able to use AI to move toward a solution more directly, rather than having to wait for information to pass through multiple roles.
For testers, this means going beyond executing test cases to participating in risk exploration, problem analysis, and quality improvement.
For developers, it means taking responsibility not just for receiving requirements and writing code, but for solving a more complete problem.
For product managers, it means going beyond describing requirements and using AI to validate and implement their ideas more directly.
Ultimately, a team's competitiveness may no longer depend on how many people complete how many separate tasks. It may depend on whether the team can move accurate information quickly and enable everyone to use AI to solve more complete problems.
An AI testing presentation eventually became a discussion about how the entire engineering organization should change.
Perhaps this is one of the most important changes to pay attention to in the AI era:
AI is making coding easier, but what will truly make software development faster may be our willingness to redesign the way people work together.
Top comments (65)
xulingfeng, lovely to see the Stratagems preoccupation turn up in non-fiction. #32 had an AI confessing it kept turning an incomplete observation into a complete state. Here it’s an agent working faithfully from an incomplete PRD. 😄
One thing in your numbers is worth more than the headline. 36 defects from AI, with 21 valid, and 11 found manually, with 9 valid by subtraction, so roughly 42% invalid for the agents against 18% for the people. Small sample, but part of that gap is the undocumented context your testers were carrying in their heads.
So the invalid rate measures the organisation, not the AI, and it’s the first number you’ve had that counts decisions made without updating the document. The trap is that it vanishes once you fix the PRD process. I’d tag each invalid defect by reason, and the “spec was stale” bucket becomes a drift counter that keeps working afterwards.
Thank you for writing this one as carefully as you write the fiction. 🙏
Wow, this is such a sharp breakdown. You nailed the parallel between Stratagems #32 and this real-world case — both are about incomplete information being treated as full truth.
You’re absolutely right about the false positive rate. It’s not a score for the AI itself; it’s a mirror reflecting our undocumented tribal knowledge. Just to add context: these numbers are only from one single project. We’ve seen very different figures on another project, so we still need case data from more projects to draw solid conclusions.
Tagging invalid defects by root cause and tracking spec drift long-term is a great actionable suggestion. That’s exactly what we plan to build next.
Also, I just woke up from a nap. Those red notification numbers startled me, haha — looks like you’re quite the like spammer!
Thanks so much for reading closely and picking apart the numbers!🤣🤣🤣
Caught red-handed. I do leave a trail of hearty reactions on anything I like, and I’m not sorry about that part. Genuinely sorry about the nap, though. 🤣
But hold on. You’re the QA engineer here. You ran a system with no rate limiting on inbound notifications, in a known hostile environment, while unconscious. I switch everything off before a power nap. Why didn’t you?
Severity: Critical. Repro rate: 100%, every time I open your comments. Recommendation: block the release of my reaction finger immediately.
On the serious bit, about your comment, I’d push it one step further. If one project sits around 42% and another looks nothing like it, then the spread between projects is the measurement. A single project gives you a number. Several give you a map of which teams actually write things down.
Now go back to sleep. I’ll try to be quiet. 🙏
Haha, severity Critical accepted! This notification system definitely lacks proper rate limiting and snooze mode for napping hours. I’ll add this bug to my personal backlog 😂
You’re totally right — the variance across projects is the far more interesting metric than any single number. That spread really reveals the gap between teams on documentation habits. Yeah, agent-based automated testing still needs plenty of real-world project data. I plan to run more rounds of practical tests across different projects and put together a dedicated analysis post once I have enough data.
Thanks for being considerate, but I’m fully awake now hahahaha🤣
🤣🤣
哈哈哈哈🤣强迫症看了受不了啊
Whoa!!!! 36 awesome notifications. Well, I spam for a good reason😀
🤣
@xulingfeng is obsessed with marking notifications as read. 🤣
Wait, that count of 36 feels like a little nudge for me to wrap up the Thirty-Six Stratagems series sooner! Hahaha🤣🤣
Yaaaay! But wait. There is no rush here, take your time.
Fair enough! I won’t force it. The 36 omen will just wait patiently 😄
But, i have zero patience.😕
Uh-oh, impatient reader detected! I’ll speed up a little 🤭
Actually, I’ve had a question I’ve been wanting to ask: what do you all think the ending of the Thirty‑Six Stratagems series will be? Only four chapters left! And I know someone here has zero patience for it 😂
Ahhhhhh! Divya, I don't think you love the series as much as I do. My opinion though😁. Let's let the creator let it flow naturally. Oh geez! What an awesome use of the word "let"😀😀😀
Top Gun was 1986, Maverick was 2022. Thirty-six years. Given your chapter number, that’s less a coincidence than a warning, so don’t let it wait patiently for longer than necessary. 😄
On your question: you planted the Zurich membership page in #32, twelve founding industry seats, with the twelfth marked to be filled, right after #31 had Lena taking a seat. That seat gets filled.
And the 36th stratagem is the retreat one, so the series can’t end in a win by its own structure. Someone walks away, and the question is who chooses it.
My bet is the last chapter comes from the ledger, and the AI’s final entry is that it can’t verify who sat down. It spent #32 learning that it keeps turning incomplete observations into complete states. The fitting ending is the one where it refuses to. 🤣
Oh man, the Top Gun 36-year warning hits different! And your ending theory is chef’s kiss. It aligns perfectly with everything the AI learned in #32. I’ll see how close your guess lands 😄👍🤭
A little teaser — well, it’s not really a spoiler. The final six stratagems fall into the retreat and defeat section of the Thirty‑Six Stratagems. Losing isn’t the point; what matters is how to lose gracefully, hahaha. Let’s wait and see!🤭
I honestly can’t wait bro. Your series made me want to read the 36 Stratagems of Power: Ancient Secret to win at Modern life. That’s my next read this Christmas. 🤤
In this age flooded with short dramas and instant messages, anyone who can slow down and settle in to read a book carefully is already ahead of so many people, haha. I hope I can wrap up the Thirty‑Six Stratagems before Christmas, though I can’t make any promises, hahaha!🤣
I’ve been following this series for so long!🙂
No promises needed, my friend. 😄 I’ll have the book over Christmas either way, so if your final chapter lands in the same week, that’s a better present than anything under the tree.
Just don’t apply the thirty-sixth stratagem to your own deadline. 🤣
That’s a good reminder! The retreat stratagem is always an option, lol.🤣
What a great article! I really need to think about implementing something similar in our project, because we're facing a very similar problem. Knowledge gets scattered, people leave, nobody updates the documentation, and important information just gets lost along the way.
Such interesting times ahead! Looking forward to more articles like this!
Thank you! I think this is exactly where AI could make a real difference.
But there's one thing I've learned from years in software testing: simply putting all the information into a knowledge base doesn't solve the problem. The real challenge is keeping that knowledge accurate and aligned with how the product actually works.
That's one of the things we're trying to explore: using AI not just to retrieve information, but also to help identify gaps, outdated assumptions, and inconsistencies between documentation and reality.
We actually have another interesting idea here. We're working on a hardware project — a voice recorder card. Imagine people carrying this little device around, recording conversations and important information throughout the day. AI could then automatically organize the recordings and sync the latest information into the knowledge base.
Sounds great in theory, right? I'm just a little worried the device might start making things up. Haha! 🤣
We're still figuring things out on our side, so I'll share more as the experiment evolves!
Of course, my next article might be Stratagems #33 instead. You never know! Haha! 😆
मैं आपकी इस बात से 100% सहमत हूँ। आपने वाकई बहुत अच्छी बात कही है!
I don't believe AI will turn everyone into a programmer, nor do I think software teams will no longer need professional roles such as product, development, and QA.
I love this perspective! AI is a powerful helper, yet specialized roles like QA, product and development carry critical context and judgement that no automated tool can fully take over.🙌
But that was your point, right? I agree with what you said.
👍🤣👍Thanks for the approval from this top student, haha 😆
Love this line. Using AI to extend our capabilities rather than replace humans entirely.
Thanks! AI is our force multiplier, not a replacement. It runs overnight, and we steer the decisions.😆
😂
🤣
🤣
Love how an AI testing demo turned into the real conversation: once tests get cheap, the bottleneck moves to everything around them. The single test environment is a perfect example. Before adding a second static one, it might be worth looking at short-lived preview environments per branch or requirement, spun up on push and torn down on merge. Parallel work stops colliding, and your AI test agents each get a clean target instead of fighting over one. Looking forward to the next Stratagems chapter too!
That’s a fantastic suggestion! 👏Short-lived preview environments are exactly the direction we’re exploring next. A single shared test environment becomes a huge pain once many AI test agents run in parallel — they step on each other’s data constantly. Ephemeral environments per branch would give every agent an isolated clean state. And thanks, I’m hard at work on the next Stratagems chapter!😂
The stale-requirements finding is the part that will stay with me. AI flagged 21 of the 30 confirmed defects, but the ones marked invalid were not really AI mistakes — the official requirements had quietly drifted from what product and development had actually agreed in conversation. That reframes the false-positive rate as a measure of organizational drift, not model quality, which is a much more useful way to read those numbers.
I also liked the practical detail about defect reports: finding issues overnight is only half the job if the report is a dense block of text nobody can act on. The shift to readable reports, and to AI running overnight while humans review in the morning, feels like the right division of labor — the machine explores, the people decide.
Such a great way to interpret those false positives! They aren’t just AI errors, they reveal gaps between documented requirements and what the team actually agreed on.
The overnight‑run + human morning review setup has become our sweet spot too. AI digs up potential issues, but clean, actionable reports are what make all that exploration useful. Yep, we’re braced for the long haul ahead.💪
Requirements falling out of sync is usually where the testing problem really starts, so it makes sense the talk moved there. When the AI generates test cases from requirements, who signs off that a case matches what the product owner meant? That handoff looks like the new job.
Yep, we’re braced for the long haul ahead.😂
The stale-PRD bit is the one that hit me — the AI wasn't wrong, the doc was just quietly out of date, and manual testers had been papering over that gap with hallway conversations for years. That's the part I'd steal: the defect report is only as good as the shared understanding behind it. Did the founders' discussion actually change how PRDs get updated now, or is that still the part everyone agrees on in the meeting and forgets on Monday?
It can’t get brushed aside anymore. Our founder officially kicked off this initiative, and we have weekly syncs moving forward. Still, this is definitely a long game.
Automating test generation usually reveals that test authoring was never the scarce resource. When generation costs fall toward zero, the bottleneck shifts to specification stability and defect triage. If acceptance criteria drift across sprints, high-volume synthetic tests mostly produce a flood of false-positive regressions that human developers still have to inspect and dismiss. Teams end up rewriting how they define requirements because running ten thousand automated checks against ambiguous specs costs more in review time than writing fewer tests by hand.
This is exactly one of the key observations in the post.
Test‑authoring is no longer the bottleneck once AI generates tests cheaply. What surfaces instead is messy, drifting acceptance criteria that human testers historically absorbed via ad‑hoc chats and tribal knowledge.
AI agents can’t pick up those off‑the‑record conversations. They strictly work from written specs. When specs fall behind real‑world decisions, we get floods of false‑positive regression findings.
That’s why our discussion shifted beyond testing tools: high‑volume synthetic test output forces the whole team to fix requirement alignment, otherwise the triage overhead eats away all efficiency gains. Really appreciate you calling this out!👍
Interesting that a testing talk became a conversation about how the organization works. Generating more test cases won't resolve product, engineering, and QA having different definitions of “done.” What changed first afterward: ownership, acceptance criteria, or the review process?
That’s a really good question! Honestly none of these things can be fixed in isolation — they’re all tangled up. Different ideas of what “done” means bleed into acceptance criteria, ownership, and how reviews work. Just churning out more test cases will never fix the root issue, exactly like you said. We’re sorting everything out little by little, and we now have a dedicated weekly sync to keep track of all these efficiency improvements.