xulingfeng, lovely to see the Stratagems preoccupation turn up in non-fiction. #32 had an AI confessing it kept turning an incomplete observation into a complete state. Here it’s an agent working faithfully from an incomplete PRD. 😄
One thing in your numbers is worth more than the headline. 36 defects from AI, with 21 valid, and 11 found manually, with 9 valid by subtraction, so roughly 42% invalid for the agents against 18% for the people. Small sample, but part of that gap is the undocumented context your testers were carrying in their heads.
So the invalid rate measures the organisation, not the AI, and it’s the first number you’ve had that counts decisions made without updating the document. The trap is that it vanishes once you fix the PRD process. I’d tag each invalid defect by reason, and the “spec was stale” bucket becomes a drift counter that keeps working afterwards.
Thank you for writing this one as carefully as you write the fiction. 🙏
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Wow, this is such a sharp breakdown. You nailed the parallel between Stratagems #32 and this real-world case — both are about incomplete information being treated as full truth.
You’re absolutely right about the false positive rate. It’s not a score for the AI itself; it’s a mirror reflecting our undocumented tribal knowledge. Just to add context: these numbers are only from one single project. We’ve seen very different figures on another project, so we still need case data from more projects to draw solid conclusions.
Tagging invalid defects by root cause and tracking spec drift long-term is a great actionable suggestion. That’s exactly what we plan to build next.
Also, I just woke up from a nap. Those red notification numbers startled me, haha — looks like you’re quite the like spammer!
Thanks so much for reading closely and picking apart the numbers!🤣🤣🤣
Caught red-handed. I do leave a trail of hearty reactions on anything I like, and I’m not sorry about that part. Genuinely sorry about the nap, though. 🤣
But hold on. You’re the QA engineer here. You ran a system with no rate limiting on inbound notifications, in a known hostile environment, while unconscious. I switch everything off before a power nap. Why didn’t you?
Severity: Critical. Repro rate: 100%, every time I open your comments. Recommendation: block the release of my reaction finger immediately.
On the serious bit, about your comment, I’d push it one step further. If one project sits around 42% and another looks nothing like it, then the spread between projects is the measurement. A single project gives you a number. Several give you a map of which teams actually write things down.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Haha, severity Critical accepted! This notification system definitely lacks proper rate limiting and snooze mode for napping hours. I’ll add this bug to my personal backlog 😂
You’re totally right — the variance across projects is the far more interesting metric than any single number. That spread really reveals the gap between teams on documentation habits. Yeah, agent-based automated testing still needs plenty of real-world project data. I plan to run more rounds of practical tests across different projects and put together a dedicated analysis post once I have enough data.
Thanks for being considerate, but I’m fully awake now hahahaha🤣
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
What a great article! I really need to think about implementing something similar in our project, because we're facing a very similar problem. Knowledge gets scattered, people leave, nobody updates the documentation, and important information just gets lost along the way.
Such interesting times ahead! Looking forward to more articles like this!
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Thank you! I think this is exactly where AI could make a real difference.
But there's one thing I've learned from years in software testing: simply putting all the information into a knowledge base doesn't solve the problem. The real challenge is keeping that knowledge accurate and aligned with how the product actually works.
That's one of the things we're trying to explore: using AI not just to retrieve information, but also to help identify gaps, outdated assumptions, and inconsistencies between documentation and reality.
We actually have another interesting idea here. We're working on a hardware project — a voice recorder card. Imagine people carrying this little device around, recording conversations and important information throughout the day. AI could then automatically organize the recordings and sync the latest information into the knowledge base.
Sounds great in theory, right? I'm just a little worried the device might start making things up. Haha! 🤣
We're still figuring things out on our side, so I'll share more as the experiment evolves!
Of course, my next article might be Stratagems #33 instead. You never know! Haha! 😆
मैं आपकी इस बात से 100% सहमत हूँ। आपने वाकई बहुत अच्छी बात कही है! I don't believe AI will turn everyone into a programmer, nor do I think software teams will no longer need professional roles such as product, development, and QA.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
I love this perspective! AI is a powerful helper, yet specialized roles like QA, product and development carry critical context and judgement that no automated tool can fully take over.🙌
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Love how an AI testing demo turned into the real conversation: once tests get cheap, the bottleneck moves to everything around them. The single test environment is a perfect example. Before adding a second static one, it might be worth looking at short-lived preview environments per branch or requirement, spun up on push and torn down on merge. Parallel work stops colliding, and your AI test agents each get a clean target instead of fighting over one. Looking forward to the next Stratagems chapter too!
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
That’s a fantastic suggestion! 👏Short-lived preview environments are exactly the direction we’re exploring next. A single shared test environment becomes a huge pain once many AI test agents run in parallel — they step on each other’s data constantly. Ephemeral environments per branch would give every agent an isolated clean state. And thanks, I’m hard at work on the next Stratagems chapter!😂
The stale-requirements finding is the part that will stay with me. AI flagged 21 of the 30 confirmed defects, but the ones marked invalid were not really AI mistakes — the official requirements had quietly drifted from what product and development had actually agreed in conversation. That reframes the false-positive rate as a measure of organizational drift, not model quality, which is a much more useful way to read those numbers.
I also liked the practical detail about defect reports: finding issues overnight is only half the job if the report is a dense block of text nobody can act on. The shift to readable reports, and to AI running overnight while humans review in the morning, feels like the right division of labor — the machine explores, the people decide.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Such a great way to interpret those false positives! They aren’t just AI errors, they reveal gaps between documented requirements and what the team actually agreed on.
The overnight‑run + human morning review setup has become our sweet spot too. AI digs up potential issues, but clean, actionable reports are what make all that exploration useful. Yep, we’re braced for the long haul ahead.💪
Requirements falling out of sync is usually where the testing problem really starts, so it makes sense the talk moved there. When the AI generates test cases from requirements, who signs off that a case matches what the product owner meant? That handoff looks like the new job.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The stale-PRD bit is the one that hit me — the AI wasn't wrong, the doc was just quietly out of date, and manual testers had been papering over that gap with hallway conversations for years. That's the part I'd steal: the defect report is only as good as the shared understanding behind it. Did the founders' discussion actually change how PRDs get updated now, or is that still the part everyone agrees on in the meeting and forgets on Monday?
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
It can’t get brushed aside anymore. Our founder officially kicked off this initiative, and we have weekly syncs moving forward. Still, this is definitely a long game.
Automating test generation usually reveals that test authoring was never the scarce resource. When generation costs fall toward zero, the bottleneck shifts to specification stability and defect triage. If acceptance criteria drift across sprints, high-volume synthetic tests mostly produce a flood of false-positive regressions that human developers still have to inspect and dismiss. Teams end up rewriting how they define requirements because running ten thousand automated checks against ambiguous specs costs more in review time than writing fewer tests by hand.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
This is exactly one of the key observations in the post.
Test‑authoring is no longer the bottleneck once AI generates tests cheaply. What surfaces instead is messy, drifting acceptance criteria that human testers historically absorbed via ad‑hoc chats and tribal knowledge.
AI agents can’t pick up those off‑the‑record conversations. They strictly work from written specs. When specs fall behind real‑world decisions, we get floods of false‑positive regression findings.
That’s why our discussion shifted beyond testing tools: high‑volume synthetic test output forces the whole team to fix requirement alignment, otherwise the triage overhead eats away all efficiency gains. Really appreciate you calling this out!👍
Interesting that a testing talk became a conversation about how the organization works. Generating more test cases won't resolve product, engineering, and QA having different definitions of “done.” What changed first afterward: ownership, acceptance criteria, or the review process?
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
That’s a really good question! Honestly none of these things can be fixed in isolation — they’re all tangled up. Different ideas of what “done” means bleed into acceptance criteria, ownership, and how reviews work. Just churning out more test cases will never fix the root issue, exactly like you said. We’re sorting everything out little by little, and we now have a dedicated weekly sync to keep track of all these efficiency improvements.
Great write-up, and I understand your point about how this process has ultimately lead to a rethinking of your company's development organization, and that's super cool ...
However, when I read what you accomplished in bullet point 1:
"1. 🤖 I Wanted AI to Handle the Entire Testing Process"
... from a technical/professional point of view, that's what I found the most impressive! ... even though I "get it" that you then go on to elaborate on the organizational "big picture" thing :-)
But yes, from a conceptual/philosophical point of view you're right - AI has the potential to "break down siloes" and reduce the friction in organizational processes, and I agree that ultimately that's the "big prize" which will really make companies realize the sought-after productivity improvements with AI ...
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Thanks a lot for this thoughtful comment! Totally get that the idea of end‑to‑end AI testing is where it all started. Honestly, my very first goal was to let AI take care of the whole testing workflow. And yeah, let’s be real — part of me just wanted more breathing room. If AI could carry the load, I’d have time for other stuff, including goofing off a little 😂
But as it turned out, the biggest roadblocks weren’t just technical. They were team silos and messy, out‑of‑date requirements that nobody bothered to write down properly. That’s how we wound up rethinking things across the whole company, instead of just rolling out a new testing tool. Knocking down those silos is really where the real value lives.
Yep I totally get it - knocking down those walls and silos is the holy grail that companies are looking for, and which will ultimately unlock those productivity benefits through AI ! So yeah, that's the "big prize" ...
It's just that, as a 'techie', I couldn't help but being impressed by how you 'automated' that whole test cycle so effectively (bullet point 1) :-)
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Haha, maybe deep down I’ve never fully trusted AI. After all, I’ve written so many posts about AI failures. That’s why I pushed the automated‑testing agents I built really hard right from the start — heavy prompting with all kinds of tough feedback language, plus rigorous review and iteration methods. If a human had to go through this level of grind, they’d probably want to pick a fight with me 😂😂😂
Our company helped a lot too; they gave us plenty of token quota to make this possible. There’s no way I could afford running this out of my own pocket.🤣
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Hi! Wouldn't it be amazing if we have one platform where we can install ai model locally and do everything we want in our daily lives there only. Like there are some apps but we have to connect them integrate many other apps why not just 1.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
That would be great! I'd definitely love to see something like that.
But I think the real challenge isn't just putting everything into one platform. It's making all those capabilities work reliably together.
Different tools have different data structures, permissions, workflows, and failure modes. Even if we put them behind one interface, those integration problems don't magically disappear.
Still, a well-designed local AI platform that handles all of this seamlessly would be incredibly useful!👍
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
the cheap gate on generated tests isn't review, it's a red run: git worktree add /tmp/pre
, point the new test at that tree, and require a non-zero exit before the
test is allowed anywhere near the branch. a generated test that passes on the broken build is
a characterization test -- it will pin the bug and read as coverage for six months until
someone deletes the assertion as redundant.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Great insight! Validating generated tests against a broken build is a smart filter. Tests that blindly accept faulty behaviour only inflate coverage without real value, exactly the trap we watch out for. Worth experimenting with.
This is the part that would actually change how a small team works: the AI wasn't mostly wrong, the requirements had drifted. I've seen the same thing outside software — the agreement changes in a conversation, the written spec doesn't, and then someone gets blamed for following the old version. Making the doc update part of the decision, not a chore for later, feels like the real fix here. The overnight run plus a human morning review is also the right split — let the machine do the searching, and keep the judgment with people.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
So true! Specs falling behind verbal agreements is a cross‑industry pain point. Updating docs alongside decisions, not later, is the real fix. Machine searches, humans judge.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the second review before a defect gets assigned is the piece that makes this usable, because a flood of unverified agent bug reports is how a dev team learns to ignore the tracker. in our agent testing the biggest time sink was false positives where the page was simply mid load, so we added a rule that a defect must reproduce on a fresh session before it files. locator maintenance dropping is real, but triage cost tends to replace it. how are you measuring the false positive rate on filed defects, and does it jump after a UI redesign?
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
We’ve run into lots of pitfalls here too. When I review the defects submitted by the AI, I’m also checking against that same outdated requirement document. That’s exactly what pushed us to keep specs updated as early and often as we can.
For our metrics, we count a defect as false positive once it gets sent back by engineers or marked as “not to fix” by product after review. Moving forward, we plan to have the AI scan for requirement changes before every run, update its test cases accordingly, and restart the testing cycle. At the end of the day, the core thing is keeping requirements complete and synced in a timely manner.
Hey, I'm Hema 👋
Developer, writer, and creator of Dev Opportunity Radar, a weekly series published every Friday on DEV, helping people discover opportunities they might otherwise miss.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
So true! My automated testing agent can handle lots of complex test scenarios, but it can’t fix people forgetting to update the specs after offline discussions. That human workflow piece is impossible to automate easily 😂
We're a place where coders share, stay up-to-date and grow their careers.
xulingfeng, lovely to see the Stratagems preoccupation turn up in non-fiction. #32 had an AI confessing it kept turning an incomplete observation into a complete state. Here it’s an agent working faithfully from an incomplete PRD. 😄
One thing in your numbers is worth more than the headline. 36 defects from AI, with 21 valid, and 11 found manually, with 9 valid by subtraction, so roughly 42% invalid for the agents against 18% for the people. Small sample, but part of that gap is the undocumented context your testers were carrying in their heads.
So the invalid rate measures the organisation, not the AI, and it’s the first number you’ve had that counts decisions made without updating the document. The trap is that it vanishes once you fix the PRD process. I’d tag each invalid defect by reason, and the “spec was stale” bucket becomes a drift counter that keeps working afterwards.
Thank you for writing this one as carefully as you write the fiction. 🙏
Wow, this is such a sharp breakdown. You nailed the parallel between Stratagems #32 and this real-world case — both are about incomplete information being treated as full truth.
You’re absolutely right about the false positive rate. It’s not a score for the AI itself; it’s a mirror reflecting our undocumented tribal knowledge. Just to add context: these numbers are only from one single project. We’ve seen very different figures on another project, so we still need case data from more projects to draw solid conclusions.
Tagging invalid defects by root cause and tracking spec drift long-term is a great actionable suggestion. That’s exactly what we plan to build next.
Also, I just woke up from a nap. Those red notification numbers startled me, haha — looks like you’re quite the like spammer!
Thanks so much for reading closely and picking apart the numbers!🤣🤣🤣
Caught red-handed. I do leave a trail of hearty reactions on anything I like, and I’m not sorry about that part. Genuinely sorry about the nap, though. 🤣
But hold on. You’re the QA engineer here. You ran a system with no rate limiting on inbound notifications, in a known hostile environment, while unconscious. I switch everything off before a power nap. Why didn’t you?
Severity: Critical. Repro rate: 100%, every time I open your comments. Recommendation: block the release of my reaction finger immediately.
On the serious bit, about your comment, I’d push it one step further. If one project sits around 42% and another looks nothing like it, then the spread between projects is the measurement. A single project gives you a number. Several give you a map of which teams actually write things down.
Now go back to sleep. I’ll try to be quiet. 🙏
Haha, severity Critical accepted! This notification system definitely lacks proper rate limiting and snooze mode for napping hours. I’ll add this bug to my personal backlog 😂
You’re totally right — the variance across projects is the far more interesting metric than any single number. That spread really reveals the gap between teams on documentation habits. Yeah, agent-based automated testing still needs plenty of real-world project data. I plan to run more rounds of practical tests across different projects and put together a dedicated analysis post once I have enough data.
Thanks for being considerate, but I’m fully awake now hahahaha🤣
🤣🤣
哈哈哈哈🤣强迫症看了受不了啊
What a great article! I really need to think about implementing something similar in our project, because we're facing a very similar problem. Knowledge gets scattered, people leave, nobody updates the documentation, and important information just gets lost along the way.
Such interesting times ahead! Looking forward to more articles like this!
Thank you! I think this is exactly where AI could make a real difference.
But there's one thing I've learned from years in software testing: simply putting all the information into a knowledge base doesn't solve the problem. The real challenge is keeping that knowledge accurate and aligned with how the product actually works.
That's one of the things we're trying to explore: using AI not just to retrieve information, but also to help identify gaps, outdated assumptions, and inconsistencies between documentation and reality.
We actually have another interesting idea here. We're working on a hardware project — a voice recorder card. Imagine people carrying this little device around, recording conversations and important information throughout the day. AI could then automatically organize the recordings and sync the latest information into the knowledge base.
Sounds great in theory, right? I'm just a little worried the device might start making things up. Haha! 🤣
We're still figuring things out on our side, so I'll share more as the experiment evolves!
Of course, my next article might be Stratagems #33 instead. You never know! Haha! 😆
मैं आपकी इस बात से 100% सहमत हूँ। आपने वाकई बहुत अच्छी बात कही है!
I don't believe AI will turn everyone into a programmer, nor do I think software teams will no longer need professional roles such as product, development, and QA.
I love this perspective! AI is a powerful helper, yet specialized roles like QA, product and development carry critical context and judgement that no automated tool can fully take over.🙌
But that was your point, right? I agree with what you said.
👍🤣👍Thanks for the approval from this top student, haha 😆
Love this line. Using AI to extend our capabilities rather than replace humans entirely.
Thanks! AI is our force multiplier, not a replacement. It runs overnight, and we steer the decisions.😆
😂
🤣
🤣
Love how an AI testing demo turned into the real conversation: once tests get cheap, the bottleneck moves to everything around them. The single test environment is a perfect example. Before adding a second static one, it might be worth looking at short-lived preview environments per branch or requirement, spun up on push and torn down on merge. Parallel work stops colliding, and your AI test agents each get a clean target instead of fighting over one. Looking forward to the next Stratagems chapter too!
That’s a fantastic suggestion! 👏Short-lived preview environments are exactly the direction we’re exploring next. A single shared test environment becomes a huge pain once many AI test agents run in parallel — they step on each other’s data constantly. Ephemeral environments per branch would give every agent an isolated clean state. And thanks, I’m hard at work on the next Stratagems chapter!😂
The stale-requirements finding is the part that will stay with me. AI flagged 21 of the 30 confirmed defects, but the ones marked invalid were not really AI mistakes — the official requirements had quietly drifted from what product and development had actually agreed in conversation. That reframes the false-positive rate as a measure of organizational drift, not model quality, which is a much more useful way to read those numbers.
I also liked the practical detail about defect reports: finding issues overnight is only half the job if the report is a dense block of text nobody can act on. The shift to readable reports, and to AI running overnight while humans review in the morning, feels like the right division of labor — the machine explores, the people decide.
Such a great way to interpret those false positives! They aren’t just AI errors, they reveal gaps between documented requirements and what the team actually agreed on.
The overnight‑run + human morning review setup has become our sweet spot too. AI digs up potential issues, but clean, actionable reports are what make all that exploration useful. Yep, we’re braced for the long haul ahead.💪
Requirements falling out of sync is usually where the testing problem really starts, so it makes sense the talk moved there. When the AI generates test cases from requirements, who signs off that a case matches what the product owner meant? That handoff looks like the new job.
Yep, we’re braced for the long haul ahead.😂
The stale-PRD bit is the one that hit me — the AI wasn't wrong, the doc was just quietly out of date, and manual testers had been papering over that gap with hallway conversations for years. That's the part I'd steal: the defect report is only as good as the shared understanding behind it. Did the founders' discussion actually change how PRDs get updated now, or is that still the part everyone agrees on in the meeting and forgets on Monday?
It can’t get brushed aside anymore. Our founder officially kicked off this initiative, and we have weekly syncs moving forward. Still, this is definitely a long game.
Automating test generation usually reveals that test authoring was never the scarce resource. When generation costs fall toward zero, the bottleneck shifts to specification stability and defect triage. If acceptance criteria drift across sprints, high-volume synthetic tests mostly produce a flood of false-positive regressions that human developers still have to inspect and dismiss. Teams end up rewriting how they define requirements because running ten thousand automated checks against ambiguous specs costs more in review time than writing fewer tests by hand.
This is exactly one of the key observations in the post.
Test‑authoring is no longer the bottleneck once AI generates tests cheaply. What surfaces instead is messy, drifting acceptance criteria that human testers historically absorbed via ad‑hoc chats and tribal knowledge.
AI agents can’t pick up those off‑the‑record conversations. They strictly work from written specs. When specs fall behind real‑world decisions, we get floods of false‑positive regression findings.
That’s why our discussion shifted beyond testing tools: high‑volume synthetic test output forces the whole team to fix requirement alignment, otherwise the triage overhead eats away all efficiency gains. Really appreciate you calling this out!👍
Interesting that a testing talk became a conversation about how the organization works. Generating more test cases won't resolve product, engineering, and QA having different definitions of “done.” What changed first afterward: ownership, acceptance criteria, or the review process?
That’s a really good question! Honestly none of these things can be fixed in isolation — they’re all tangled up. Different ideas of what “done” means bleed into acceptance criteria, ownership, and how reviews work. Just churning out more test cases will never fix the root issue, exactly like you said. We’re sorting everything out little by little, and we now have a dedicated weekly sync to keep track of all these efficiency improvements.
Great write-up, and I understand your point about how this process has ultimately lead to a rethinking of your company's development organization, and that's super cool ...
However, when I read what you accomplished in bullet point 1:
"1. 🤖 I Wanted AI to Handle the Entire Testing Process"
... from a technical/professional point of view, that's what I found the most impressive! ... even though I "get it" that you then go on to elaborate on the organizational "big picture" thing :-)
But yes, from a conceptual/philosophical point of view you're right - AI has the potential to "break down siloes" and reduce the friction in organizational processes, and I agree that ultimately that's the "big prize" which will really make companies realize the sought-after productivity improvements with AI ...
Thanks a lot for this thoughtful comment! Totally get that the idea of end‑to‑end AI testing is where it all started. Honestly, my very first goal was to let AI take care of the whole testing workflow. And yeah, let’s be real — part of me just wanted more breathing room. If AI could carry the load, I’d have time for other stuff, including goofing off a little 😂
But as it turned out, the biggest roadblocks weren’t just technical. They were team silos and messy, out‑of‑date requirements that nobody bothered to write down properly. That’s how we wound up rethinking things across the whole company, instead of just rolling out a new testing tool. Knocking down those silos is really where the real value lives.
Yep I totally get it - knocking down those walls and silos is the holy grail that companies are looking for, and which will ultimately unlock those productivity benefits through AI ! So yeah, that's the "big prize" ...
It's just that, as a 'techie', I couldn't help but being impressed by how you 'automated' that whole test cycle so effectively (bullet point 1) :-)
Haha, maybe deep down I’ve never fully trusted AI. After all, I’ve written so many posts about AI failures. That’s why I pushed the automated‑testing agents I built really hard right from the start — heavy prompting with all kinds of tough feedback language, plus rigorous review and iteration methods. If a human had to go through this level of grind, they’d probably want to pick a fight with me 😂😂😂
Our company helped a lot too; they gave us plenty of token quota to make this possible. There’s no way I could afford running this out of my own pocket.🤣
🥰
😁
🗽
🤣👍
Thanks🗽🗽🗽🗽
Hi! Wouldn't it be amazing if we have one platform where we can install ai model locally and do everything we want in our daily lives there only. Like there are some apps but we have to connect them integrate many other apps why not just 1.
That would be great! I'd definitely love to see something like that.
But I think the real challenge isn't just putting everything into one platform. It's making all those capabilities work reliably together.
Different tools have different data structures, permissions, workflows, and failure modes. Even if we put them behind one interface, those integration problems don't magically disappear.
Still, a well-designed local AI platform that handles all of this seamlessly would be incredibly useful!👍
the cheap gate on generated tests isn't review, it's a red run: git worktree add /tmp/pre
, point the new test at that tree, and require a non-zero exit before the
test is allowed anywhere near the branch. a generated test that passes on the broken build is
a characterization test -- it will pin the bug and read as coverage for six months until
someone deletes the assertion as redundant.
Great insight! Validating generated tests against a broken build is a smart filter. Tests that blindly accept faulty behaviour only inflate coverage without real value, exactly the trap we watch out for. Worth experimenting with.
This is the part that would actually change how a small team works: the AI wasn't mostly wrong, the requirements had drifted. I've seen the same thing outside software — the agreement changes in a conversation, the written spec doesn't, and then someone gets blamed for following the old version. Making the doc update part of the decision, not a chore for later, feels like the real fix here. The overnight run plus a human morning review is also the right split — let the machine do the searching, and keep the judgment with people.
So true! Specs falling behind verbal agreements is a cross‑industry pain point. Updating docs alongside decisions, not later, is the real fix. Machine searches, humans judge.
the second review before a defect gets assigned is the piece that makes this usable, because a flood of unverified agent bug reports is how a dev team learns to ignore the tracker. in our agent testing the biggest time sink was false positives where the page was simply mid load, so we added a rule that a defect must reproduce on a fresh session before it files. locator maintenance dropping is real, but triage cost tends to replace it. how are you measuring the false positive rate on filed defects, and does it jump after a UI redesign?
We’ve run into lots of pitfalls here too. When I review the defects submitted by the AI, I’m also checking against that same outdated requirement document. That’s exactly what pushed us to keep specs updated as early and often as we can.
For our metrics, we count a defect as false positive once it gets sent back by engineers or marked as “not to fix” by product after review. Moving forward, we plan to have the AI scan for requirement changes before every run, update its test cases accordingly, and restart the testing cycle. At the end of the day, the core thing is keeping requirements complete and synced in a timely manner.
Honestly, getting everyone on the same page with specs is way harder than building the actual tools. That workflow side of things is the real test 😄
So true! My automated testing agent can handle lots of complex test scenarios, but it can’t fix people forgetting to update the specs after offline discussions. That human workflow piece is impossible to automate easily 😂