Using AI for writing unit tests has taken over like a storm and while most people do it blindly they miss out on how dangerous it actually is and here are the reasons why i believe so in my experience of using AI for testing.
Most AI code tests are done in the following manner, you are done writing with your feature and ask AI to look at it, and build relevant tests for it. The issue with this is that the AI itself has no context apart from the code in the repo about that feature which is greatly a human error or a prompting error, because what happens in these scenarios is that the AI looks at the code, looks at the intended work the code snippet or function does and makes tests usually starting with the happy paths (the tests that will succeed) and here lies the issue. What if the feature you have written, the function that is being tested you have done the incorrect logic, lets say for example instead of turning a boolean value to true, you mistakenly turned it to false. The AI would assume that is the intended response as it was never prompted the feature and its usecase, all it had as reference was the code which it assumed was the happy path for the use case.
I have recently seen this pattern to be very evident especially when I am checking PRs of juniors where a small mistake was made in a feature which was not intended and it passed the test cases as well, however had they been written by a user aware of the feature or the user who had explained the AI it may (not always hallucinations are an issue with LLMs) have failed the test case and the error would have been caught there and then.
What are your thoughts about this? have you faced this issue and what exactly did you do to put the correct workflows in place for this matter
Top comments (0)