If you test with random values, wouldn't that make your tests flaky?
They might pass with one value and fail with an other one. And in my experience if the developer sees this kind of random failures she/he would just rerun the test to make CI green. And then when that happens again and again the risk is that code with failed tests will be merged because "we know this tests sometimes fail, so we can merge it"
That's a good question, similar to concerns I've heard before!
If the values that are important for your test are randomised, then of course your test may flip (unless your test dies some kind of "double accounting, calculating a result via a different method).
The point here is that in a test like "when a customer places a delivery order before a certain time, she should get a discount", then the other data points shouldn't matter. If a test flops because of a random data value that shouldn't have mattered, then it helps to find a problen with the production code. I've experienced this exact scenario - of course debugging it requires running the test until failure with lots of logging, but it's saved me in the long run.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
If you test with random values, wouldn't that make your tests flaky?
They might pass with one value and fail with an other one. And in my experience if the developer sees this kind of random failures she/he would just rerun the test to make CI green. And then when that happens again and again the risk is that code with failed tests will be merged because "we know this tests sometimes fail, so we can merge it"
That's a good question, similar to concerns I've heard before!
If the values that are important for your test are randomised, then of course your test may flip (unless your test dies some kind of "double accounting, calculating a result via a different method).
The point here is that in a test like "when a customer places a delivery order before a certain time, she should get a discount", then the other data points shouldn't matter. If a test flops because of a random data value that shouldn't have mattered, then it helps to find a problen with the production code. I've experienced this exact scenario - of course debugging it requires running the test until failure with lots of logging, but it's saved me in the long run.