A flaky test is one whose attempts disagree: it fails once and passes on another try, with the code and the test unchanged. In Nexus StudioQA the word has one meaning. FLAKY is a test that failed, then passed on a retry, with nothing healed. Something is unstable, in the application or in the test, and nothing was fixed.
A hard failure that happens on every attempt is not flaky, even when the self-healing step tried and failed to repair a selector. An earlier version of the engine filed that case as FLAKY. Teams read FLAKY as "the test is bad" and move on, so a broken application shipped as noise. The engine no longer does that.
Top comments (0)