DEV Community

Discussion on: What is smoke, sanity, regression testing? How to CI/CD with them.

Collapse
 
jessekphillips profile image
Jesse Phillips

I am vehemently disagree that the tests are non-blocking. I see two primary reasons the tests are considered acceptable is

  1. The tests are flaky and no trusted to report on the system behavior.
  2. The the system has changed in the expected manner.

Rarely do I see the a regression introduced to the system and that is acceptable to release.

The first indicates the tests are still under development and should not be introduced into the pipeline.

The second indicates a lack of appropriate collaboration or poorly designed test. Neither should be ignored just to get a release out. Your test suite is run during the night, you don't have time to wait. Your next deployment is on the way and has its own set of validation failures.