I'm sick of unreliable CI/CD pipelines caused by flaky tests

If you have a company of 200 engineers and they each waste 1 hr per week waiting to retry flaky tests, you are probably losing $500,000 per year. I can't even tell you how often it is that deploys fail due to flaky tests at most companies where I have worked.

I've set out on a mission to reduce flaky tests by 80-90%. I'm working on a general purpose flaky test management tool that anybody can use no matter how you run your tests, what programming language you use, or what test framework you are using. If you have flaky tests in your codebase, this post is for you.

When a test goes bad at your company, you usually find yourself in a position where dozens of engineers are all trying to fix the same problem at the same time, but in isolation. Each person think that they somehow broke a test and only later after someone in Slack realizes that they are all fixing the same thing does the team realize how much of a loss you're taking.

I've been working on something I call Flaptastic now for 5 months or so. I invite you to give it a look at to ping me if you'd like to try it.

Flaptastic can instantly detect when a unit test has flaked so that you avoid wasting any time trying to fix code you didn't break. It uses pattern matching to figure out if the test failed because it had merit, or if it the root cause was flaky test problems.

The "killer" feature is a checkbox that will instantly disable any unit test across all feature branches and pull requests. In other words, you don't need to wait for everybody to rebase before a test gets marks as skipped. Flaptastic's unique approach allows for the unit test to be disabled instantly for everybody.

I can't find anything quite like it. I heard of an internal tool used at Google which is vaguely similar but is not setup to integrate with existing projects. So far the people who have tried this love it. We aren't charging for it at this time because we want to focus on making this as useful as possible and get the features right.

If you have flaky unit tests in a big company, please ping me and I'll work with you to help come up with a strategy to increase your test health even if you don't end up using this tool.

In the event you want to use Flaptastic, your only requirement is that your code is stored on GitHub and bonus points if your team uses Slack.