DEV Community

loading...
Cover image for How long does it take for your test suite to run?

How long does it take for your test suite to run?

Andrew Stuntz
I'm a full stack developer working on building web applications for all sorts of sized companies using a wide variety of web technologies. Mostly working in Ruby and JavaScript.
・1 min read

I think long-running test suites are the bane of the development flow. They cause friction in what is already a fairly fraught process and often times don't even lead you to the correct results.

How long does it take your test suite to run? How many tests do you have?

Do you run those tests in the cloud? How often do you run tests locally?

Let me know. How long does it take for your test suite to run? Does it take too long?

Discussion (35)

Collapse
vaclavhodek profile image
vaclavhodek

Around 60 minutes for the backend part of Localazy. However, it includes also integration tests that run tens of thousands of simulated requests to verify everything from the basic behavior to concurrent request measuring, etc. It tests around 700 scenarios. Many of them are complex situations involving, e.g., uploading strings for translation, translating them, reviewing, exporting, verifying output, notifications, sent emails, status changes, etc.

These tests are in place to verify not that "units" are working well but that everything together works as expected when all the architectural pieces (DB, cache, message broker, storage, etc.) are involved.

Also, for each of the integration tests, the environment is "restarted" which means to recreate the database structure, etc. It adds a lot of overhead.

However, this is not an issue as the complete tests suite is not used during the development. It's mandatory to pass before releasing on internal testing servers and to production.

Collapse
drews256 profile image
Andrew Stuntz Author

How often do you run the entire codebase? Are you guys just running the unit tests locally during development?

Collapse
vaclavhodek profile image
vaclavhodek

The entire test suite is used once per several days only.

Unit tests for the backend are run locally and used extensively during development. It's API development, so basically TDD. Sometimes, instead of a unit test, integration test is used to verify the complex workflow during development.

For the frontend (not my part, so I have only limited knowledge), e2e tests are run automatically using Github Actions.

Thread Thread
drews256 profile image
Andrew Stuntz Author

Would you run the entire test suite more often if it was faster? What events trigger a full test suite run? When you’re going to deploy? I’m a big proponent of continuous delivery and run my test suite for a particular code base pretty much every commit.

Thread Thread
vaclavhodek profile image
vaclavhodek

We plan to move it to the cloud, but since the integration tests need the whole environment to be running, it needs more time than I currently have.

However, it's run before the deployment to testing or production as a mandatory check.

For other things like our CLI tool, we have the whole release chain for the RC version as a Github Action, so except for the backend, we use CD/CI extensively.

Thread Thread
drews256 profile image
Andrew Stuntz Author

Ahh. I see. Get it building and running in the cloud! IMO it's always worth it in the end.

Thread Thread
vaclavhodek profile image
vaclavhodek

Check this article what we already do with Github Actions: localazy.com/blog/localazy-cli-sof...

And from yesterday, we also build the NodeJS version and publish it to NPM automatically. I'm going to write more about it soon :-).

Thread Thread
drews256 profile image
Andrew Stuntz Author

Yesssss. This is great. I haven't done too much with Kotlin or related tools. But, I have used fastlane quite a bit as a CLI for working with native apps.

fastlane.tools/

It's seriously a god send. Reduces repetitive deploys from 30-45 minute dances of uploads and clicking through UI's to about 1 or 2 commands. It's great. We even used it to start doing real time deployments of one of our native applications. I love stuff like this and how much more productive it makes you!

Thread Thread
vaclavhodek profile image
vaclavhodek

In fact, the Github Action is not only doing everything for us automatically, but it also documents what is necessary to do.

It's a simple CLI tool but there are so many different operations to do with every release - many more than described in the article such as signing the binaries for Windows and macOS, building RC version, packing 3 different docker containers, uploading to NPM, Docker, APT repo, RPM repo, Brew, our servers, etc.

E.g., for Brew, it needs to be a tar archive accompanied by a YAML descriptor containing SHA checksum. It's not possible for a mortal person to remember all of these steps and to reproduce them for each release.

Thread Thread
vaclavhodek profile image
vaclavhodek

Btw, if you like how we do it, you can upvote us on ProductHunt. We've launched today :-).

producthunt.com/posts/localazy

Collapse
yoursunny profile image
Junxiao Shi • Edited

Timings are just for running the test suite, excluding the compiling step.

NDNts: TypeScript, 305 test cases, 99 seconds on GitHub Actions
NDNph: C++, 88 test cases, 71 seconds on Travis CI
NDN-DPDK: C & Go, 146 test cases, 257 seconds on a dedicated server (it needs 6 CPU cores; GitHub Actions has only 2 cores so half of the tests are failing there)
ndn-cxx: C++, 1178 test cases, 136 seconds on Jenkins (but compiling can take an hour)

Collapse
_garybell profile image
Gary Bell

Too long. Was 30 minutes, but we've gotten it down to 10 in the cloud.

There's a lot of optimisation we can do on the tests, and will do in time. Also lots we can do with the pipeline. It's just not even remotely a priority until next year.

Collapse
drews256 profile image
Andrew Stuntz Author

For sure! Are you guys optimizing but increasing server costs? A lot of the solutions I have used in the past have all been parallelizing tests across multiple machines.

Collapse
_garybell profile image
Gary Bell

No. Our unit tests run on GitLab so there's no additional cost to us.

Basicslly instead of running all tests a-z, we split them into batches where each one takes 6-8 minutes.

Whilst it may use a few more pipeline minutes, I'm not as worried about that as I would be having a developer sat round doing nothing for 30 minutes

Thread Thread
drews256 profile image
Andrew Stuntz Author

Oh this is totally my point. I’ve worked at a few places where they refuse to increase the infrastructure for testing “cause it’s too expensive” but there devs would spend 2 or 3 hours a day waiting for tests to run. It’s just a wild culture and poor understanding by management and poor ownership by the dev team.

Thread Thread
_garybell profile image
Gary Bell

I work in a brilliant place where they understand the cost of having people sat round waiting for half a day.

We will get to a point where the runners for the pipeline are in our infrastructure, and if they are slow we can throw more resource at them. Or each developer can have their own dedicated runner. It won't cost a lot, and free up a lot of time.

All stuff for next year!

Collapse
qmenoret profile image
Quentin Ménoret

We have 21k tests (half are end to end tests).
It takes ~15 minutes to run on our CI (including build time, tests are ~11 minutes)
Each build uses 128 instances to reach that speed. And we build that for every commit pushed!

Collapse
drews256 profile image
Andrew Stuntz Author

That’s a lot of instances. Do you have a limit on the number of instances you scale to? If every dev commits at once do they sit and wait?

Collapse
qmenoret profile image
Quentin Ménoret

I think we can have around 60 to 80 builds in parallel (which is more than enough given the number of devs we have). So if you multiply that by the number of instances... That's a lot in total! And if it gets too slow, we'll increase the number of instances per build again 🚀

Collapse
aghost7 profile image
Jonathan Boudreau

I try to keep it down to 15 minutes (including end to end) because any longer than that and people start complaining. If you've already optimized your tests consider running multiple builds concurrently.

Collapse
drews256 profile image
Andrew Stuntz Author

Yeah, I think 15 minutes is a good timeframe. Do you run multiple builds concurrently now?

Collapse
aghost7 profile image
Jonathan Boudreau

I implemented it on some projects in the past, yes. If I didn't the CI run would've probably lasted 50+min.

Thread Thread
drews256 profile image
Andrew Stuntz Author

Yep. I’ve worked at places with where an entire run is between 2 and 3 hours without running test concurrently.

Collapse
cyberhck profile image
Nishchal Gautam

having worked in big companies with about 1.5 hours of complete ci pipeline, now I'm more into smaller modules. I absolutely don't want developer's time to be wasted.

I prefer smaller projects, with parallel tasks as much as I can do.

Currently I do code formatting, test, build + docker all in separate pipeline, which reduces time to less than 2 minutes, (even though I have ridiculously high test coverage, I tend to simply split smaller services into their own repo), which means each repo takes less than 5 mins to (merge PR + test + release cycle)

Collapse
drews256 profile image
Andrew Stuntz Author

Yep. It’s an often unmentioned advantage of SOA.

Collapse
fallenstedt profile image
Alex Fallenstedt • Edited

Our frontend tests with Jasmine used to take 40 minutes to run. It was ridiculous, so I looked at it, read some docs, and reduced it to 1 minute a 30 seconds.

The problem was not isolating modules for a test:
github.com/kulshekhar/ts-jest/blob...

How long is too long? I would say that depends on how fast your team is working, and what your budget is. For us, iterating faster on features and bug fixes is the right quality we want to deliver. Waiting for tests to finish can get expensive quick, especially if your team is blocked from merging code.

Collapse
drews256 profile image
Andrew Stuntz Author

Hey! That's a serious change in pace!

Collapse
oguimbal profile image
Olivier Guimbal • Edited

~1000 nodejs unit & integration tests, takes less than 30s to run locally. I run the whole lot around once a day (our CI pipeline does it on every commit though)

I was able to achieve this mostly due to pg-mem, a lib I developped that allows to fake a db in-memory. Which has zero overhead, and zero network cost => Friggin fast.

Moreover, i wrote an article that tells you how to leverage HMR in order to have a very short dev cycle (click "run" on a test, you wont even have half a second to wait before the test runs)

Collapse
drews256 profile image
Andrew Stuntz Author

Nice! This is cool.

Collapse
eelstork profile image
Tea

Atomically? 2.5 seconds. In my dev environment 6.5 seconds for 900 tests.
In CI, perhaps a minute or so.
So it's not running the tests (at least not in my case) it's the boiler plate/CI related installs and testing against build flags etc.
Active Logic is a small library but library size is not the real issue here.
I have another library running much fewer tests in a minute, because it triggers web requests via .NET core.
I have yet other libraries running functional tests as time-based simulations. Normally this runs at 10x real time and the results are still reliable enough.
Running tests in CI is not enough, and almost invariably too slow. Detecting failures before committing is really important imho (even if you do PRs).
Not all the good scores are just luck. When it goes over 5 seconds I have a look to see what's happening. Over 30 seconds I sit until things are smooth again.

Collapse
drews256 profile image
Andrew Stuntz Author

Impressive! I wish I could even get a RoR environment to spin up in 30 seconds! (I’m sure it does but, man it feels super slow most of the time...)

Collapse
liviufromendtest profile image
Liviu Lupei

We have over 80 test suites, we try to keep the test execution for each one under 10 minutes.
We usually run them in parallel, so we get a quick feedback.
We run them in the cloud, on multiple browsers on Windows and MacOS machines.
It's a good example of dogfooding, since we use our very own Endtest to do all of that.

Collapse
andres profile image
Andrés Pérez • Edited

~10,000 tests (unit + integration tests) take anywhere from 15 to 20 minutes running them on parallel using Azure DevOps.

Collapse
drews256 profile image
Andrew Stuntz Author • Edited

Yep. As you scale you are you continuing to parallelize the tests? Do you have a target? As in keep the tests under the 20 minute mark? Or?

Collapse
aniketd4x profile image
aniketd4x

Very useful 15 min!