I think long-running test suites are the bane of the development flow. They cause friction in what is already a fairly fraught process and often times don't even lead you to the correct results.
How long does it take your test suite to run? How many tests do you have?
Do you run those tests in the cloud? How often do you run tests locally?
Let me know. How long does it take for your test suite to run? Does it take too long?
Top comments (35)
Around 60 minutes for the backend part of Localazy. However, it includes also integration tests that run tens of thousands of simulated requests to verify everything from the basic behavior to concurrent request measuring, etc. It tests around 700 scenarios. Many of them are complex situations involving, e.g., uploading strings for translation, translating them, reviewing, exporting, verifying output, notifications, sent emails, status changes, etc.
These tests are in place to verify not that "units" are working well but that everything together works as expected when all the architectural pieces (DB, cache, message broker, storage, etc.) are involved.
Also, for each of the integration tests, the environment is "restarted" which means to recreate the database structure, etc. It adds a lot of overhead.
However, this is not an issue as the complete tests suite is not used during the development. It's mandatory to pass before releasing on internal testing servers and to production.
How often do you run the entire codebase? Are you guys just running the unit tests locally during development?
The entire test suite is used once per several days only.
Unit tests for the backend are run locally and used extensively during development. It's API development, so basically TDD. Sometimes, instead of a unit test, integration test is used to verify the complex workflow during development.
For the frontend (not my part, so I have only limited knowledge), e2e tests are run automatically using Github Actions.
Would you run the entire test suite more often if it was faster? What events trigger a full test suite run? When you’re going to deploy? I’m a big proponent of continuous delivery and run my test suite for a particular code base pretty much every commit.
We plan to move it to the cloud, but since the integration tests need the whole environment to be running, it needs more time than I currently have.
However, it's run before the deployment to testing or production as a mandatory check.
For other things like our CLI tool, we have the whole release chain for the RC version as a Github Action, so except for the backend, we use CD/CI extensively.
Ahh. I see. Get it building and running in the cloud! IMO it's always worth it in the end.
Check this article what we already do with Github Actions: localazy.com/blog/localazy-cli-sof...
And from yesterday, we also build the NodeJS version and publish it to NPM automatically. I'm going to write more about it soon :-).
Yesssss. This is great. I haven't done too much with Kotlin or related tools. But, I have used fastlane quite a bit as a CLI for working with native apps.
fastlane.tools/
It's seriously a god send. Reduces repetitive deploys from 30-45 minute dances of uploads and clicking through UI's to about 1 or 2 commands. It's great. We even used it to start doing real time deployments of one of our native applications. I love stuff like this and how much more productive it makes you!
In fact, the Github Action is not only doing everything for us automatically, but it also documents what is necessary to do.
It's a simple CLI tool but there are so many different operations to do with every release - many more than described in the article such as signing the binaries for Windows and macOS, building RC version, packing 3 different docker containers, uploading to NPM, Docker, APT repo, RPM repo, Brew, our servers, etc.
E.g., for Brew, it needs to be a tar archive accompanied by a YAML descriptor containing SHA checksum. It's not possible for a mortal person to remember all of these steps and to reproduce them for each release.
Btw, if you like how we do it, you can upvote us on ProductHunt. We've launched today :-).
producthunt.com/posts/localazy
Timings are just for running the test suite, excluding the compiling step.
NDNts: TypeScript, 305 test cases, 99 seconds on GitHub Actions
NDNph: C++, 88 test cases, 71 seconds on Travis CI
NDN-DPDK: C & Go, 146 test cases, 257 seconds on a dedicated server (it needs 6 CPU cores; GitHub Actions has only 2 cores so half of the tests are failing there)
ndn-cxx: C++, 1178 test cases, 136 seconds on Jenkins (but compiling can take an hour)
Too long. Was 30 minutes, but we've gotten it down to 10 in the cloud.
There's a lot of optimisation we can do on the tests, and will do in time. Also lots we can do with the pipeline. It's just not even remotely a priority until next year.
For sure! Are you guys optimizing but increasing server costs? A lot of the solutions I have used in the past have all been parallelizing tests across multiple machines.
No. Our unit tests run on GitLab so there's no additional cost to us.
Basicslly instead of running all tests a-z, we split them into batches where each one takes 6-8 minutes.
Whilst it may use a few more pipeline minutes, I'm not as worried about that as I would be having a developer sat round doing nothing for 30 minutes
Oh this is totally my point. I’ve worked at a few places where they refuse to increase the infrastructure for testing “cause it’s too expensive” but there devs would spend 2 or 3 hours a day waiting for tests to run. It’s just a wild culture and poor understanding by management and poor ownership by the dev team.
I work in a brilliant place where they understand the cost of having people sat round waiting for half a day.
We will get to a point where the runners for the pipeline are in our infrastructure, and if they are slow we can throw more resource at them. Or each developer can have their own dedicated runner. It won't cost a lot, and free up a lot of time.
All stuff for next year!
Our frontend tests with Jasmine used to take 40 minutes to run. It was ridiculous, so I looked at it, read some docs, and reduced it to 1 minute a 30 seconds.
The problem was not isolating modules for a test:
github.com/kulshekhar/ts-jest/blob...
How long is too long? I would say that depends on how fast your team is working, and what your budget is. For us, iterating faster on features and bug fixes is the right quality we want to deliver. Waiting for tests to finish can get expensive quick, especially if your team is blocked from merging code.
Hey! That's a serious change in pace!
I try to keep it down to 15 minutes (including end to end) because any longer than that and people start complaining. If you've already optimized your tests consider running multiple builds concurrently.
Yeah, I think 15 minutes is a good timeframe. Do you run multiple builds concurrently now?
I implemented it on some projects in the past, yes. If I didn't the CI run would've probably lasted 50+min.
Yep. I’ve worked at places with where an entire run is between 2 and 3 hours without running test concurrently.
We have 21k tests (half are end to end tests).
It takes ~15 minutes to run on our CI (including build time, tests are ~11 minutes)
Each build uses 128 instances to reach that speed. And we build that for every commit pushed!
That’s a lot of instances. Do you have a limit on the number of instances you scale to? If every dev commits at once do they sit and wait?
I think we can have around 60 to 80 builds in parallel (which is more than enough given the number of devs we have). So if you multiply that by the number of instances... That's a lot in total! And if it gets too slow, we'll increase the number of instances per build again 🚀
having worked in big companies with about 1.5 hours of complete ci pipeline, now I'm more into smaller modules. I absolutely don't want developer's time to be wasted.
I prefer smaller projects, with parallel tasks as much as I can do.
Currently I do code formatting, test, build + docker all in separate pipeline, which reduces time to less than 2 minutes, (even though I have ridiculously high test coverage, I tend to simply split smaller services into their own repo), which means each repo takes less than 5 mins to (merge PR + test + release cycle)
Yep. It’s an often unmentioned advantage of SOA.
~1000 nodejs unit & integration tests, takes less than 30s to run locally. I run the whole lot around once a day (our CI pipeline does it on every commit though)
I was able to achieve this mostly due to pg-mem, a lib I developped that allows to fake a db in-memory. Which has zero overhead, and zero network cost => Friggin fast.
Moreover, i wrote an article that tells you how to leverage HMR in order to have a very short dev cycle (click "run" on a test, you wont even have half a second to wait before the test runs)
Nice! This is cool.
Atomically? 2.5 seconds. In my dev environment 6.5 seconds for 900 tests.
In CI, perhaps a minute or so.
So it's not running the tests (at least not in my case) it's the boiler plate/CI related installs and testing against build flags etc.
Active Logic is a small library but library size is not the real issue here.
I have another library running much fewer tests in a minute, because it triggers web requests via .NET core.
I have yet other libraries running functional tests as time-based simulations. Normally this runs at 10x real time and the results are still reliable enough.
Running tests in CI is not enough, and almost invariably too slow. Detecting failures before committing is really important imho (even if you do PRs).
Not all the good scores are just luck. When it goes over 5 seconds I have a look to see what's happening. Over 30 seconds I sit until things are smooth again.
Impressive! I wish I could even get a RoR environment to spin up in 30 seconds! (I’m sure it does but, man it feels super slow most of the time...)
~10,000 tests (unit + integration tests) take anywhere from 15 to 20 minutes running them on parallel using Azure DevOps.
Yep. As you scale you are you continuing to parallelize the tests? Do you have a target? As in keep the tests under the 20 minute mark? Or?
We have over 80 test suites, we try to keep the test execution for each one under 10 minutes.
We usually run them in parallel, so we get a quick feedback.
We run them in the cloud, on multiple browsers on Windows and MacOS machines.
It's a good example of dogfooding, since we use our very own Endtest to do all of that.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.