DEV Community: Ivan Tanev

Quick Tip: Simpler sharding configuration for Playwright

Ivan Tanev — Fri, 23 Feb 2024 21:07:46 +0000

Playwright’s sharding docs present the following GitHub actions configuration:

strategy:
  fail-fast: false
  matrix:
    shardIndex: [1, 2, 3, 4]
    shardTotal: [4]
[...]
steps:
- name: Run Playwright tests
  run: |
    npx playwright test \ 
      --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

While it works, I think the following reads much better:

strategy:
  fail-fast: false
  matrix:
    shard: ["1/4", "2/4", "3/4", "4/4"]
[...]
steps:
- name: Run Playwright tests
  run: npx playwright test --shard=${{ matrix.shard }}

You don't even need the quotes around "1/4", you could write them naked, but I prefer to be explicit in yaml:

    shard: [1/4, 2/4, 3/4, 4/4]

What do you think?

Make Your Jest Tests up to 20% Faster by Changing a Single Setting

Ivan Tanev — Thu, 25 Mar 2021 22:06:22 +0000

Originally posted on ivantanev.com

TL;DR

When you have Jest as your test runner, passing the --maxWorkers=50% option will make the tests faster in most cases. For watch mode, use --maxWorkers=25%, and for CI disable Jest workers with --runInBand. You can experiment with the percentage and fine-tune for your particular setup.

// package.json
{
  "scripts": {
    // standalone Jest
    "test": "jest --maxWorkers=50%",
    "test:watch": "jest --watch --maxWorkers=25%",
    "test:ci": "jest --runInBand",

    // or with Create React App
    "test": "react-scripts test --watchAll=false --maxWorkers=50%",
    "test:watch": "react-scripts test --maxWorkers=25%",
    "test:ci": "react-scripts test --watchAll=false --runInBand"
  }
}

Update 2021-03-29

While a lot of people have reported great results, I have seen some indication that on older Intel CPUs without hyperthreading the above setting results in a performance degradation. You should benchmark and validate for your particular setup.

How Jest selects the number of workers to use

The Jest test runner—that is also supplied by default with Create React App—does not run optimally out of the box.

By default, Jest will run on all available CPU threads, using one thread for the cli process and the rest for test workers. When in watch mode, it will use half the available CPU threads.

This however results in sub-optimal performance on all systems I tested on.

We can adjust --maxWorkers by either providing a number of threads, or a percentage of the available system threads. I prefer using percentage, as it's usually easy to find a value that works across multiple systems with different CPUs.

Benchmarking Jest with --maxWorkers=50%

These are the stats for the testsuite used. It's a React app with mostly unit tests:

Test Suites: 43 passed, 43 total
Tests:       1 skipped, 258 passed, 259 total
Snapshots:   2 passed, 2 total

Here are the results on an Intel i9-9900KS (5GHz / 8 cores 16 threads):
A 21% speedup.

$ hyperfine 'npm test' 'npm test -- --maxWorkers=50%'
Benchmark #1: npm test
  Time (mean ± σ):      4.763 s ±  0.098 s    [User: 49.334 s, System: 5.996 s]
  Range (min … max):    4.651 s …  4.931 s    10 runs

Benchmark #2: npm test -- --maxWorkers=50%
  Time (mean ± σ):      3.925 s ±  0.044 s    [User: 27.776 s, System: 4.028 s]
  Range (min … max):    3.858 s …  3.973 s    10 runs

Summary
  'npm test -- --maxWorkers=50%' ran
    1.21 ± 0.03 times faster than 'npm test'

And here are the results on a 2016 13" MacBook Pro (3.3GHz / 2 cores 4 threads):
A 14% speedup.

$ hyperfine 'npm test' 'npm test -- --maxWorkers=50%'
Benchmark #1: npm test
  Time (mean ± σ):     14.380 s ±  0.230 s    [User: 22.869 s, System: 3.689 s]
  Range (min … max):   14.049 s … 14.807 s    10 runs

Benchmark #2: npm test -- --maxWorkers=50%
  Time (mean ± σ):     12.567 s ±  0.213 s    [User: 19.628 s, System: 3.290 s]
  Range (min … max):   12.258 s … 12.942 s    10 runs

Summary
  'npm test -- --maxWorkers=50%' ran
    1.14 ± 0.03 times faster than 'npm test'

And finally, a 2020 M1 MacBook Air:
A 12% speedup.

$ hyperfine 'npm test' 'npm test -- --maxWorkers=50%'
Benchmark #7: npm run test
  Time (mean ± σ):      5.833 s ±  0.025 s    [User: 30.257 s, System: 6.995 s]
  Range (min … max):    5.813 s …  5.861 s    3 runs

Benchmark #4: npm test -- --maxWorkers=50%
  Time (mean ± σ):      5.216 s ±  0.060 s    [User: 19.301 s, System: 3.523 s]
  Range (min … max):    5.179 s …  5.285 s    3 runs

Summary
  'npm test -- --maxWorkers=50%' ran
    1.12 ± 0.01 times faster than 'npm test'

What about running alongside other programs?

Measuring this is harder, but I have noticed that running with --maxWorkers=25% performs the best for my use cases.
This gives the best performance for test:watch alongside code watch/hot reloading, and for running husky commit hooks in parallel.

What about CI?

In my and other's experience, --runInBand can be the fastest option for CI runs.

What does --runInBand do? From the official docs:

Run all tests serially in the current process, rather than creating a worker pool of child processes that run tests. This can be useful for debugging.

Turns out, it's also useful in resource-constrained environments like CI, where the overhead of worker processes is higher than the speedup of running tests in parallel.

Finding the optimal number of threads for a given testsuite/system

It's easy to write a small script to find the optimal number of threads for your particular usecase:

export MAX_WORKERS=15; hyperfine --parameter-scan num_threads 1 $MAX_WORKERS 'npm run test -- --maxWorkers={num_threads}' -m 3 -w 1

Here are the results on an Intel i9-9900KS (5GHz / 8 cores 16 threads):

Summary
  'npm run test:jest -- --maxWorkers=7' ran
    1.01 ± 0.01 times faster than 'npm run test:jest -- --maxWorkers=8'
    1.02 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=6'
    1.04 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=5'
    1.05 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=9'
    1.08 ± 0.03 times faster than 'npm run test:jest -- --maxWorkers=10'
    1.11 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=11'
    1.11 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=4'
    1.18 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=13'
    1.19 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=14'
    1.21 ± 0.04 times faster than 'npm run test:jest -- --maxWorkers=12'
    1.23 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=15'
    1.25 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=3'
    1.58 ± 0.02 times faster than 'npm run test:jest -- --maxWorkers=2'
    2.55 ± 0.04 times faster than 'npm run test:jest -- --maxWorkers=1'

As you can see, the optimal number of workers in this case is 7, not the 8 that 50% would give us. However the difference between the two is within the margin of error, and 50% is more flexible.

Conclusion

Jest performance out of the box can be easily improved by tweaking maxWorkers. If you decide to test this for yourself, hyperfine makes it very easy.

Hope this was helpful! Feel free to reach out to me on Twitter @VanTanev.

Happy hacking!

Async Functions and AngularJs 1.X Do Not Mix

Ivan Tanev — Sun, 21 Mar 2021 16:50:11 +0000

Originally posted on ivantanev.com

Recently, I was refactoring an AngularJS 1.x project and wrote the following code:

// DO NOT USE THIS CODE, IT BREAKS ANGULARJS!
// Combining $http with async functions does not work in AngularJS 1.X
async function updateItem(item) {
  const { data } = await $http.put(`/items/${item.id}`, item)
  return data
}

While the above might seem innocent enough, it breaks Angular's digest cycle. When the promise fulfills, you will not see the page update.

What you should do instead?

Do not use await/async with $http. Instead, use the old promise style, with .then().

// Use promise.then() instead of async/await
function updateItem(item) {
  return $http.put(`/items/${item.id}`, item).then(({ data }) => data)
}

While the above might seem like completely reasonable and modern JavaScript, attempting to use it will break Angular's digest cycle.

Where does it break?

Let's look at what the browser does when it executes the async/await code

// DO NOT USE THIS CODE, IT BREAKS ANGULARJS!
// Combining $http with async functions does not work in AngularJS 1.X

// This function:
async function updateItem(item) {
  const { data } = await $http.put(`/items/${item.id}`, item)
  return data
}


// Is equivalent to the following code:
function updateItem(item) {
  return Promise.resolve($http.put(`/items/${item.id}`, item)).then((value) => {
    const { data } = value
    return data
  })
}

As you can see, the original $q promise returned from $http.put() is wrapped in a new Promise.resolve(). This means AngularJS can no longer track when the promise settles.

The issue comes up when you try to use the async version of updateItem() in a controller:

function SomeCtrl($scope, initialItem) {
  $scope.item = initialItem

  this.onClick = function() {
    updateItem($scope.item)
      .then(updatedItem => ($scope.item = updatedItem))
  }
}

The async function implementation of updateItem() would break the highlighted line. The changed $scope.item variable will not be reflected in the DOM, or watchers, until a random digest cycle executes.

The reason is that AngularJS cannot know when a plain promise resolves.

AngularJS has special wrappers for browser primitives--$timeout, $interval and its own Promise library $q. AngularJS needs to wrap these asynchronous interfaces in order to track when they complete, and run a $rootScope.$digest() cycle.

When we used an async function, we're in the same predicament as if we'd used setTimeout() directly instead of $timeout()--there is no way for AngularJS to track when the execution of the async function completed.

To make the async function work in our controller, we would need to re-wrap it with $q.resolve():

function SomeCtrl($q, $scope, initialItem) {
  $scope.item = initialItem

  this.onClick = function() {
    $q.resolve(updateItem($scope.item)).then(
      updatedItem => ($scope.item = updatedItem)
    )
  }
}

Or we could use another async function and a $apply() around our controller property assignment:

function SomeCtrl($scope, initialItem) {
  $scope.item = initialItem

  this.onClick = async function() {
    const updatedItem = await updateItem($scope.item)
    $scope.$apply(() => {
      $scope.item = updatedItem
    })
  }
}

We end up having to manually wrap any effects of async function code with $scope.$apply(), or wrap Promises with $q.resolve(). This makes it not worth using async/await in the first place. This is unfortunate when we need to coordinate multiple async tasks, as the async/await interfaces make that much nicer.

In conclusion

Modern async/await functions are great, and its tempting to want to use them when refactoring old code. However, it's not worth the hassle in AngularJS 1.X projects. We'll have to stick to the $q promise interfaces instead.

Say hi on Twitter or check out my other posts on my website.
Happy coding!