Using Multiple WebWorkers With Create React App

Tal Perry — Mon, 23 Nov 2020 18:46:39 +0000

There are a few great articles about using one WebWorker with Create React App. The gist of them is to use worker-loader and something like rescripts-worker-loader to modify the Webpack config. Easy (after trying ten times).

Problem

I tried and succeeded to run multiple WebWorkers with Create React App and that was harder. At first I got this lovely error message

Conflict: Multiple assets emit to the same filename bundle.worker.js

Yuck!

Solution

Here's the solution, assuming you're using rescript, and an explanation follows.

In your rescriptsrc.js add the following:

function makeMultipleWebworkersWork(config){
    // Change the output file format so that each worker gets a unique name
    config.output.filename = 'static/js/[name].bundle.js'
    // Now, we add a rule for processing workers
    const newRules = [{

        test: /\.worker\.(c|m)?[tj]s$/i,
        type: "javascript/auto",
        include:  config.module.rules[1].include,
        use: [
            {
                loader: "worker-loader",
            },
            {
                loader: "babel-loader",
                options: {
                    presets: ["@babel/preset-env"],
                },
            },
        ],
        // Here we append all the old rules
    },...config.module.rules]
    // now update Webpack's config with the new rules
    config.module.rules = newRules
    return config
}
module.exports =[
    makeMultipleWebworkersWork,
/*Your other stuff goes here */
]

Explanation

After a day of debugging, I realized that CRA's configuration of Webpack doesn't take filenames into account for output (at least in dev mode), so all the workers get emitted to to the same bundle.worker.js file, but they overwrite each other not append.

The code above configures worker-loader and makes sure each worker goes to a different file, as desired.

Why

I'm building an open source in browser NLP thing. As you might guess, it trains NLP models in the browser interactively.
Using multiple workers is a great why to crunch a lot of data without blocking the users main thread, e.g. keeping the app responsive and engaging.

Still a bit early to release it, but follow me here and on twitter if you'd like to keep updated.

My in-browser search with IndexDB and WebWorkers

Tal Perry — Fri, 30 Oct 2020 11:44:53 +0000

Hi,
I'm new to Dev, thanks for checking out my first post.
I'm a data scientist doing data products for actual humans. I want my users to get to wow fast, and I want to develop fast.

You know what gets in my way ? Servers. Writing a server, integrating with a server, authenticating a user on a server and persisting data on a server all stand between me and my goal.

Recently I wanted to let my user upload a csv with a text, give them full text search over that, make some annotations on the text and have the result saved for later.

Back in the day I would have set up a python server, a postgres database, connected them and then tried to get my webapp to talk to the python.

That's annoying because it's a lot of setup for me, and some setup and waiting for my users. It's 2020 no-one has any patience. Let's see how we can get the time to wow down to something a millennial or genexer like me can stomach:

So here's what I did.

Moved the heavy computation (building the index and search) in a WebWorker .
Wrapped the worker in a class that gives the main thread a promise when it searches.
Used react-query to delegate away the request/response state management instead of having a mountain of Redux code.
Used the Dexie library to store the users data in Indexedb so thay can come back and pick up with they left off.
And I used ndx which is a really nice in JS full text search implementation.

The end result looks like this, granted the design is rough but a satisfying POC of an inbrowser search engine

This is all going open source soon and I have a lot to share about each of these steps, but I'm not sure if this is interesting and exciting or kind of meh. Let me know what you think and I'll be glad to share more!
Tal

DEV Community: Tal Perry

Using Multiple WebWorkers With Create React App

Problem

Solution

Explanation

Why

My in-browser search with IndexDB and WebWorkers