Harnessing JavaScript's Power: Conquering the Beasts of Large File Manipulation with TypedArrays and WebWorkers

Safwat Fathi — Fri, 03 May 2024 07:24:11 +0000

Oh, the dreaded experience of trying to read a monstrous, hundreds-of-megabytes-large file in a web application. It’s a path many have walked down, only to witness the horror of page crashes, the haunting UI freezes, and the ever-so-helpful browser Violation warning messages. Have you ever seen a "[Violation] 'something' handler took 43129ms" message? It's the digital equivalent of "I'm too old for this."

Violation message example

The Horrors of Large Files

Let's paint the picture here: you have a large file, and you need to process it right in the user's browser. What could possibly go wrong? For starters, JavaScript was not designed as a heavyweight lifting crane; think of it more like a versatile Swiss Army knife. When you load a massive file using traditional methods, JavaScript tries to gulp it down in one go. The result? The browser chokes, gasps for memory, and might even crash. Cue the panic as users furiously refresh their browsers, to no avail.

The Saviors: TypedArrays and WebWorkers

Before we dive into our superhero tech, let’s talk about what these tools are:

TypedArrays: A Brief Explanation

TypedArrays is not your everyday hero; it's JavaScript’s less-talked-about special forces unit. These arrays allow you to handle binary data directly, with precision and finesse, similar to how you might imagine a surgeon operates. As we know, JavaScript traditionally uses UTF-16 encoding for strings, which is about as efficient as using a butter knife to cut a steel beam. TypedArrays, on the other hand, let you manipulate raw binary data efficiently, slicing through data with the precision of a laser.

WebWorkers: A Brief Explanation

Enter WebWorkers: the covert operatives who take heavy tasks off the main thread, working behind the scenes, unbeknownst to the user. WebWorkers allow you to run JavaScript in the background, in a parallel universe where they don’t block the user interface. Imagine sending your file processing task to a quiet, dedicated worker who sits in a soundproof room, so the rest of the application can continue humming along smoothly without any hiccups.

Let's roll up our sleeves and dive into the solution

WebWorker file

postMessage("Hello from web worker");

    onmessage = event => {
        const textDecoder = new TextDecoder("utf-8");

        const data = new Uint8Array(event.data);

        const text = textDecoder.decode(data);

        postMessage(text);
    };

Handling file change

Here we handle file change event and sending file as chunks to web worker to process it and receive its message

const worker = new Worker("./web-worker.js");

worker.onmessage = e => {
  const $p = document.createElement("p");
  $p.textContent = e.data;
  if ($output) $output.append($p);
};

worker.onerror = function(e) {
  console.error('Error from worker: ', e.message);
};

const $fileInput = document.querySelector<HTMLInputElement>("#file-input");

const handleFileInputChange = async (e: Event) => {
    try {
        const file = (e.target as HTMLInputElement).files?.[0];

        const chunkSize = 1024 * 1024 * 15; // 15MB chunk size

        if (!file) return;

        await processFileInChunks(file, chunkSize);
    } catch (error) {
        console.error("Error:", error);
    }
};

$fileInput.addEventListener("change", handleFileInputChange);

And here where the magic happens ✨

const readFileAsync = (reader: FileReader): Promise<ArrayBuffer> => {
    return new Promise((resolve, reject) => {
        reader.onload = event => {
            const target = event.target;

            if (!target) return;
            const buffer = target.result as ArrayBuffer;
            const slice = new Uint8Array(buffer);

            // Process the buffer data here
            worker.postMessage(slice, [buffer]);

            resolve(buffer); // Resolve promise with the buffer
        };

        reader.onerror = error => {
            reject(error); // Reject promise on error
        };
    });
};

async function handleChunk(chunk: Chunk) {
    const reader = new FileReader();

    // Read specific byte range
    reader.readAsArrayBuffer(chunk.blob.slice(0, chunk.end - chunk.start));

    await readFileAsync(reader);
}

async function processFileInChunks(file: File, chunkSize: number) {
    for (let start = 0; start < file.size; start += chunkSize) {
        const end = Math.min(start + chunkSize, file.size);
        const chunk: Chunk = { start, end, blob: file.slice(start, end) };

        await handleChunk(chunk);
    }
}

In this thrilling TypeScript adventure, first we set up a worker. When a file is selected, instead of reading it directly in the main thread, we use FileReader to read the file as an ArrayBuffer. This buffer then gets sent to the WebWorker, away from the fragile main thread, allowing your application to remain responsive and crash-free.

Conclusion: A Brighter Future for File Processing

With the combined powers of TypedArray and WebWorkers, governed by the wise laws of TypeScript, processing large files in the browser no longer needs to be a nightmare. This strategy not only prevents the UI from freezing but also optimizes performance, making your application a robust, modern marvel that can handle large datasets gracefully. So, why not give your users the smooth, responsive web experience they deserve? After all, nobody likes a slow and cranky application, right?

Now, brave reader, go forth and conquer those gargantuan files with the power of JavaScript, TypeScript, and a little help from your friends, TypedArray and WebWorkers!

Of course any suggestions or thoughts about this will be much appreciated.

The complete code

Complete code can be found in this repo

References and Further Reading

Here are some resources that can deepen your understanding of TypedArrays, WebWorkers, and handling large files in JavaScript:

MDN Web Docs on TypedArray

Dive deep into the world of TypedArrays and see all the possibilities for handling binary data in JavaScript.
TypedArray - MDN
MDN Web Docs on Web Workers

Learn more about the power and potential of Web Workers in offloading tasks and keeping your UI smooth.
Web Workers - MDN
Using Web Workers with TypeScript

A comprehensive guide to implementing Web Workers in a TypeScript project.
Using Web Workers - TypeScript
HTML5 Rocks - The Basics of Web Workers

Get a historical perspective and practical examples of using Web Workers to improve performance in web applications.
HTML5 Rocks - Web Workers
Using Promises with FileReader a good article on how to read files in async
Processing huge files using FileReader.readAsArrayBuffer() in web browser a good article about file reader with array buffer

These resources will not only help you understand the technical foundations but also give you practical insights into applying these concepts in real-world applications.

Build your Flexbox utility CSS classes in 4 simple lines

Safwat Fathi — Sun, 15 Oct 2023 04:18:29 +0000

This tutorial assume you have a good understanding of Flexbox. You can read this great article about Flexbox.

Now let's get into business.

What is CSS utility classes?

Utility classes are self-descriptive, single-purpose CSS classes.

.padding-1 {
  padding: 10px;
}

So you can add this utility class to any element, and save time and specific styles to that element.

<div class="padding-1">
    div element with 10px padding
</div>

Of course you have seen this before if you used (TailwindCSS, Bootstrap, Bulma)

But why we bother reinventing the wheel?

There is always use cases when you have to work with HTML and CSS only, no frameworks or even JavaScript allowed.
One of such cases is email templates.

The Problem

Now if we want to implement the concept we discussed earlier to layout our elements with Flexbox we will have to do a long list of CSS classes. Here is why.

.flex {
  display: flex;
}

/* utility classes to align items */
.align-center {
    align-items: center
}

.align-base {
    align-items: baseline;
}

.align-start {
    align-items: flex-start;
}

.align-end {
    align-items: flex-end;
}

/* other utility classes to handle justify property */
/* ... */

/* other utility classes to handle direction */
/* ... */

/* other utility classes to handle gap between elements */
/* ... */

What a MESS right?

Note: before we go on to the soultion check out this article about how to pass values from HTML to CSS.

The Solution

We can make a good use of injecting CSS variables from HTML, here is how.

Our HTML:

<section class="flex" style="--direction: column; --align: center; --justify: center;">
    <div>Element One</div>
    <div>Element Two</div>
</section>

Our CSS:

.flex {
  display: flex;
}

* {
  flex-direction: var(--direction);
  align-items: var(--align);
  justify-content: var(--justify);
}

That is it, by passing down your direction, align and justify values you can have a full control of your layout.

Thank you for reading! I hope this approach could be handy one day to you.

Also don't you think this approach can be very helpful in other use cases? if you do please share your thoughts in the comments, or if you have a better one or enhance my approach please share with us your thoughts in the comments.

DEV Community: Safwat Fathi