Processing Images with Web Assembly using wasm-vips

Backstory

For the last few years I have run a site called tiny.photos, I built it because I couldn't find a tool that worked the way I wanted, and it was great! Except it was costing a lot of money. As you might expect, processing images is a bit resource intensive. So I set out on a quest to reduce costs, web assembly had always been in the back of my mind as a cool option but I wasn't sure if it would actually work. I figured I would play around with it and see what I could come up with. After a few long days of banging my head against the wall I got it working! In an effort to help other people along I thought I would do this little write up that will hopefully save some people the headache of figuring out wasm-vips and help others discover the power of in browser image processing.

wasm-vips

If you do some googling on this subject you might run across a library called wasm-vips, which is exactly what it sounds like, a web assembly wrapper for libvips. Libvips is a fantastic C library for processing images so naturally wasm-vips should be a great option for processing images in the browser, right? Yes and no. Yes, wasm-vips is a fantastic option but it comes with the caveat that it is in early development and has very minimal documentation. So having spent several days tinkering and sifting through the wasm-vips repo I'm sharing what I learned that was not available in the docs by creating a simple image resizing/cropping script.

Getting an image from the end user to wasm-vips

Surprisingly, this is not as trivial as you would think given that you can't process an image if you don't have an image.

Let's start with the basics, getting the files from a form.

A basic file input

<input id="fileElem" type="file" multiple accept="image/*">

Retrieve the files from the input

// In my case I have additional logic to handle
// drag and drop functionality but this could be handled with 
// an event listener or any other logic that you need
function handleDrop(e) {
    let dt = e.dataTransfer
    let files = dt.files
    handleFiles(files)
};

function handleFiles(files) {
    ([...files]).forEach(function(file){
        processImages(file);
    });
};

Now we get to the good stuff, the processImages() function. This is where everything wasm-vips related happens.

function processImages(file) {
    // make sure a file was actually passed in
    if (file.length === 0) return;

    // Create a new FileReader.
    // This is what gets us our image data
    const fr = new FileReader();

    // This fires once the image has been loaded
    // as an ArrayBuffer
    fr.onload = (event) => {
        // This is where the wasm-vips code will go
    }

    // This initiates reading the file.
    // Note that we are using readAsArrayBuffer
    // this makes an ArrayBuffer that we can 
    // then give to wasm-vips. There are other
    // options but this seems to work the best in my testing.
    fr.readAsArrayBuffer(file);

Now inside our fr.onload event we will have event.target.result, this is our image in the form of an ArrayBuffer.

Note: going forward we will be working with wasm-vips, I'm not including some of the basic setup stuff here that is available in the documentation, just the stuff that is not clearly documented anywhere.

So now that we have our image data we want to process it with wasm-vips, right? Well it's actually not that hard, what we need to use is vips.Image.thumbnailBuffer(). This function takes our event.target.result as the file, a width and some settings.
It looks something like this:

im = vips.Image.thumbnailBuffer(event.target.result, width, {
    height: height,
    no_rotate: true,
    crop: crop
});

The file and width are required, the settings are optional.

Processing images with wasm-vips

Now that you hopefully have an idea of how this works let's look at some examples of processing images.

Resize an image

Say we want to resize an image within certain dimensions(1000px x 1000px) but not crop any of the image out.

im = vips.Image.thumbnailBuffer(event.target.result, 1000, {
    height: 1000,
    no_rotate: true,
    crop: vips.Interesting.none
});

vips.Interesting.none on the crop setting tells wasm-vips not to remove any part of the image.

Also note where we have set the values for width and height. This ensures our image will be within these dimensions, if you removed the height setting then the image would only be constrained by the width. It's important to remember that width is required.

Crop an image

Now if we want to crop the image we simply change the crop setting to vips.Interesting.attention. This has the added benefit of applying a "smart cropping" that attempts to focus the crop on the most interesting part of the image.

im = vips.Image.thumbnailBuffer(event.target.result, 1000, {
    height: 1000,
    no_rotate: true,
    crop: vips.Interesting.attention
});

Note: if you have ever used libvips or one of it's associated libraries you are probably familiar with its tendency to rotate images unexpectedly(there are reasons, they are just beyond this post) so I always include the no_rotate: true setting.

Converting and Compressing

The final step we need to take with wasm-vips is outputting our image, this also happens to be where you can convert and compress the image. For this post I am only going to cover saving a jpeg, as png compression is much more complicated and probably needs a post of it's own. Let's dive right into that code.

outBuffer = im.jpegsaveBuffer({ Q: 60});

Not that scary right? A couple notes about this line, if you remember from up-top we are storing our image in the im variable, so that is how we are actually getting the new image and saving it. And our Q setting is the quality of the image a.k.a. the compression. It's on a scale of 1-100, 100 being the highest quality and least compression.

saveBuffer functions

The saveBuffer functions are how you save different file types, if we wanted to save a png it would be pngsaveBuffer(), webp would be webpsaveBuffer(). There are a whole bunch of the saveBuffer functions available. You can run these functions without passing any settings as well and they generally default to reasonable settings, however it's worth pointing out that different file formats have different settings and sometimes they can be difficult to decipher.

Saving an image

At this point we have our processed image but we need to do something with it. In a lot of cases I'm sure the image will need to be uploaded to a server or displayed on screen but for tiny.photos I send the image right back to the user, so let me show you how I do that.

// using our outBuffer variable from the last step
// to create a blob
const blob = new Blob([outBuffer], { type: `image/jpeg` });
const blobURL = URL.createObjectURL(blob);

// here we create a hidden button that links to the blob
// and automatically clicks it with JS
// and the download is initiated on the client
const downloadLink = document.createElement("a");
downloadLink.href = blobURL;
// this is where you can name the file
// make sure you have the correct extension
downloadLink.download = `file-name.jpeg`;
document.body.appendChild(downloadLink);
downloadLink.click();
document.body.removeChild(downloadLink);

And there you have it, an image entirely processed with web assembly on the client, no server required!

If you are diving head first into wasm-vips and had the same troubles I had learning a library that doesn't have any documentation I would recommend looking at the libvips documentation, finding what you want to use and then searching the wasm-vips repo for functions of the same or similar names. It's not easy but it gets the job done. Also when you match functions you can use libvips as a reference for what arguments the function takes, I haven't found any case where they didn't work the exact same.

If you would like to see wasm-vips in action please checkout tiny.photos.