Best practices for file uploads in Nodejs and HTML

Although it seems like uploading a file to the server is a simple task, there are many ways to do it and there are some pitfalls and edge cases, so the purpose of this article is to overview the whole process end to end and to take a closer look at each aspect.

Let's begin from the

Creating upload form

To upload files to the server from the browser generally we need to create a html form. The most stupid simple example of the upload form that can be is:

<form method="POST" enctype="multipart/form-data">
  <input type="text" name="myText" />
  <input type="file" name="myImage" />
  <input type="submit" />
</form>

It works as is. No javascript needed.

However most likely we want something more sophisticated, for example if we are uploading images we may want to show preview thumbnails with a file size label or we may want to see a progress bar with pause/resume button or whatever else it may be.

Possibly the most common requirement is to replace default html <input type="file"> with nice dropzone area, there are javascript libs that can do that, but in reality you may not even need library, it can be done with simple input type="hidden" (to prevent that less-than-attractive user interface from being displayed). Because it is not affecting functionality we will skip form ui and styling here, there are plenty of good tutorials in the web (MDN) on this topic.

The html of basic upload form looks like this:

<form>
  <h4>Multipart from data upload example</h4>
  <input id="text" type="text" name="myText" />
  <input id="image" type="file" name="myImage" 
                                multiple accept="image/*" />
  <input id="submit" type="submit" value="Upload" />
  <hr />
  <h4>Preview</h4>
  <div id="preview"></div>
</form>

Couple of interesting points here:

1. has accept attribute that is handy to limit input from accepting undesired file types. Another multiple attribute allows input to accept multiple files, omit it if you want to limit it to one (In this tutorial we will upload a single image).
2. Example of text input here was added just for example, it may be handy to pass a caption with image in the same payload or some initial data that is needed to create an entry in db.

Let's add some javascript to bring form alive:

window.onload = function () {
  const submitButton = document.getElementById('submit')
  const imageInput = document.getElementById('image')
  const textInput = document.getElementById('text')
  const preview = document.getElementById('preview')

  submitButton.addEventListener('click', async e => {
    e.preventDefault()

    const [file] = imageInput.files

    if (!file) {
      throw new Error('File was not selected')
    }

    // TODO - send file
  })
}

Nothing special, just some boring boilerplate to get values from inputs and register click handler for the submit button.

Select file

To add some spice we can show preview for the image, when the user drops it into input.

imageInput.addEventListener('change', e => {
  // we are taking only the first element
  // because we are doing single file upload
  const [file] = imageInput.files
  function updatePreviewImage(file){
    // TODO - update preview <img> src 
  }
})

Now when we have reference to the selected file we need to create a DOMString for <img> tag. There is browser interface method URL.createObjectURL() that can take underlying ArrayBuffer and create DOMString that represents the specified File object.

function updatePreviewImage(file) {
  const url = URL.createObjectURL(file)
  preview.innerHTML = `<img src="${url}" />`
}

Let’s take a look on src attribute of appeared <img> tag:
<img src="blob:http://localhost:3000/1b2a4ac9-4bd4-4726-b302-d74e6ed2ba48">

As you can see, url of page where our html is hosted is part of DOMString, if page will be opened with file:/// protocol and not being hosted, then DOMString will work but look like this:
<img src="blob:null/f8111cf8-d598-4305-9bdd-4ba5b7db22f7">.

This illustrates that URL lifetime is tied to the document in the window on which it was created. That means that we have to release an object URL, by calling revokeObjectURL() after the form submission.

Building FormData

Building FormData is straightforward:

const formData = new FormData()
formData.append('myImage', file)
formData.append('myText', textInput.value || 'default text')

• First parameter is the name of the property in the request.body object when we will get a request later on server.
• Second is the value and there is an optional third parameter that may hold the original filename of the Blob object if we are sending Blob.

Adding file as a Blob

Sometimes we need to upload files that are not images, like 3d models, audio records or any other binary file format. In some cases it may be useful to treat them as Blobs, example:

const [file] = imageInput.files
const arrayBuffer = await file.arrayBuffer()
const myBlob = new Blob([new Uint8Array(arrayBuffer)], {
  type: file.type,
})
formData.append('myBlob', myBlob, file.name)

Sending FormData to server

We can send data to server with simple fetch

await fetch(uploadUrl, {
  method: 'POST',
  body: formData,
})

There is small pitfall however:

Warning: When using FormData to submit POST requests using XMLHttpRequest or the Fetch_API with the multipart/form-data Content-Type (e.g. when uploading Files and Blobs to the server), do not explicitly set the Content-Type header on the request. Doing so will prevent the browser from being able to set the Content-Type header with the boundary expression it will use to delimit form fields in the request body.

Open DevTools and take a look on request headers, you will see that browser automatically adds Content-Type: multipart/form-data and then it appends random boundary value that is used to separate parts of form-data

Content-Type:
  multipart/form-data; boundary=---WebKitFormBoundaryrHPgSrtbIrJmn

Display progress bar

Unfortunately currently it is not possible to get file upload progress for fetch() method chromestatus.com.

Solution for now is to use good ol' fella XMLHttpRequest

let request = new XMLHttpRequest()
request.open('POST', '/upload')

// upload progress event
request.upload.addEventListener('progress', function (e) {
  // upload progress as percentage
  console.log((e.loaded / e.total) * 100) 
})

// request finished event
request.addEventListener('load', function (e) {
  // HTTP status message (200, 404 etc)
  console.log(request.status)
})

// send POST request to server
request.send(formData)

Server side

Before we dive into the code let's stop and think for a while.

• Do we need to store files on the server?
• What exactly has to be done with files on the server, is there any part of that responsibility that can be passed to third-party tools?
• Can move files to external storage like AWS S3 or Azure Blob Storage without temporarily saving them on the server?

Libraries for processing file uploads

To process uploaded files we need a library that knows how to do it in an efficient and secure way. There is good comparison article on this (link at the bottom of the page).

We gonna stick with busboy becasue it is considered the most production stable solution (other libraries using it under the hood) and becasue it do not creates temporary files.

If we do need to save files occasionally, we can stream file contents to disk like this:

const imageBuffer = req.raw.files['myImage'].data;
  const fileName = getRandomFileName();
  const stream = fs.createWriteStream(fileName);
  stream.once('open', function (fd) {
    stream.write(imageBuffer);
    stream.end();
  });

Or we can can take multer library that is based on busboy too and it has option to automatically save files on disk upon receiving.

Web Framework

In this tutorial we will use a web framework, despite the fact that we do not need a web framework to receive uploaded files.

Why? It’s because in real projects, almost always we do need a web framework, unless we are doing something very specific, so we want to know how to properly tie our library with the framework.

Official Fastify plugin for uploading files is fastify-file-upload, if will take a closer look at it’s repo, we will see that is it nothing more than a wrapper around another library express-fileupload, that is by itself a wrapper around busboy.

So for Fastify we gonna use fastify-file-upload and for Express express-fileupload. Using wrapper is convinient for example, you may define validation schema for formdata, but we use busboy directly without wrapper too. Let's write our own wrapper around busboy library.

Writing Fastify wrapper for busboy

Writing a wrapper is really simple task, there is only one tiny thing that Fastify out of the box supports only the application/json context-type, so we need to define our parser for multipart/form-data

fastify.addContentTypeParser('multipart/form-data', function (request, payload, done) {
  done(err, payload)
})

Fasity exposes original nodejs request and response under request.raw and reply.raw

fastify.post('/upload', function (request, reply) {
  const req = request.raw
  const res = reply.raw
  // TODO - copy paste code from busboy example as is, 
  // example: new Busboy({ headers: req.headers }) ...
  // full example in the demo repo

We just put some code in route handler, and it works, but this is not the right approach, Fastify gives us much cleaner was to do it, to register our hander as a plugin.

Blobs

There is nothing special in receiving Blobs, the same server code works as is, the only difference is that it may be missing original filename if it was not provided as third parameter to formData.append

Security

It is important to know that there are many types of vulnerabilities that may be exploited if there is a bug in the processing buffers, more information here.

It is considered a good practice to take out upload endpoints to separate microservice that will have an additional layer of security.

Secure file uploads rules

1. Always create a new unique filename, never use one provided by a client, because it may intentionally include paths to critical system files on the server.

2. Never host uploaded files from the same document root, better to host them on totally different machines in different networks.

3. Any file may be malicious, extension doesn’t mean anything, it is better to perform some third-party malware scan if it is possible.

4. Keep control of permissions, files should not be executable

5. Authenticate file uploads, limit number of uploaded files per session, limit file size range

Link to example repository

https://github.com/bfunc/file-upload-example

Further reading

• Guide on how to manage upload of large files (more than 512mb)

• Good post about differences in libraries for processing file uploads