Upload Files to S3 in Node.js

#node #javascript #aws

My excitement to implement file upload didn't take too long to turn into a struggle, then dread but finally, a victorious push to the finish. This is my humble attempt to help you skip the line and jump directly to the third phase.

If you are someone who is here for a specific piece of information, you may skip to any of the below:

1. Upload file as a whole using Multer

2. Upload in chunks with Busboy

Let's begin by making an uploader for Vue.js

First off, let's allow our user to upload a file using Vue.js so that it may reach our API.

For that, we starts with the tag:

<input type="file" :accept="allowedMimes" ref="inputFile" @change="onChange"/>

The above input tag allows a user to upload a single file. Once a file is selected, the onChange method is called with the file data.

The onChange method looks like below:

function onChange() {
  const data = new FormData();
  for (const [key, value] of Object.entries(this.options)) {
    data.append(key, value);
  }

  const file = this.$refs.inputFile.files[0];
  data.append('file', fileToUpload, file.name);
  const {data: res} = await axios.post(API`/files`, data);
}

With this, our front-end is good to go and now, we are ready to send our file off to S3.

Multer-S3 saves the day

This approach will let you upload a file directly to AWS S3, without having to do anything in between.

When to use this approach:

You want to pipe your data to a location in your S3 bucket without modifying or accessing the file bytes. In short, this method will pipe your whole file without you having to do anything.

Here's how the basic skeleton looks like. It contains your multer declaration and the API endpoint.

const upload = multer({});

router.post('/file', upload.single('file'), async (req, res) => {

});

We start by specifying the upload method:

const multer = require('multer');
const multerS3 = require('multer-s3');

const upload = multer({
  storage: multerS3({
    s3, // instance of your S3 bucket
    contentDisposition: 'attachment',
    contentType: multerS3.AUTO_CONTENT_TYPE,
    bucket(req, file, callb) {
      // logic to dynamically select bucket
      // or a simple `bucket: __bucket-name__,`
      callb(null, '_my_bucket_');
    },
    metadata(req, file, cb) {
      cb(null, {
        'X-Content-Type-Options': 'nosniff',
        'Content-Security-Policy': 'default-src none; sandbox',
        'X-Content-Security-Policy': 'default-src none; sandbox',
      });
    },
    async key(req, file, abCallback) {
      try {
        // logic to dynamically select key or destination
        abCallback(null, ' _dest/key_');
      } catch (err) {
        abCallback(err);
      }
    },
  }),
  limits: {}, // object with custom limits like file size,
  fileFilter: filterFiles, // method returns true or false after filtering the file
});

We then pass it as a middleware to our API end-point.

router.post('/file', upload.single('file'), async (req, res) => {
    // you can access all the FormData variables here using req.file._var_name
});

This is it! All the data pertaining to your S3 upload will be available under the req.file variable.

With that, we have successfully uploaded your file to s3, the easy way.

When save the day with Busboy

Then comes a situation where you want to have access of the bytes you are piping to your S3 bucket, before the actual upload happens. You might want to compress them, uncompress them, check for virus, or fulfil any other endless requirements. I decided to use Busboy here, it's a tried, tested and an easy to use library. Other options you may go for are libraries like Formidable or Multiparty.

When to use this approach:

You want to access the file chunks, modify them or use them before you pipe them to your S3 bucket.

Here's how the basic structure looks like. It again, contains the basic definition along with our usual API endpoint.

const busboyUpload = (req) => {};

router.post('/file', async (req, res) => {
});

So, let's dive right in. The Busboy is called as a method from our API with the request as its parameter as defined below.

router.post('/file', async (req, res) => {
  try {
    const uploadedFileData = await busboyUpload(req);
    req.file = uploadedFileData;
    res.sendStatus(200);
  } catch (err) {
    res.sendStatus(500);
  }
}

Our Busboy uploader will be set up in a simple and straight forward manner.

We start by returning a Promise and initiate our Busboy instance along with the basic structure.

const busboyUpload = (req) => new Promise((resolve, reject) => {
  const busboy = new Busboy({});
});

We then define an array that will help us check whether the upload has finished or not. This will allow us to return a suitable response.

const fileUploadPromise = [];

In this next step, we will work on the actual file. We define the listener that executes when a file is encountered.

busboy.on('file', async (fieldname, file, filename, encoding, mimetype) => {
  // check for conditions and set your logic here
  // s3Bucket = '_Bucket_';
  // s3Key = '_Key_';
  // check file size and file type here
});

Inside the onFile listener above, we will upload to S3 using Read and PassThrough stream. The way our streams and S3 upload will be defined is:

const { Readable, PassThrough } = require('stream');
const s3 = require('@/utils/awsConnect').getS3();

const passToS3 = new PassThrough();
const fileReadStream = new Readable({
  read(size) {
    if (!size) this.push(null);
    else this.push();
  },
});
fileUploadPromise.push(new Promise((res, rej) => {
  s3.upload({
    Bucket: bucket,
    Key: key,
    Body: passToS3,
    contentDisposition: 'attachment',
  }, (err, data) => {
    if (err) {
      rej();
    } else {
      res({ ...data, originalname: filename, mimetype });
    }
  });
}));
fileReadStream.pipe(passToS3);

Whats happening here: We create the Read stream, pass it to PassThrough and after creating PassThrough we pipe it to the S3 upload function. Before beginning the upload, we push it as a Promise to the fileUploadPromise array we created earlier.

To begin the file upload, we define the following listeners inside our onFile listener. On a chunk/data event, we push the same to the Read stream that will in turn push it to our S3.

file.on('data', async (data) => {
  fileReadStream.push(Buffer.from(nextChunk));
});
file.on('end', () => {
  fileReadStream.push(null);
});

Lastly, we define our onFinish event, pipe the request to BusBoy, sit back and relax. You will notice, we wait for the fileUploadPromise to complete here before we send a response back.

busboy.on('finish', () => {
  Promise.all(fileUploadPromise).then((data) => {
    resolve(data[0]);
  })
    .catch((err) => {
      reject(err);
    });
});
req.pipe(busboy);

In the end this is how your BusBoyUpload structure should look like.

const busboyUpload = (req) => new Promise((resolve, reject) => {
  const busboy = new Busboy({ });
  busboy.on('file', async (fieldname, file, filename, encoding, mimetype) => {
    fileReadStream.pipe(passToS3);
    file.on('data', async (data) => {
    });
    file.on('end', () => {
    });
  });
  busboy.on('finish', () => {
  });
  req.pipe(busboy);
});