Upload Media directly to S3 bucket from Client.

Web is becoming media rich day by day.

Most of the webapps now provides functionality of uploading user generated content, this content could be an Image or Video.

Generally the cloud storages used for storing media are storage buckets like S3 buckets

Now if we want to upload image to S3 bucket, there are 2 methods:

1. From Server - media upload to S3 bucket will happen on server and it is costly process on large scale

2. From Client - we are about to see it in detail here, in this method media upload to S3 bucket happens from the client which saves server processing and cost

Client Side Media Upload Algo

1. Server uses aws sdk to use S3 bucket methods
2. Server expose and API to get Signed Url from aws and send to client
3. Client hits the API to get the signed url from the server
4. Client post the file on the signed url using XHR request
5. Client will be able to track the progress of the upload and can take some action once the Upload is complete
6. Enable CORS on S3 bucket setting

---

Now lets do some coding

Server

1. Create simple Express Server
2. Expose an endpoint to get signed URL

// server
// npm install aws-sdk

const express = require("express");
const app = express();
const port = 3000;

const AWS = require("aws-sdk");
const s3 = new AWS.S3({
  accessKeyId: "<aws_access_key_id>", // aws access id here
  secretAccessKey: "<aws_secret_access_key>", // aws secret access key here
  useAccelerateEndpoint: true
});
const params = {
  Bucket: "<Bucket Name>",
  Key: "<Put your key here>",
  Expires: 60*60, // expiry time
  ACL: "bucket-owner-full-control",
  ContentType: "image/jpeg" // this can be changed as per the file type
};

// api endpoint to get signed url
app.get("/get-signed-url", (req, res) => {
  const fileurls = [];
  s3.getSignedUrl("putObject", params, function(err, url) {
    if (err) {
      console.log("Error getting presigned url from AWS S3");
      res.json({
        success: false,
        message: "Pre-Signed URL error",
        urls: fileurls
      });
    } else {
      fileurls[0] = url;
      console.log("Presigned URL: ", fileurls[0]);
      res.json({
        success: true,
        message: "AWS SDK S3 Pre-signed urls generated successfully.",
        urls: fileurls
      });
    }
  });
});

app.listen(port, () => console.log(`Server listening on port ${port}!`));

---

S3 Bucket setting

1. Create a DNS compliant bucket name
2. Set default encryption
3. Give appropriate read and write permissions
4. Get aws accelerated URL like yourbucketname.s3-accelerate.amazonaws.com
5. Add following CORS rules

<?xml version=”1.0" encoding=”UTF-8"?>
<CORSConfiguration>
<CORSRule>
 <AllowedOrigin>*</AllowedOrigin>
 <AllowedMethod>POST</AllowedMethod>
 <AllowedMethod>GET</AllowedMethod>
 <AllowedMethod>PUT</AllowedMethod>
 <AllowedMethod>DELETE</AllowedMethod>
 <AllowedMethod>HEAD</AllowedMethod>
 <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

---

Client

1. Make API call to server, get signed URL
2. Post multipart formdata to signed URL
3. Track progress, make UI changes accordingly

import axios from "axios";
const getSignedURL = () => {
  return new Promise((resolve, reject) => {
    axios
      .get("<server-base-url>/get-signed-url")
      .then(data => {
        resolve(data);
      })
      .catch(err => {
        reject(err);
      });
  });
};

const uploadMediaToS3 = () => {
  const config = {
    onUploadProgress: function(progressEvent) {
      var percentCompleted = Math.round(
        (progressEvent.loaded * 100) / progressEvent.total
      );
      console.log(percentCompleted);
    }
  };

  let fd = new FormData();
  fd.append("file", files[0]);

  getSignedURL().then(data => {
    axios
      .put(data.urls[0], fd, config)
      .then(res => console.log("Upload Completed", res))
      .catch(err => console.log("Upload Interrupted", err));
  });
};

// add event listener to a html element
const uploadButton = document.getElementById("uploadButton");
uploadButton.addEventListener("onclick", uploadMediaToS3);

Advantages of using client side upload method

1. It reduces load on servers.
2. Client can show actual upload progress
3. Client can abort the upload
4. Client can handle upload stop, pause and resume functionalities
5. At AWS we can pipe lambdas to do processing on these images and make it public only after processing it.

As per the use case server may want to maintain the state of uploads started by client

Comments

[gwendall]

Thanks for the great tutorial ! As a side note, attempting to upload a file with a signed URL without setting the region property of my server S3 instance resulted in a 400 error.

[Vishal Gupta]

It was not coming for me. But I will surely checkand get back.

Thanks for pointing out.

[Akshat Tulsani]

Hi, I am new to serverless, I was trying to do something similar by creating a Lambda function to generate my signed URL for my S3 bucket, although my signed URL is generated when I try to do a PUT request on the same I get a 403 forbidden response. I have double-checked my permissions can you please help me out.
Detailed on StackOverflow: stackoverflow.com/questions/636237...

Thank you for the help !!

[Abhishek]

Nice.. Really way will help in speed up the uploading media content. On the contrary note, image validation, image score calculation and duplicate image check handling seems reliable and easy on server side. Please suggest if we can do above checks in client side uploading.

[Vishal Gupta]

Once the image is uploaded on S3 you can initiate a lambda to do any processing or validation on the image.
These lambdas are horizontal scalable and superfast.