ReactNative Large file upload using Amplify S3

Currently while using Amplify library to upload a to AWS S3 we have to read that file into memory and pass it to Storage.put call. If the file which we are uploading is large for example a video, it causes memory issues and the ReactNative app crashes.
The Amplify Storage plugin supports Managed upload. It will divide the file into small chunks and upload them in batches.
But there is no support to read the file in small chunks.

The Amplify library supports custom plugins, using which we can connect with services other than AWS or add a wrapper to AWS services.

To support the uploading of large files using Amplify, I have used the same mechanism as a custom plugin.

NPM Library amplify-s3-chunk-upload

---

Usage

Install

npm i -s amplify-s3-chunk-upload

Configure Storage plugin in App.js (ReactNative)

import { StorageChunkUpload } from 'amplify-s3-chunk-upload';
import { Credentials } from '@aws-amplify/core';



// put following code after Amplify.configure

// Load StorageChunkUpload Plugin
const storagePlugin = new StorageChunkUpload({}, Credentials);
Storage.addPluggable(storagePlugin);
storagePlugin.configure(config);

File upload call

// get File stats
const { size } = await RNFS.stat(fileURI);

// here we are simulating an array of bytes
const fileObject = {
  // set the size
  size: size,

  // here we will read file as per bodyStart & bodyEnd, this will avoid reading complete file in the memory.
  slice: (bodyStart, bodyEnd) => {
    // Here in this sample code, we are using react-native-fs to read files.
    return RNFS.read(fileURI, bodyEnd - bodyStart, bodyStart, 'base64')
      .then((data) => Buffer.from(data, 'base64'))
      .catch((error) => {
        // Log error if required.
      });
    },
  };

  // Upload call, for parameters, refer to Amplify docs.
  const result = await Storage.put(`Your-file-name.mp4`, fileObject, {
    contentType: 'video/mp4',
    level: 'protected',
    provider: 'StorageChunkUpload',
  });

Since we are making standard Storage.put call and the underlying code also uses the same Amplify Library code, you can pass all other parameters such as progressCallback etc.

---

We have this in action for the last couple of days.
Please feel free to share any feedback.
NPM
GitHub

Comments

[loljohndoe]

hey,
Im getting this error.
ERROR [ERROR] 44:48.50 axios-http-handler [Error: Request failed with status code 403] ERROR [ERROR] 44:48.91 AWSS3ProviderManagedUpload - error happened while uploading a part. Cancelling the multipart upload [Error: Request failed with status code 403] ERROR [ERROR] 44:48.135 axios-http-handler [Error: Request failed with status code 403] ERROR [ERROR] 44:48.162 axios-http-handler [Error: Request failed with status code 403] ERROR [ERROR] 44:48.188 axios-http-handler [Error: Request failed with status code 403] ERROR [ERROR] 44:48.288 axios-http-handler [Error: Request failed with status code 404] WARN [WARN] 44:48.304 StorageChunkUpload - error uploading [Error: Upload was cancelled.undefined] ERROR [Error: Upload was cancelled.undefined]

[Nikhil Walvekar]

Let me know if this fix works for you
github.com/walvekarnikhil/amplify-...

[loljohndoe]

Im using the patched version 2.0.1 but im still getting the error. please tell me if i miss anything in the thread.

[Dennis Orsini]

I get a 403 error when passing credentials from amplify (v. 3.3.24) and trying to upload with the plugin, using storage without the plugin works with no such error.

[Nikhil Walvekar]

Sorry for the late reply, I somehow missed the comment.
There was a similar issue reported and a new version was published.
github.com/walvekarnikhil/amplify-...

Let me know if this fix works for you.

[Kevin López González]

when the network disconnects, the app crash