DEV Community: Sanjeet Sahay

axios/got faceoff: uploading a file to Amazon WorkDocs using Amazon S3 pre-signed url

Sanjeet Sahay — Mon, 18 May 2020 22:11:56 +0000

Overview

If you are using Amazon WorkDocs as your managed cloud based content management and/or storage system, and if you are planning to automate tasks such as integrate it with other document/content storage systems, then you must have come across the use case of uploading a file. After some automation, you should be able to do this at scale. In a future post, I will share a detailed reference architecture on how to build such an integrated system.

The following sections demonstrate the various aspects of the app, starting with setting up a simple Node.js app. However, there are some prerequisites

AWS IAM user with sufficient privileges e.g. I am using my development account and I have created a user with admin privileges without AWS Management console access, and I rotate its access keys regularly. For more, read AWS IAM best practices
an existing Amazon WorkDocs site
Install and configure AWS CLI with named profiles

Initialize npm project

I have used the following commands to initialize a new npm project

➜ mkdir workdocs-sample && cd workdocs-sample
➜ npm init
➜ npm install aws-sdk axios form-data got
➜ touch index.js

After initialization, my folder structure looks like this:

➜  workdocs-sample ls
da-quiz-storage-result.pdf 
index.js                   
node_modules               
package-lock.json          
package.json               
yarn.lock

Initialize the WorkDocs client

Setup AWS credentials in index.js. For more information, read best practices to use AWS credentials in your development environment

const AWS = require("aws-sdk");
const credentials = new AWS.SharedIniFileCredentials({ profile: "default" });
AWS.config.credentials = credentials;

In addition to that, you'll need the following declarations

const got = require("got");
const fs = require("fs");
const FormData = require("form-data");
const workdocs = new AWS.WorkDocs();

Finally, initialize the WorkDocs client

const workdocs = new AWS.WorkDocs();

Steps to upload a file

To upload a file to a WorkDocs folder you need the following:

a folder ID to upload
- to get the root folder ID, you need to make a call to describeUsers API
- if you have created new folders at the root, then you need to call describeFolderContents with the root folder ID
call initiateDocumentVersionUpload with the folder ID, name of the file, and optionally, a content type. It returns an Amazon S3 pre-signed upload url, document ID, and a version ID among other things
use got to upload the file to the returned uploadUrl
call updateDocumentVersion with document ID, version ID, and set VersionStatus to ACTIVE

Get the root folder ID

Every user has a root folder which can contain one or more children - nothing fancy, just the usual nested folder structure. The root folder has an ID that can used to create folders inside it. Using the describeUsers API call, we'll get the root folder ID for the user defined by the query parameter. You can look up OrganizationId from your Amazon WorkDocs AWS console.

const describeUsers = async () => {
  const user = await workdocs
    .describeUsers({
      OrganizationId: "d-92672xxxxx", // your WorkDocs organization Id
      Query: "sahays", // name of an existing WorkDocs user
    })
    .promise();
  return user;
};

Initialize upload

The following code uses initiateDocumentVersionUpload to initiate the process of uploading a file. The api requires ParentFolderId to upload the file to, and a Name. It returns a documentId for the document, versionId for the first version of the document, uploadUrl containing the Amazon S3 pre-signed url, and signedHeaders containing the content-type and x-amz-server-side-encryption encryption type.

const initUpload = async ({ folderId, filename }) => {
  try {
    console.log("initUpload");
    const contentType = "application/octet-stream";
    const initResult = await workdocs
      .initiateDocumentVersionUpload({
        ParentFolderId: folderId,
        Name: filename,
        ContentType: contentType,
        ContentCreatedTimestamp: new Date(),
        ContentModifiedTimestamp: new Date(),
      })
      .promise();
    const documentId = initResult.Metadata.Id;
    const versionId = initResult.Metadata.LatestVersionMetadata.Id;
    const { UploadUrl, SignedHeaders } = initResult.UploadMetadata;
    console.log("initUpload complete");
    return {
      documentId,
      versionId,
      uploadUrl: UploadUrl,
      signedHeaders: SignedHeaders,
    };
  } catch (e) {
    console.log("failed initUpload", e);
    throw e;
  }
};

The header looks like the following:

headers: {
    'Content-Type': 'application/octet-stream',
    'x-amz-server-side-encryption': 'AES256'
  }

Upload a file using `got`

The following code uses got npm library to upload a local file. Please note, we are using a PUT request. The file is appended to FormData using a file stream object. The headers retrieved from the previous call initiateDocumentVersionUpload is used to set a PUT request header.

const uploadFile = async ({ filename, signedHeaders, uploadUrl }) => {
  try {
    if (fs.existsSync(filename)) {
      console.log("reading file stream");
      const fileStream = fs.createReadStream(filename);
      console.log("preparing form data");
      const formData = new FormData();
      formData.append(filename, fileStream);
      console.log("uploading to ", uploadUrl);
      const extendParams = {
        headers: signedHeaders,
      };
      console.log("got extendParams", extendParams);
      const client = got.extend(extendParams);
      await client.put(uploadUrl, {
        body: formData,
      });
      console.log("upload complete");
    } else {
      console.log("file doesn't exist");
      throw "file doesn't exist";
    }
  } catch (e) {
    console.error("failed uploadFile", e);
    throw e;
  }
};

Update document version

This important step completes the file upload transaction by setting the VersionStatus to ACTIVE which tells Amazon WorkDocs to mark the just uploaded file as the most recent/active version.

const updateVersion = async ({ documentId, versionId }) => {
  try {
    await workdocs
      .updateDocumentVersion({
        DocumentId: documentId,
        VersionId: versionId,
        VersionStatus: "ACTIVE",
      })
      .promise();
    console.log("document version updated");
  } catch (e) {
    console.log("failed updateVersion", e);
    throw e;
  }
};

Time for that faceoff: `got` vs `axios`

Let's take a look at axios invocation first.

await axios.put(uploadUrl, formData, {
        headers: signedHeaders
      });

This results in Amazon S3 rejecting the request with the following error:

<Error>
<Code>NotImplemented</Code>
<Message>A header you provided implies functionality that is not implemented</Message>
<Header>Transfer-Encoding</Header>
<RequestId>016D6B18F95E6923</RequestId><HostId>QgYnoYEQTZR4jG7wvdLfAe6lcd2Tg+/eAOeHLvtM+CamqyDxZX8p7CV4ZL+Hph7+IOUiFJkayT8=</HostId>
</Error>

The server returns a 501: not implemented response

response: {
    status: 501,
    statusText: 'Not Implemented',
    headers: {
      'x-amz-request-id': '016D6B18F95E6923',
      'x-amz-id-2': 'QgYnoYEQTZR4jG7wvdLfAe6lcd2Tg+/eAOeHLvtM+CamqyDxZX8p7CV4ZL+Hph7+IOUiFJkayT8=',
      'content-type': 'application/xml',
      'transfer-encoding': 'chunked', // extra header
      date: 'Mon, 18 May 2020 22:00:24 GMT',
      connection: 'close',
      server: 'AmazonS3'
    },...
}

Now, let's take a look at the got invocation:

const extendParams = {
  headers: signedHeaders,
};
console.log("got extendParams", extendParams);
const client = got.extend(extendParams);
await client.put(uploadUrl, {
  body: formData,
});

This results in a successful 200: OK response with the same inputs

Bring it all together

The following is the entry point function that runs as a result of running index.js using node index.js

const start = async () => {
  try {
    const user = await describeUsers();
    const rootFolderId = user.Users[0].RootFolderId;
    const filename = "da-quiz-storage-result.pdf";
    const {
      documentId,
      versionId,
      uploadUrl,
      signedHeaders,
    } = await initUpload({ folderId: rootFolderId, filename });
    await uploadFile({ filename, signedHeaders, uploadUrl });
    await updateVersion({ documentId, versionId });
  } catch (e) {
    console.error(e);
  }
};

start();

Finally

After running node index.js in your terminal, you'll see an output similar to the following:

initUpload
initUpload complete
reading file stream
preparing form data
uploading to  https://gb-us-west-2-prod-doc-source.s3.us-west-2.amazonaws.com/1b45f47aa1c4d1d1c1f0978587e10f1e56ce801824ca5d5fce0565dea6f76baf/1589767973739-0d3c7a46986cfe7d0fd8beec8258628a8b6ca0e9b0f412afafcdaf9c6aa7a00e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200518T021253Z&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-server-side-encryption&X-Amz-Expires=900&X-Amz-Credential=AKIAIM5HWZT6CVS2WHIA%2F20200518%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=025e9ed29fe7f8ab85593c51a4a09b396909de47ea1e893148df14e3435ea080
got extendParams {
  headers: {
    'Content-Type': 'application/octet-stream',
    'x-amz-server-side-encryption': 'AES256'
  }
}
upload complete
document version updated

The file da-quiz-storage-result.pdf is now uploaded as shown in this screenshot:

Best practices to use AWS access key and secret in your development environment

Sanjeet Sahay — Sun, 17 May 2020 17:36:01 +0000

If you are an AWS developer and are using AWS services in your app, then you must have found yourself looking for the best way to securely store and access your AWS credentials. To keep our AWS account secure, it's important for us to understand the AWS shared responsibility model.

In a nutshell, it states that AWS is responsible for the security of the cloud and us, the customers, are responsible for the security in the cloud. Simply put, for developers, it means that we should take special care of our AWS credentials like Access key ID and Secret Access Key.

If you are new to AWS, use the references section below for more information.

1. Anti-pattern: Hardcoding credentials

This is an anti-pattern and must be avoided at all costs. If your code looks like the following then you must act now



const AWS = require("aws-sdk");
AWS.config.update({
 credentials: {
  access_key_id: "<your-access-key-id>",
  secret_access_key: "<your-secret-access-key>"
 }
})

1.1. Why is this bad?

As a developer, you are most likely to commit this code in some repository like a private GitHub repo or your team repository such as BitBucket or AWS CodeCommit. Besides running a risk of using an anti-pattern, you don't want someone to access your hard-coded keys, because it will allow them to access/manage all the resources that these credentials provide access to. If the IAM policy attached to the user whose credentials you are using looks like the following, it means that you have handed over the keys to your AWS kingdom to anybody who has access to your code



{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*"
  }
}

1.2. How do I mitigate?

If you think that you can't make changes to your code, then you must modify the IAM policy attached to that role or move them to an IAM group with restrictive privileges e.g. IAM policy that grants least privileges to only a given Amazon S3 bucket:



{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListYourObjects",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": ["arn:aws:s3:::bucket-name"]
        },
        {
            "Sid": "ReadWriteDeleteYourObjects",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": ["arn:aws:s3:::bucket-name"]
        }
    ]
}

2. Look ma, no hardcoded credentials

With that anti-pattern out of the way, you may take one of the following approaches to use your AWS credentials.

2.1. Use environment variables



$ export AWS_ACCESS_KEY_ID="<your-access-key-id>"
$ export AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"

then, in your JavaScript/Node.js app, use the following



const AWS = require("aws-sdk");
AWS.config.update({
 credentials: {
  access_key_id: process.env.AWS_ACCESS_KEY_ID,
  secret_access_key: process.env.AWS_SECRET_ACCESS_KEY
 }
})

2.2. Use AWS profile

You can use AWS named profiles to store more than one credential. You can inspect the following two files:

~/.aws/credentials: contains aws_access_key_id and aws_secret_access_key
~/.aws/config: contains region and output

My ~/.aws/credentials file looks like the following and it shows that I am using 2 profiles: default and personal



[default]
aws_access_key_id = "<your-access-key-id>"
aws_secret_access_key = "<your-secret-access-key>"

[personal]
aws_access_key_id = "<your-access-key-id>"
aws_secret_access_key = "<your-secret-access-key>"

My ~/.aws/config file looks like the following:



[default]
region = us-west-2
output=json

[profile personal]
region = us-west-2
output = json

If I want to use my default account, I can use the following code:



const AWS = require("aws-sdk");

const credentials = new AWS.SharedIniFileCredentials({ profile: "default" });

AWS.config.credentials = credentials;

What about my code running in Amazon EC2, AWS Lambda?

I have 3 words for you: "Use IAM roles".

If you have your code running in a Docker container on an Amazon EC2 instance, then understand that every single process on the system has access to IAM roles and your container will assume that role without you having to specify it.

Conclusion

For my development environment, I have found the latter approach of using AWS profiles and using them to access credentials in code better than having to export it. The code is much cleaner and doesn't change if I rotate my keys. All I need to do is to run aws configure on my developer workstation and be done with it. Also, it saves me from the anti-pattern of hard-coding credentials in my code. However, this approach means that you may have to change the code or write conditional code (rarely a good practice) or use environment specific files for your non-development environments where the profile may or may not exist. If you run into such a decision making process, using the environment variable approach is the way to go.