AWS S3 File Upload via Amazon API Gateway

Introduction

Amazon Simple Storage Service (Amazon S3) is a cloud object storage service that offers industry-leading scalability, data availability, security, and performance. You can use Amazon S3 to store and retrieve any amount of data from anywhere on the web.

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. You can use API Gateway to create RESTful APIs and WebSocket APIs that enable real-time two-way communication applications.

Of course, there are multiple ways to upload file to AWS S3 bucket, however, in this section, we will discuss on how to create an API Gateway endpoint for uploading files to Amazon S3. We will also go through on how to test the API Gateway endpoint. I assume that you have some basic knowledge of AWS services and tools, such as IAM, S3, and API Gateway.

Tool(s)/Framework(s)

Cloud Provider  — Amazon Web Service(AWS)

AWS Services  — API Gateway, Simple Storage Service(S3), Identity & Access Management(IAM)

High-Level Design

Let me start with high level design & what we are trying to achieve here. First, we will create s3 bucket & then we will configure AWS API Gateway using which we should be able to upload any types of file to S3 bucket.

S3 File Upload Via API Gateway

Implementation

→ S3 Bucket Creation & Configuration

• Let’s navigate to AWS console & head towards S3.

AWS S3 Console Navigation

• Click on “Create bucket” button

S3 Bucket — Create

• Enter only minimal required details & keep all other attributes as it is without any change. Bucket name  — enter valid bucket name, for Ex: “apigateways3bucket” AWS Region  — choose appropriate AWS region in which you want to create bucket

S3 Bucket — Configuraion

• Click on “Create bucket” button which will result in bucket to be created.

• we can verify that bucket has been created from S3 dashboard as shown below.

S3 Bucket — After creation on S3 Dashboard

Now that we have created S3 bucket, Let’s move on to create & configure API Gateway.

→ API Gateway Creation/Configuration

• Navigate to API Gateway from AWS Console

• Click on “Create API” button

API Gateway — Create API

• Choose “Rest API” & click on “Build”

API Gateway — Build

• Now, let’s enter the required details & click on “Create API” button.
• “Choose the protocol” — Choose “REST”
• “Create new API” — Choose “New API”
• Enter “API name” as “s3upload” & choose “Endpoint Type” as “Regional” as shown below under “Settings” section.

API Gateway — Settings

• We can see that API Gateway resource has been created successfully

API Gateway — Resources

• Also, we can navigate back to the API Gateway dashboard & can see that new API Gateway with name “S3 Upload” is created.

API Gateway — Dashboard

• Now, we have to configure resources with request path where
• Under “Resources” panel of API page, select /.
• Choose “Create Resource” from the “Actions” menu.

• Now, we will create two resources where each resources signifies the S3 bucket & the object.
• Create a resource under “/” (root) with “Resource Name” as “folder” and “Resource Path” as “{folder}” & click on “Create Resource”
• Create another resource under “/{folder}” (root) with “Resource Name” as “object” and “Resource Path” as “{object}” & click on “Create Resource”

API Gateway — Create Resource

• Once these resources are created which would look something like this.

API Gateway — Resources

• Once resources are created, now let’s create HTTP methods which will help us to upload & download the files to/from S3.
• Select “/{object}” from “Resources” & choose “Create Method”

API Integration — Create Metod

• Now, from Actions dropdown, choose ‘ Put ’ method & provide required configuration as given & shown below.
• Integration Type — AWS Service
• AWS Region — Choose the appropriate region
• Execution Rolle — Choose execution role created earlier
• Under “URL Path Parameters” section, add below parameters.
  1. Name — bucket, Mapped From — method.request.path.folder
  2. Name — key, Mapped From — method.request.path.object

API Integration — Put method

• Now, from Actions dropdown, choose ‘ Get ’ method & provide required configuration as given & shown below.
• Integration Type — AWS Service
• AWS Region — Choose the appropriate region
• Execution Rolle — Choose execution role created earlier
• Under “URL Path Parameters” section, add below parameters.
  1. Name — bucket, Mapped From — method.request.path.folder
  2. Name — key, Mapped From — method.request.path.object

API Integration — Get Configuration

• Let’s setup binary media type to allow file upload/download via HTTP request. Click on “Add Binary Media Type” & enter “*/*”

API Integration — Binary Media Type

Testing

Now, you might be thinking so much of configuration but believe me, you will really enjoy & may explore other functionalities on the way which may excite you more & more.

Huuhhh, enough of configuration now, it’s time for testing :)

Let’s navigate to the “Put” method which would looks something like shown below. Let’s click on “Test” link.

API Integration — Test

Enter the required input parameters

{folder} — apigateways3bucket (which we have created ealier)

{object} — testfile (object name by which object would be created under S3 bucket)

Request Body — enter the saple text as file content here

API Integration — Test Inputs Parameters

Once all parameters are set, we are all set to click on “Test” button

API Integration — Test

Now, navigate to the AWS S3 dashboard & check the bucket. it will have the file uploaded in previous step.

AWS S3 — verification after upload

Hurrrrayyy, finally we were able to upload file to S3 using AWS API Gateway.

Conclusion

In this section, we have learned how to create an API Gateway endpoint for uploading files to Amazon S3. We have also learned how to test the endpoint. By using this approach, we can leverage the benefits of both Amazon S3 and API Gateway, such as scalability, availability, security, performance, and ease of development. We can also avoid the limitations of uploading files directly to API Gateway, such as the payload size and encoding restrictions. This way, we can create a robust and efficient solution for uploading files to the cloud.

I hope you have enjoyed this tutorial and found it useful. If you have any questions or feedback, please feel free to connect.

If this post was helpful, please clap for few times or follow to show your support which keeps me constantly motivated to share my learnings.

Learning, sharing & growing together.