Amazon S3 - Web Based Upload Object with POST request and Presigned URL in Python Boto3

#aws #boto3 #s3 #postobject

In this article, I will show you how to generate S3 Presigned URL for HTTP POST request with AWS SDK for Boto3(Python). The unique part of this article is that I will show you how to apply Server Side Encryption with KMS key, Tagging objects, Updating Object Metadata and more with S3 Presigned URL for HTTP POST.

When using S3, there is a scenario about "Broswer-Based Uploads Using HTTP POST". However, it is required to calculate AWS SigV4 Signature to follow the section.

Instead of calculating the signature by you own codes, you can actually use AWS Boto3 SDK with method "generate_presigned_post" to generate S3 PreSigned URL. This is not only saving your time to debug "Signature Mismatch" error with your own codes, you don't have to figure out requirements of crypto modules used by your codes to generate right signature. It will all be handled by AWS SDK.

For example, you owns an S3 bucket in your account. One customer of yours is running a business to allow users of the customer to upload images. The images will be directly uploaded from the customer's website into your S3 bucket. The customer is not familiar with Amazon S3 service and does not own an AWS account, so you need to provide your customer an easy method uploading objects from the customer website directly into your S3 bucket. At the meantime, you don't need to make your bucket public for uploading objects.

This is where S3 Presigned URL is needed. You can generate the S3 Presigned URL for HTTP POST from AWS Lambda function by having these benefits. Then you can provide the S3 Presigned URL with your customer to integrate into the customer's website.

But you might ask this question:

Why are you not using S3 Presigned URL for PutObject API call?

S3 Presigned URL for HTTP POST from broswer-based uploads provides a unique feature. You can define "starts-with" condition in the policy. You and your customers can both have some controls on requirements of the uploaded objects.

For example, you only want your customer to upload text files, so you can use the following "start-with" condition to restrict value of "Content-Type" starting with "plain" in uploading request. The uploading request is created from your customer's website. The value of "Content-Type" request header is set when a file is being uploaded from your customer's website by using your S3 Presigned URL.

["starts-with", "$Content-Type", "plain"],

In document of AWS SDK for Boto3, it did not share much information regarding how to use "Fields" and "Conditions" parameters mentioned at "generate_presigned_post". It took me some time to figure it out, so I added my understanding in the code example.

I hope they will save your time in your code development.

Here is the Python Code Example. Before you test it, you will need to update the constants to match your resources.

import boto3
import requests
from botocore.config import Config

ACCESS_KEY="AKIAIOSFODNN7EXAMPLE"
SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

BUCKET_NAME="example-bucket-name"
OBJECT_NAME="example-key-name"
REGION_LOCATION="ap-southeast-2"

KMS_KEY_ARN="arn:aws:kms:<region>:<account-id>:key/<key-id>"

EXPIRATION_TIME = 60*60 * 12 # 12 hours

TEST_FILE_NAME="/Absolute/Path/To/Local/FileName"

my_config = Config(
    region_name = REGION_LOCATION,
    signature_version = 'v4',
    retries = {
        'max_attempts': 10,
        'mode': 'standard'
    }
)

# define S3 client in ap-southeast-2 region
s3=boto3.client('s3',
                 aws_access_key_id=ACCESS_KEY,
                 aws_secret_access_key=SECRET_ACCESS_KEY,
                 config=my_config)


fields={
    "tagging": "<Tagging><TagSet><Tag><Key>type</Key><Value>test</Value></Tag></TagSet></Tagging>",
    "x-amz-storage-class": "STANDARD_IA",
    "Cache-Control": "max-age=86400",
    "success_action_status": "200",
    "x-amz-server-side-encryption": "aws:kms",
    "x-amz-server-side-encryption-aws-kms-key-id": KMS_KEY_ARN,
    "x-amz-server-side-encryption-bucket-key-enabled": "True"
    # "acl": "public-read"
    }


conditions=[
    {
        "x-amz-storage-class": "STANDARD_IA"
    },
    ["starts-with", "$Content-Type", "plain"],
    {
        "tagging": "<Tagging><TagSet><Tag><Key>type</Key><Value>test</Value></Tag></TagSet></Tagging>"
    },
    {
        "Cache-Control": "max-age=86400"
    },
    {
        "success_action_status": "200"
    },
    {
        "x-amz-server-side-encryption": "aws:kms"
    },
    {
        "x-amz-server-side-encryption-aws-kms-key-id": KMS_KEY_ARN
    },
    # {
    #     "acl": "public-read"
    # },
    {
        "x-amz-server-side-encryption-bucket-key-enabled": "True"
    }
]

# generate S3 Presigned URL for HTTP POST Request
response_presigned_url_post=s3.generate_presigned_post(
    BUCKET_NAME,
    OBJECT_NAME,
    Fields=fields,
    Conditions=conditions,
    ExpiresIn=EXPIRATION_TIME
)
print(response_presigned_url_post)

# User requests.post to test the URL
post_fields=response_presigned_url_post['fields']

# files={'file': open(TEST_FILE_NAME, 'rb')}
# you will see 403 error
#comment the following line and uncomment the second following line, you will see 200 successful

post_fields["Content-Type"]="application/octet-stream"
#post_fields["Content-Type"]="plain/text"

# file key must be the last key in the "files" parameter(form)
# it is defined at https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html#RESTObjectPOST-requests-form-fields

post_fields["file"]=open(TEST_FILE_NAME, 'rb')
print(post_fields)

# making POST Request
response_post_request=requests.post(response_presigned_url_post['url'], files=post_fields)

# print response, by default status code is 204,
# "success_action_status": "200" change it to 200

print(f'Response Status of POST request with S3 Presigned URL: {response_post_request.status_code}')
print(f'Response Headers of POST request with S3 Presigned URL: {response_post_request.headers}')
print(f'Response Body of POST request with S3 Presigned URL: {response_post_request.text}')