DEV Community: Amanda Quint

Debugging S3's Pre-signed URLs with Boto3 and Python - Dealing with Unexpected Region Behavior

Amanda Quint — Fri, 28 Mar 2025 16:01:31 +0000

Amazon’s Simple Storage Service (S3) is an extremely robust cloud storage solution, and it has a ton of settings and functionality: including the ability to create a “pre-signed” URL that gives others permission to access objects in your bucket for a given time.

This story documents that functionality and an unexpected quirk I ran into while implementing it.

Background

I have a small application that allows a user to upload a file to an S3 bucket that the application owns. This action kicks off a series of processing events on the file.

When a user first initiates this process, a Lambda function hosting a Python API is hit and creates a “Pre-signed” POST URL, which gives the application a few seconds to upload the file.

I originally wrote the code for this in my development environment, which is located in us-east-1 — however, when I deployed the code to a new environment located in us-east-2 the application wouldn’t work, although I was deploying the same code in both places.

After a lot of trial and error (and googling), I finally figured out what was happening.

Reproducing the Issue

To reproduce this, I created the following resources:

testpresignedpostbucket — a private S3 bucket in us-east-1
testS3presignedpostUSEAST1 — a Lambda function deployed in us-east-1 with a role that gives it permission to read and write to S3
testpresignedpostbucketeast2 — a private S3 bucket in us-east-2
testS3presignedpostUSEAST2 — a Lambda function deployed in us-east-2 with a role that gives it permission to read and write to S3

Both Lambdas are running Python 3.12, are set to be invoked using a Function URL, and have the same small snippet of code deployed; the only change being the corresponding bucket name:

import json
import boto3

bucket = 'testpresignedpostbucket' # append "east2" in that region

def lambda_handler(event, context):
    s3_client = boto3.client('s3')
    return s3_client.generate_presigned_post(bucket, 'test.txt', ExpiresIn=60)

This code uses boto3 to generate a pre-signed post, which will pass back a URL and the form fields needed to allow a user to upload an object to the bucket key “test.txt” for the next 60 seconds. I’m using the generate_presigned_post method because users are only writing to the bucket, they never read from it.

Finally, I also created a small text file to try to upload: test.txt , which only contained the words “Hello World!”

Testing the Resources

Note: The resources have all been deleted as of my writing this, so the following curl calls are for examples only.

I hit my Lambda in us-east-1 with the following curl command:

curl --request GET \
  --url https://isyo6u4s6j2mb2lfngzmbdas4u0pzkll.lambda-url.us-east-1.on.aws/

In response, I got the following:

{
 "fields": {
  "signature": "********",
  "AWSAccessKeyId": "********",
  "x-amz-security-token": "********",
  "key": "test.txt",
  "policy": "********=="
 },
 "url": "https:\/\/testpresignedpostbucket.s3.amazonaws.com\/"
}

Based on this response, I wrote the code in my application to use this information to upload my test.txt file:

import requests

def get_presigned_url(function_url):
    response = requests.get(function_url)
    response.raise_for_status()
    upload_file(response.json())

def upload_file(presigned_url_fields):
    payload = {
        "signature": presigned_url_fields["fields"].get("signature"),
        "AWSAccessKeyId": presigned_url_fields["fields"].get("AWSAccessKeyId"),
        "policy": presigned_url_fields["fields"].get("policy"),
        "x-amz-security-token": presigned_url_fields["fields"].get(
            "x-amz-security-token"
        ),
        "key": "test.txt",
    }
    with open("test.txt", "rb") as f:
        response = requests.post(
            presigned_url_fields["url"], data=payload, files={"file": f}
        )
        response.raise_for_status()
        print(response.status_code)

This worked as expected, I got an HTTP 204 in response, and my file ended up where I expected in my bucket:

Test.txt was successfully uploaded to S3

In `us-east-2`

However, as soon as I deployed this code to us-east-2 my Python script started throwing an HTTP 403 Client Error: Forbidden for url. My initial thought was that my bucket permissions were somehow misconfigured, but that didn’t make sense, as I had deployed the same CDK stack in both regions.

Curling the us-east-2 Lambda revealed a different set of keys, even though it was running the same code:

curl --request GET \
  --url https://yyji6kiile2t74orqi6bxppjxq0ybhei.lambda-url.us-east-2.on.aws/

{
 "fields": {
  "x-amz-date": "********",
  "x-amz-signature": "********",
  "x-amz-security-token": "********",
  "key": "test.txt",
  "x-amz-algorithm": "AWS4-HMAC-SHA256",
  "x-amz-credential": "********",
  "policy": "********=="
 },
 "url": "https:\/\/testpresignedpostbucketeast2.s3.amazonaws.com\/"
}

Since this was the same code deployed (via Amazon CDK) in two different regions, I was really confused!

Solution

It turns out that this issue is due to the AWS Signature Version.

AWS S3 uses Signature Version 4 in all regions. Regions created after January 30th, 2014 also default to using Version 4. However, regions created before January 30th, 2014 still support Version 2. See AWS’s documentation on Authenticating Requests for more information.

us-east-1 was AWS’s first region and launched in 2006.
us-east-2 didn’t launch until ten years later in 2016! (After the signature version change date)

Code Change

I found a GitHub issue that explained my problem: Botocore also defaults to using Version 2 while generating pre-signed posts in us-east-1 — so to make this work a simple code change to specify the signature version was needed:

import json
import boto3
from botocore.client import Config

bucket = 'testpresignedpostbucket'

def lambda_handler(event, context):
    s3_client = boto3.client('s3', config=Config(signature_version="s3v4"))
    return s3_client.generate_presigned_post(bucket, 'test.txt', ExpiresIn=60)

This results in the following response, with all the expected fields that we were seeing in us-east-2:

{
 "fields": {
  "x-amz-date": "********",
  "x-amz-signature": "********",
  "x-amz-security-token": "********",
  "key": "test.txt",
  "x-amz-algorithm": "AWS4-HMAC-SHA256",
  "x-amz-credential": "********",
  "policy": "********=="
 },
 "url": "https:\/\/testpresignedpostbucket.s3.amazonaws.com\/"
}

I would also like to note that if I had not been explicit in my Python script, I wouldn’t have run into this issue. However, the boto3 documentation for generate_presigned_post is a little bit vague on the fields dictionary — I could have used a bit more documentation.

def get_presigned_url(url):
    response = requests.get(url)
    response.raise_for_status()
    upload_file(response.json())


def upload_file(presigned_url_fields):
    with open("test.txt", "rb") as f:
        response = requests.post(
            presigned_url_fields["url"],
            data=presigned_url_fields["fields"],
            files={"file": f},
        )
        response.raise_for_status()
        print(response.status_code)

Ultimately, I feel like it’s better to use the newer signature version in both cases.

Another Gotcha - Order of Fields Keys Matter!

While writing this story, I stumbled upon another thing I didn’t expect: the order of the fields seems to matter.

Using Insomnia, I was testing uploading the file to the pre-signed URL and encountered the following 400 Bad Request error:

I received a 400 Bad Request when the fields were out of order

This was fixed by making sure that the key field came before the file:

The same information sent in a different order results in a successful 204 response

This is a minor issue, but it gave me a moment of confusion!

Conclusion

When I deploy code in different regions, I typically expect the code to act the same way. I know that brand-new features may only be available in certain regions, but for heavily used services like Lambda and S3, I wasn’t expecting to find any quirks. It took me a couple of hours to track down what was happening in this case, so I hope this article can perhaps save someone else some time in the future!

This story is also available on Medium.

I enjoy writing about software development, project management, and my journey in the AWS Cloud. If you’d like to read more, please consider following me here on Dev.to or on Medium or LinkedIn.

Fixing a Performance Anti-Pattern with DynamoDB (and Lessons Learned)

Amanda Quint — Wed, 31 Aug 2022 15:14:38 +0000

I’ve recently been working on a personal project where I’ve decided to use and learn more about DynamoDB. While I’ve used Dynamo a bit for small things in the past, the majority of my past experience has been with relational databases, and it’s taken me some time to get my head around working with NoSQL.

What is DynamoDB?

DynamoDB is Amazon Web Services’ fully managed NoSQL database.

Unlike relational databases, which store their data in structured rows and columns, Dynamo data is stored in key-value pairs. This means that the data you store in Dynamo is unstructured compared to what you’d usually see in a relational database. The only schema that you have to define is your table keys: the partition key and the sort key. Beyond that, your data can be schemaless (although your application still has to know how to handle it).

My Mistake

In my project, I’m using Python and accessing Dynamo via the boto3 library. This is running on AWS Lambda. In my code, I have a data model, collection, which I needed to do basic CRUD (Create, Read, Update, Delete) operations on.

My code looked something like this:

Do you see my mistake?

With every database operation I was doing, I was re-initializing the boto3 resource and the Dynamo Table.

It was easy to do (especially with GitHub Copilot basically writing these functions for me), and I didn’t really think about it at the time. However, while reviewing some code, my fiancé, Myles Loffler, pointed out to me that this is actually an anti-pattern when working with boto3 clients, particularly on Lambda.

Note: While I hit this anti-pattern while working with Dynamo, due to the nature of the CRUD operations — you can run into it any time you’re re-initializing the same boto3 resource over and over!

How I Fixed It

Luckily, this was both easy to fix, and fixing it made my code better overall!

I created a new utilities file to handle my database connections and added the following functions:

Then, I refactored my CRUD functions to just use get_dynamo_table instead of using boto3 directly.

You may have noticed the use of global variables in my database utility. While this is often considered a bad practice, I used them based on the Lambda best practice that suggests taking advantage of execution environment reuse.

What Kind of Difference Did It Make?

This made four major differences to my code:

Performance

This is obviously the big one!

To test, I deployed two versions of the application, one with the old code and one with the new, and ran through some of the workflows that resulted in CRUD operations.

Here are some examples of traces through my application without this change applied to it. This action included deleting an item and a query:

And here are traces of the same two actions (delete and query) through the application with this change applied:

Although we’re only talking about fractions of a second, these delays add up over time! The application feels notably faster and more responsive after this change was applied.

Cleanliness

This approach is both easier to read and, in the long term, will definitely be less code in my project. I don’t have to import boto3 and access the environment variables whenever I need to access the database. There is a single function to access Dynamo now, so if I need to change something, I’ll only have to change it in one place.

Testability

The tests are cleaner too. Before, I had to mock boto3 in every location, which was also fairly complicated, as I was mocking the return value of a return value. Now I can just mock get_dynamo_table , which is a lot more straightforward and easy to read.

Reusability

The new functions in utils.db don’t care about the table name at all — it’s retrieved by the get_dynamo_table_name function in settings. As my project grows, this code is reuseable across other Lambdas that may use different Dynamo tables.

As I mentioned above, I’m fairly new to using Amazon’s DynamoDB — and I’m still learning!