Using Bash to upload files to AWS S3

This blog post describes how to create a Bash script which can be used from the command line to upload files on to AWS S3. In order to follow along, you will need:

• An AWS Account
• The AWS CLI Downloaded and installed on your machine
• The AWS CLI configured with your credentials
• Git installed on your machine

For guidance on how to download and configure AWS CLI for Windows, see my post here.

Use cases

We will satisfy the following use cases in this script:

• Parse command line arguments $1, $2, to take in the file and bucket names.
• Validate the input.
• Provide error messages for incorrect or missing inputs.
• Before uploading check if the file exists locally, if not provide feedback.
• Before uploading, check if the bucket exists, if not create it, provide feedback.
• Before uploading, check if the file exists in the bucket
• If there is an error, capture it and display it to the user.

Create a bash file

• Open a git bash prompt and navigate to the folder you will be working in.
• Create a bash script using the touch command.

touch upload_to_s3.sh

• Use the chmod command inside a git bash terminal to give the script executable permissions.

chmod+x upload_to_s3.sh

• Open up the script in an editor. I am using Visual Studio Code.
• All bash scripts start with a shebang. Add one to the top of the script.

#!/bin/bash

Parse command line arguments

In bash arguments are represented by $1 and $2. Let's set two variables, the name of the file we wish to upload, and the name of the S3 bucket we want to upload to, to $1 and $2 respectively.

!#/bin/bash

FILENAME=$1
S3BUcKETNAME=$2

echo $FILENAME
echo $S3BUCKETNAME

Run the script with the following command, remember to pass in arguments, ./upload_to_s3.sh arg1 arg2

Create a test file to upload. touch testfile.txt

Check if inputs are valid

Add an if statement to check if the two inputs are valid.

if [[ -z $FILENAME ]]
then
    echo Please provide a filename.
fi

if [[ -z $S3BUCKETNAME ]]
then
    echo Please provide a bucketname.
fi

Run the script without inputs to check the conditions work. You should get these responses:

It's a good idea to test the script with correct and incorrect inputs at each stage. This will ensure you catch any errors.

If a filename is provided, we want to check if the file exists. If it does not, return the error message.

if [[ ! -f $FILENAME ]]
then
    echo File not found!
fi

Check if the bucket exists, if not, create it.

Next we will check if the bucket already exists or not, and provide feedback depending on the result.

• If the bucket does not exist and the name of the bucket is the appropriate length: create the bucket.
• If the bucket exists but we do not own it: return an error message.
• If the bucket does not exist but the proposed name is not within the required parameters: return an error message.
• If the bucket already exists and we do own it: return a message.

# check if the bucket exists
bucketstatus=$(aws s3api head-bucket --bucket "${S3BUCKETNAME}" 2>&1 )
# search the bucketstatus result for the Not Found result, if the bucket is
# indeed not found, echo that as a response, then create the bucket

if echo $bucketstatus | grep -q "Not Found";
then
    echo Bucket not found.
    # this command creates the bucket
    aws s3 mb s3://$S3BUCKETNAME

elif echo $bucketstatus | grep -q "Forbidden";
then
    echo Bucket exists but not owned.
elif echo $bucketstatus | grep "Bad Request";
then
    echo Bucket name is less than 3 or greater than 63 characters.
else
    echo Bucket already exists.
fi

Create a function to upload the file

Making this a function will allow us to reuse this piece of code without repeating it. This will also check if the file upload was successful, and return and error message if it fails.

UPLOAD() {
    echo Uploading file.
    aws s3 cp $FILENAME s3://$S3BUCKETNAME
    # this checks if the previous command is equal to 0 or not, in bash a 0 return indicates a successful completion of command, while a non-zero return indicates an error occured.
    if [[ $? -eq 0 ]]
    then
        echo File uploaded successfully
        exit 0
    else
        echo An error occured. 
        exit 1
    fi
}

Check if the file already exists inside the bucket

Check if the file already exists inside the bucket. If it does, offer an option to replace the existing file, respond appropriately based on the user's input.

# this checks if the file exists inside the bucket
FILE_EXISTS=$(aws s3api head-object --bucket $S3BUCKETNAME --key $FILENAME)
# if the file does exist
if [[ !$FILE_EXISTS ]];
then
    echo File already exists
    echo Do you want replace the file in the bucket with this version? Y/N?
    # take in the user's answer and if it is yes, then upload the new version of the file.
    read ANSWER
    if [[ $ANSWER = Y ]]
    then
        UPLOAD $FILENAME $S3BUCKETNAME
    else
        exit
    fi
# if the file does not already exist in the bucket, then upload the file.
else
    echo Uploading
    UPLOAD $FILENAME $S3BUCKETNAME  
fi

Test the script

Run the script ./bash-uploader-cli.sh filename bucketname to test it. Use the following command to check if the file has been uploaded to the correct bucket.

aws s3 ls s3://bucketname

You can use this command to delete the bucket and files inside them after testing.

aws s3 rb --force s3://bucketname

Run the script again with the same file, or a different bucket name to ensure it responds appropriately.

You can find the script at this GitHub repository.

Thanks for reading this guide, I hope it was helpful.