How to use S3 services with python

Introduction

Hi, in this article we will discuss how to connect vultr object storage with python.

You can also use (digitalocean, aws or etc..)

In the beginning, What is Object Storage?

Object Storage is Vultr's S3-compatible, web-accessible cloud object storage service for storing and retrieving file objects.
Objects can be accessed privately or publicly, over the web. Use cases for cloud object storage including:

• Share Files like ( images, videos, pdf, ...etc)
• Share software or business files
• Share static website hosting for assets such as images, JavaScript, CSS, or media files

Summary, When you need to handle vast numbers of file objects without having to manage or grow the underlying filesystem, object storage is suitable.

Getting Started

How to create object storage

Select the Objects menu at the top of the Vultr customer portal to navigate to the object storage subscription listing, then use the Add Object Storage form to deploy. After deployment, view your access keys on the subscription listing page.

If you have any question about object storage see this article.

Installing requirements

1. Create new Object storage with Vultr.
2. Download and Install Python from the python.org .
3. Installing boto3 with pip line:
   1. Open cmd for Windows or terminal from MacOS.
   2. And type pip install boto3

What is boto3?

Boto3 is the Software Development Kit (SDK) for Python, which allows Python developers to write software that makes use of services like Vultr Object Storage.

Let's get started:

• Open your favorite code editor.
• Create a new python file with any name.
• Let's code:

First thing first we must get the Acess Key & Secret Key from your object storage page from subscription listing page.

And let us save it in variable:

object_storage_config = {
    'region_name': 'ewr1',
    'endpoint_url': 'https://ewr1.vultrobjects.com',
    'aws_access_key_id': 'Your_access_key',
    'aws_secret_access_key': 'Your_secret_key'
}

How to upload a file to your file dir:

import boto3
client = boto3.client('s3', **object_storage_config)

Now we have connected to vultr s3 object storage

Let us upload a file to object storage but first we must create a new bucket:

Objects are stored in buckets. Before you can begin uploading objects, you need to create at least one bucket. Buckets are similar to folders on a file system.
Know more in this article article.

Now we create a new bucket:

client.create_bucket(Bucket="bucket_name")

Now upload file with boto3:

client.upload_file("file_name", "bucket_name", "object_name.extention")

Example code for the above:

import boto3

object_storage_config = {
    'region_name': 'ewr1',
    'endpoint_url': 'https://ewr1.vultrobjects.com',
    'aws_access_key_id': 'Your_access_key',
    'aws_secret_access_key': 'Your_secret_key'
}

client = boto3.client('s3', **object_storage_config)
client.create_bucket(Bucket="uploaded_images")
client.upload_file("./images/logo.png", "uploaded_images", "logo.png")

What about Multipart files

We can upload a multipart file with function upload_fileobj.

client.upload_fileobj(file, "bucket_name", "object_name.extention")

Example:

import boto3
object_storage_config = {
    'region_name': 'ewr1',
    'endpoint_url': 'https://ewr1.vultrobjects.com',
    'aws_access_key_id': 'Your_access_key',
    'aws_secret_access_key': 'Your_secret_key'
}

client = boto3.client('s3', **object_storage_config)
# WE ALREADY CREATED BUCKET
with open("FILE_NAME", "rb") as file:
    client.upload_fileobj(file, "uploaded_images", "logo.png")

Now we have uploaded a file to s3 storage but we have a problem. The problem is that the files we have uploaded are in private mode and there is no one can access them without an access key.
We have two options to solve this first to make the files in the public mode when we upload it and second we can create a Presigned URL.

What is Presigned URLs?

Vultr users who can access the object will generate a pre-signed URL. The generated URL will be delivered to unauthorized users. The pre-signed URL can be entered in a browser or used in an HTML web page or application. Credentials that use a pre-signed URL are the credentials of the Vultr user who generated the URL.
pre-signed URLs are valid for the limited time specified when the URL was generated.

response = client.generate_presigned_url(
    'get_object', 
    Params={'Bucket': "bucket_name", 'Key': "object_name"},
    ExpiresIn=expiration
)

Expiration: Time in seconds for the presigned URL to remain valid

Example:

import boto3
object_storage_config = {
    'region_name': 'ewr1',
    'endpoint_url': 'https://ewr1.vultrobjects.com',
    'aws_access_key_id': 'Your_access_key',
    'aws_secret_access_key': 'Your_secret_key'
}

client = boto3.client('s3', **object_storage_config)
response = client.generate_presigned_url('get_object', Params={'Bucket': "uploaded_images", 'Key': "logo.png"}, ExpiresIn=3600)
print(response)

Option two is to upload the file in public mode

We can do that by adding ExtraArgs to upload_file or upload_fileobj function.

Example:

import boto3

object_storage_config = {
        'region_name': 'ewr1',
        'endpoint_url': 'https://ewr1.vultrobjects.com',
        'aws_access_key_id': 'Your_access_key',
        'aws_secret_access_key': 'Your_secret_key'
}

client = boto3.client('s3', **object_storage_config)
client.create_bucket(Bucket="uploaded_images")
client.upload_file("./images/logo.png", "uploaded_images", "logo2.png", ExtraArgs={"ACL": "public-read"})