AWS Basics - Beginner's Guide to S3

Accessing S3

S3 stands for Simple Storage Service. It's a storage service provided by AWS.

S3 is object-based storage, i.e, it manages data as objects.

S3 allows unlimited storage.

S3 objects can have a maximum size of up to 5 TB.

S3 stores files in buckets which are globally unique.

An S3 URL looks as follows:

https://<bucket-name>.s3.<region>.amazonaws.com/<key-name>

aws s3 ls

# Unable to locate credentials. You can configure credentials 
# by running "aws configure".

# Configure aws
aws configure

AWS Access Key ID [None]: 
AWS Secret Access Key [None]: 
Default region name [None]: 
Default output format [None]: json

# List all configs
aws configure list
# OUTPUT
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************W99N shared-credentials-file
secret_key     ****************123Q shared-credentials-file
    region               ap-south-1      config-file    ~/.aws/config

# Open the config file
cat ~/.aws/config
# OUTPUT
[default]
output = json
region = ap-south-1

# Create S3 bucket
aws s3 mb s3://testbckt-mp

# Copy content to S3
aws s3 cp hello.txt s3://testbckt-mp

# List content in S3 bucket
aws s3 ls s3://testbckt-mp

Pagination

How can we control the number of items included in the output of a CLI command to list objects in a bucket?

Default page size: 1000

When the command aws s3api list-objects --bucket <bucket> is run with say 2500 objects in the bucket, it will result in 3 API calls (each API call is limited to 1000 objects) to get the complete output.

However, using the above command on a large number of resources may result in timeout error or too many results returned as default page size of 1000 would be too high.

SOLUTION:
The page-size argument:

aws s3api list-objects --bucket <bucket> --page-size <size> : The API still fetches the full list but with larger API calls in the background with smaller number of items per API call.

How about just getting a limited number of items?

We can use the max-items argument for such cases.

aws s3api list-objects --bucket <bucket> --max-items <max> : just return the first max items

CLI Tips

• Least privilege: always give your users the minimum amount of access required
• Create groups: assign users to groups
• Secret access key: Only available once, if change is required -> delete the key pair and regenerate it, followed by aws configure
• Use separate keys per developer