loading...

AWS S3 CLI docker for the impatient (also wasabi)

jaeyson profile image Jaeyson Anthony Y. ・3 min read

What's the problem with AWS cli guide (it has good docs anyway)? well, nothing really complex unless you really want to use it asap. This is more of a basic setup and quick run using docker.

0. Things needed: Docker and AWS S3 creds (access key id/secret access key)

There's a guide for linux distro, and a post-installation steps to run Docker as non-root

1. Pull Docker image

docker pull amazon/aws-cli
Enter fullscreen mode Exit fullscreen mode

2. Make credentials

Create ~/.aws dir if not exists, then inside that dir create a file credentials and config:

~/.aws/credentials

[default]
aws_access_key_id=fffff3333ff55555GGGG
aws_secret_access_key=kkkkkkkkkkkkkkkeeeeeeeeeeeeeyyyyyyyyyyyy
Enter fullscreen mode Exit fullscreen mode

~/.aws/config

[default]
region=us-west-2
output=json
Enter fullscreen mode Exit fullscreen mode

Or use an interactive setup

# interactive setup for creds
docker run --rm -it -v ~/.aws:/root/.aws amazon/aws-cli configure
Enter fullscreen mode Exit fullscreen mode

Check creds:

docker run --rm -it -v ~/.aws:/root/.aws amazon/aws-cli configure
Enter fullscreen mode Exit fullscreen mode

3. cp copy, sync and ls list files

3.a. cp copy command

$ aws s3 cp <source> <target> [--options]

source could mean your files from your computer or from S3

Add --recursive option if dir:

# upload a local file hyena.png (current dir) to mammals dir
docker run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  cp hyena.png \
  s3://animal-bucket/mammals
Enter fullscreen mode Exit fullscreen mode
# download lynx.png from s3 to current dir
docker run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  cp s3://animal-bucket/mammals/lynx.png ./
Enter fullscreen mode Exit fullscreen mode
# upload all contents from reptiles dir (local) to s3
docker run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  cp reptiles \
  s3://animal-bucket/reptiles \
  --recursive
Enter fullscreen mode Exit fullscreen mode

3.b ls command

# list all files in reptiles dir
docker run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  ls s3://animal-bucket/reptiles
Enter fullscreen mode Exit fullscreen mode

3.c sync command

aws s3 sync <source> <target> [--options]

Using cp will require the --recursive option to copy multiple files in a dir, while sync command will, by default, copy a whole directory. It will only copy new/modified files.

Add --delete option to remove <targe> files that aren't in <source> dir.

# using new-reptiles folder as basis
docker run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  sync new-reptiles s3://animal-bucket/old-reptiles
Enter fullscreen mode Exit fullscreen mode

The rest of the commands can be found in their docs.

4. Using another cloud storage (ie Wasabi)

Same thing with step 2:

~/.aws/credentials

[default]
aws_access_key_id=fffff3333ff55555GGGG
aws_secret_access_key=kkkkkkkkkkkkkkkeeeeeeeeeeeeeyyyyyyyyyyyy
[wasabi]
aws_access_key_id=wwwaaasssaaabbbiiiii
aws_secret_access_key=wwwwwwwaaaaaassssssaaaaaabbbbbbbiiiiiiii
Enter fullscreen mode Exit fullscreen mode

~/.aws/config

[default]
region=us-west-2
output=json
[wasabi]
region=us-east-1
output=table
Enter fullscreen mode Exit fullscreen mode

Or use an interactive setup. Notice I added --profile <PROFILE_NAME> option:

# interactive setup for creds
docker run --rm -it -v ~/.aws:/root/.aws amazon/aws-cli configure --profile wasabi
Enter fullscreen mode Exit fullscreen mode

Check creds:

docker run --rm -it -v ~/.aws:/root/.aws amazon/aws-cli configure --profile wasabi
Enter fullscreen mode Exit fullscreen mode

Doing some commands (with --profile and --endpoint-url):

docker run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli s3 \
  ls s3://wasabi-bucket \
  --profile wasabi \
  --endpoint-url=https://s3.us-west-2.wasabisys.com
Enter fullscreen mode Exit fullscreen mode

Shorten the command with alias:

alias aws=$(docker run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli)
Enter fullscreen mode Exit fullscreen mode

Discussion

pic
Editor guide