DEV Community

Cover image for Move files from SSH server to S3 bucket
Avinash Dalvi for AWS Community Builders

Posted on • Updated on • Originally published at

Move files from SSH server to S3 bucket

Hello, devs

This blog going to explain the steps to move files or folders from the SSH server ( where AWS credentials are not configured ) to the AWS S3 bucket easily without missing any files or folders.

I was working on moving away from a legacy server to S3 bucket hosting for static file hosting. Being legacy there was many files and folder was there and the legacy server was not having AWS access enabled. Due to the limitation of storage cannot run the zip command to compress and download files. Later those files and folders have to move to the S3 bucket.

While googling found good solutions to tackle this problem.

Let's follow below steps:

Considering you have access to the SSH server and S3 bucket AWS credentials.

  1. Need to use rsync command. There is SCP the command is also available but if the internet interrupts you have to start again. And FileZilla took more time than the terminal command. sshdirectorypath will be SSH server directory from where you have to get files and folders and it will do all recursively. Same /localmachine/directory/ your local machine directory.
> rsync -avz -e ssh username@hostname:/sshdirectorypath/ /localmachine/directory/
receiving file list ... done
Enter fullscreen mode Exit fullscreen mode
  1. If the SSH server is already enabled AWS credentials like secrete key and access key under ~/.aws/credential then can skip the first step.

  2. Running command to copy the local backup to the S3 bucket.

s3 cp ./ s3://bucketname/ --recursive
or if you have to select specific folder inside bucket then 
s3 cp ./ s3://bucketname/folder/folder --recursive
Enter fullscreen mode Exit fullscreen mode

Benefits of using rsync command:

  • is no data leakage will happen

  • even if the internet interruption will resume where it last stop

  • faster and it lists the first contents and copies.

I hope this blog helps you to learn. Feel free to reach out to me on my Twitter handle @AvinashDalvi_ or comment on the blog.

Keep learning and keep sharing!

References :

Top comments (4)

ravavyr profile image

For anyone curious. If you're using AWS [since you have S3] , i'm gonna wager your server is an EC2 instance....and did you know you can just mount that S3 bucket to your instance so your files that you uploaded to a folder, you could just put them in that S3 folder instead and it's like magic.

Here's an article someone posted here about it:

avinashdalvi_ profile image
Avinash Dalvi

Yes we can do that. Some of use case as legacy machine that provision will not be there this blog for particular use case. Thanks for sharing this article.

Sloan, the sloth mascot
Comment deleted
avinashdalvi_ profile image
Avinash Dalvi