DEV Community

Cover image for Migrate Data into S3 using AWS Transfer Family
AWS Community Builders profile image Robina Mirbahar
Robina Mirbahar for AWS Community Builders

Posted on • Edited on

Migrate Data into S3 using AWS Transfer Family

#aws #migration #s3 #opensource

What is AWS Transfer Family?

AWS Transfer Family is a fully managed AWS service that you can use to transfer files into and out of Amazon Simple Storage Service (Amazon S3) storage or Amazon Elastic File System (Amazon EFS) file systems over the following protocols:

  • Secure Shell (SSH)
  • File Transfer Protocol (SFTP)
  • File Transfer Protocol (SFTP)
  • File Transfer Protocol Secure (FTPS)
  • File Transfer Protocol (FTP)

Read more: AWS Transfer Family

Benefits Of AWS Transfer Family?

  • It is a fully managed, serverless File Transfer Workflow service that makes it simple to set up, execute, automate, and monitor file processing utilizing AWS Transfer Family.
  • With your data safely stored in Amazon S3, you can utilize native AWS services for processing, analytics, reporting, auditing, and archiving.
  • There are no upfront costs, and you pay only for the use of the service.
  • You don't need to modify your applications or run any file transfer protocol infrastructure.

How it works?

AWS Transfer Family Architecture

AWS Transfer Family Architecture

AWS Transfer Family (SFTP, FTP, FTPS) – Step by Step Implementation Process

Setup the prerequisites for AWS transfer for SFTP 

 - S3 bucket
 - EC2 instances (Linux and Windows)
Enter fullscreen mode Exit fullscreen mode

Create an AWS IAM role and policy 

- Edit the policy to provide the S3 access
Enter fullscreen mode Exit fullscreen mode

Create the SFTP server 

 - Create users
 - Create and assign public keys
Enter fullscreen mode Exit fullscreen mode

Test the file transfer from SFTP clients 

- For Linux
- For Windows
Enter fullscreen mode Exit fullscreen mode

What is S3 Bucket?

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Read more: Amazon Simple Storage Service

Create a S3 Bucket 

- Open AWS Console Click on Services
- In the Storage Section, Choose S3
Enter fullscreen mode Exit fullscreen mode

Choose Storage

- Click Create Bucket and Create Your Bucket
- Give Bucket Name (Should be unique Name use 
small alphabets Without Space. hyphens and Periods}
- Choose your Region. (Where user wants to store their data)
 rest settings choose default.
Enter fullscreen mode Exit fullscreen mode

Create Bucket

Bucket Created

Create a VPC 

- Open AWS Console Click on services
- In the Networking & Content Delivery Section, Click on VPC
Enter fullscreen mode Exit fullscreen mode

VpC

VPC configuration
VPC Created

Create Subnets

create subnets

VPc Settings

Create Two EC2 Instances (Linux & Windows) 

- Open AWS Console Click on services 
- In the Compute section, click on EC2
- Click Launch instance
Enter fullscreen mode Exit fullscreen mode

launch instance

Security Group rules for Linux Instance

Security Group Rule Linux

Security Group rules for Windows Instance

Security Group Rule Windows

Create an IAM Role 

- Open AWS Console Click on services 
- In the Security, Identity & Compliance section, click on IAM
Enter fullscreen mode Exit fullscreen mode

Services

Create IAM roles and policies from the IAM console (Choose AWS service as Transfer)

The policy to access the S3 bucket

S3 Bucket Policy

Policy to access CloudWatch Logs


Create two roles and attached the policies with each role (Choose AWS service as Transfer)

Create the SFTP server, Users, Public Keys and Assign the keys 

- Open AWS Console Click on services 
- In the Migration & Transfer section, click on AWS Transfer Family
Enter fullscreen mode Exit fullscreen mode

Click on create Server

Choose Protocols

Choose an Identity Provider

Choose an Endpoint

Choose a Domain

Add User to the Server


Generate the public key from the Linux machine?

Log in to your EC2 machine connect via Putty

Generate the key with the below command 

ssh-keygen -P "" -f key
Enter fullscreen mode Exit fullscreen mode

Transfer the files using WinSCP for Linux

Test the file transfer from the Linux SFTP client

Connect to the server 

- sftp -i key rm-transfer-family-user@
s-c0ca408efe3d4c3e8.server.transfer.ap-southeast-1.amazonaws.com

- put AWS-Transfer-Family-Linux.txt rm-transfer-family-2022 
AWS-Transfer-Family-Linux.txt 
to/rm-transfer-family-2022/rm-transfer-family-user/AWS-Transfer-Family-Linux.txt
Enter fullscreen mode Exit fullscreen mode

Successfully transfer File from Server to Bucket

Now do the same using the Windows Machine

Generate Public key for Windows user


Two Public keys are added

Now transfer the files using WinSCP for Windows

Connection established using WinSCP for Windows

Test the file transfer from SFTP client for Windows

Successfully Tested the file transfer from SFTP client and added to the Bucket

Top comments (2)

Collapse
 
hseritt profile image
Harlin Seritt •

Bookmarking! Thanks for posting!

Collapse
 
robinamirbahar profile image
Robina Mirbahar AWS Community Builders •

Thank you so much