In the world of modern technology, secure and efficient file transfer is a critical necessity for many organizations. AWS Transfer Family, a managed secure file transfer solution offered by Amazon Web Services, allows file transfers using standard protocols such as SFTP, FTPS, FTP and AS2. When combined with Golang, a programming language known for its efficiency and ease of use, a powerful combination is created to manage and automate file transfers effectively.
In this article, we will explore how to integrate AWS Transfer Family with Golang to optimize your file transfer processes. From configuring AWS Transfer Family to implementing an SFTP client with Golang, I will guide you step by step so you can make the most of these technologies.
What is AWS Transfer Family?
AWS Transfer Family is a secure, fully managed service that allows for the transfer of files into and out of AWS storage services such as Amazon S3 and Amazon EFS using protocols like SFTP, AS2, FTPS, and FTP.
Key Features
- Protocols Supported: SFTP, FTPS, FTP, AS2
- AWS Storage Integrations: Amazon S3, Amazon EFS
- Use Cases: Data lakes, internal transfers, data distribution, compliance workflows, supply chain logistics, B2B transactions.
Benefits
- Real-time scalability.
- No need for infrastructure management.
- Leverage native AWS services for data processing and analytics.
- Fully managed, serverless workflow service.
- Pay-as-you-go pricing.
Prerequisites
- AWS Account: An active Amazon Web Services account. If you don’t have one yet, you can sign up on the AWS website.
- Golang: Knowledge of the Go (Golang) programming language.
- Administrative Privileges in AWS.
- AWS Transfer Family: Administrative read and write permissions to configure and manage AWS Transfer Family. This includes the ability to create and manage transfer servers. Note that this service has an associated cost.
- Amazon S3: Administrative permissions to read and write to Amazon S3 buckets.
- AWS IAM: Permissions to manage IAM roles and policies necessary for the integration and security of transfers.
Architectural Design
The proposed architectural design for integrating AWS Transfer Family with Golang consists of the following components, as illustrated in the diagram below:
The API proposed in this article serves as a basic example to demonstrate its fundamental functionality. In a real-world scenario, the API can be more complex and include additional features. Its primary function is to upload files to the AWS Transfer Family SFTP server and store them in an S3 bucket, whether for backup purposes or integration with other systems. Use cases vary and depend on the needs of each organization. Here are some additional use case examples: AWS Transfer Family Customers.
Furthermore, the API proposed in this article can be deployed to the AWS cloud using services such as AWS Elastic Beanstalk or AWS Fargate to achieve greater scalability and availability, depending on the specific requirements of each organization.
Let’s get started!
Implementation
Generate public and private key
Generate a public and private key pair for connecting to the AWS Transfer Family SFTP server. This can be done locally on our machine using the following command.
$ ssh-keygen -t rsa -b 4096 -f <PATH>/aws_example_ssh_key
- aws_example_ssh_key: Key name
- PATH: Directory where the key will be saved
- rsa: Encryption algorithm
- 4096: Key size
- f: Output file name
ssh-keygen -t rsa -b 4096 -f <PATH>/aws_example_ssh_key
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in <PATH>/aws_example_ssh_key
Your public key has been saved in <PATH>/aws_example_ssh_key.pub
The key fingerprint is:
SHA256:P80TXbp10tjdzJLc3KAhtWP9YWQe+A181WvqUQyaTFc rodrigoquispe@0WJMD6M
The key's randomart image is:
+---[RSA 4096]----+
| ..oE+|
| o ** +|
| + O **+|
| * *.^*|
| S o % #|
| . o + =.|
| o = o |
| . o |
| |
+----[SHA256]-----+
- Enter passphrase (empty for no passphrase): Password for the key.
- Enter same passphrase again: Confirm password for the key.
- Your identification has been saved in
<PATH>/aws_example_ssh_key: The private key will be saved in the aws_example_ssh_key file in the specified path. - Your public key has been saved in
<PATH>/aws_example_ssh_key.pub: The public key will be saved in theaws_example_ssh_key.pubfile in the specified path.
Create Transfer Family Service
Log in to the AWS Management Console and select the AWS Transfer Family service and create server.
Select the SFTP (SSH File Transfer Protocol) for secure file transfer and click Next.
Choose an identity provider and select "Service managed."
In the endpoint configuration, select "Publicly accessible" and click Next.
We select "Amazon S3" as the storage option and click Next.
In the "Configure additional details" section, leave the settings at their default values.
Review, click Next and create.
Once the creation is complete, we will have something similar to this. We can see that we do not have any users yet and that it will be starting shortly through a public endpoint and using S3 as storage.
Create a role
Create a new role for the AWS transfer family to s3 service. To do this, go to the IAM service and create new role of the AWS service type and in the use case select transfer and next.
Permission policy, AmazonS3FullAccess to have full access to S3 bucket. But this policy is very permissible and it is not recommended to use it in production. And next.
We proceed to assign the name of the role "role-example-transfer-to-s3", we review and click "Create role".
Create S3 bucket
We proceed to create the s3 bucket, to do this we go to the s3 service and create a new bucket.
We set the name of the bucket, leaving the default configuration and create bucket.
Bucket name: "bucket-example-transfer".
At the end of the creación we will have something like this:
Configure SFTP users for Transfer Family with S3
Go to the AWS Transfer Family service and select the server you created earlier.
- Select the "Users" tab and click on "Add user".
- Enter the username.
- Assign the IAM role you created earlier.
- Set "Policy" none
- In the "Home directory" field, enter
bucket-example-transfer, which is the name of the S3 bucket you created previously. - Click on "Add user".
To follow this article, you will need to provide the public key that was generated at the start aws_example_ssh_key.pub.
To view the content of the public key, use the following command:
$ cat aws_example_ssh_key.pub
The public key looks like this:
ssh-rsa AAAAB3NzaC1yc2EBBAADAQABAAACAQC/qNMJaCz8+I3G71bTFf8Cpe+AAAA0BBBBBBBA6C+/a8fyl9Dw4wjj0huc4ItlX2auKG3skxTYP1ZgO5/7M7xmp2Hf+AAAA0BBBBBBBA6CynWwfwkS5XEIfNtoNWCSeB6ra4anljFdLJH4ZReOM18xxBZCyz2gGA7ePprM0nfpgoQKjp8eDoz6HFcvH4Qynea06W72D+tUWjCRH6bgPBsw/cCfj+VrcqsQXif/X8nViS679+V+AAAA0BBBBBBBA6C+t/5dD3uFPyc8yJn2fNEHB4kfIB9JR5fPIo4dVqPebloPg9vKT2M+AAAA0BBBBBBBA6C/AAAA0BBBBBBBA6C+XLTJSAljgq9JQ4l3vnT4RSsKuK+DX2um5DsN/Zwk9Qwf8JRJey/AAAA0BBBBBBBA6C+/AAAA0BBBBBBBA6CNkVaeDOYSIeGkMBUmmMaeVNeaR967SBLoHNzBfodKKaIpsmW2yHvIitg/YKu2YXKe/DnYU3G8om13uL8qIOB85CugVQ== rodrigoquispe@AAAA0BBBBBBBA6C
This is the key that we need to include in the "SSH Public Keys" field. Ensure that you copy and paste the entire key accurately into this field to enable secure access to the server.
After creating the user, you should see a confirmation screen similar to the one below. This indicates that the user has been successfully set up and is ready to use the SFTP service.
Project structure
Below, we provide a detailed overview of the Golang project structure and include a link to the GitHub repository for reference. This overview will help you understand the organization of the project and how various components are structured and interconnected.
├── cmd
│ └── main.go
├── go.mod
├── go.sum
└── internal
├── app
│ └── service.go
├── domain
│ └── models.go
├── handler
│ └── sftp_handler.go
└── infra
├── config.go
└── sftp_client.go
The dependencies we will be using are Gin and SFTP.
$ go get -u github.com/gin-gonic/gin
$ go get -u github.com/pkg/sftp
Set the following credentials in the config.go file, where the private key and Transfer Family user credentials are located, to connect to the SFTP server.
// config.go
func LoadConfig() *Config {
return &Config{
Host: "<HOST>.server.transfer.us-east-1.amazonaws.com",
Port: "<PORT>",
User: "<USER>",
PrivateKeyPath: "<PATH>/aws_example_ssh_key",
}
}
- Host: The connection endpoint we obtained earlier.
- Port: 22
- User: The username we created earlier.
- PrivateKeyPath: The path to the private key we created earlier.
Also, consider setting the path where the files uploaded to the SFTP server will be saved. In my case, it’s /bucket-example-transfer/rodrigo.
// service.go
func (u *Service) Upload(file *domain.File) error {
return u.sftpClient.UploadFile(file, "/bucket-example-transfer/rodrigo/")
}
Function to create a new SFTP client with the given configuration:
// sftp_client.go
func NewSFTPClient(cfg *Config) (*SFTPClient, error) {
key, err := os.ReadFile(cfg.PrivateKeyPath)
if err != nil {
return nil, err
}
signer, err := ssh.ParsePrivateKey(key)
if err != nil {
return nil, err
}
sshConfig := &ssh.ClientConfig{
User: cfg.User,
Auth: []ssh.AuthMethod{
ssh.PublicKeys(signer),
},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
}
conn, err := ssh.Dial("tcp", cfg.Host+":"+cfg.Port, sshConfig)
if err != nil {
return nil, err
}
client, err := sftp.NewClient(conn)
if err != nil {
return nil, err
}
return &SFTPClient{client: client}, nil
}
Method to upload a file to the SFTP server.
// sftp_client.go
func (s *SFTPClient) UploadFile(file *domain.File, remotePath string) error {
dstFile, err := s.client.OpenFile(remotePath+file.Name, os.O_WRONLY|os.O_CREATE|os.O_TRUNC)
if err != nil {
return err
}
defer dstFile.Close()
_, err = io.Copy(dstFile, file.Content)
return err
}
Run the project using the following command:
$ go run cmd/main.go
Now, we upload a file to the SFTP server using Postman. Then, we navigate to S3 to verify that the file has been uploaded correctly.
The files were successfully uploaded without any issues.
Thank you for reading.
Top comments (0)