AWS DataSync: The Smart Way to Migrate Data to AWS

Move large amounts of data into AWS quickly, securely, and without writing your own copy scripts. Stop nine in the AWS Hidden Gems series.

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

AWS Hidden Gems covers those underrated services you shouldn't ignore. Each article picks one, then explains why it exists, what it does, where it fits, and how to set it up from the console. Know the four basics above and you can follow along. Everything else gets explained as it comes up.

Today's service: AWS DataSync

Moving a few files to AWS is easy. Moving terabytes of files from an office server into S3, reliably and without days of babysitting, is not. DataSync is the managed service built for exactly that: fast, secure, repeatable data transfer into and within AWS.

Why does this service exist?

The obvious way to move a lot of data is a script with a tool like rsync or a copy command. It works for small jobs. At scale it falls apart: it is slow because it copies one thing at a time, it has no built in verification that everything arrived intact, it does not easily pick up only what changed since last time, and when it fails partway you are left sorting out what actually transferred.

DataSync is purpose built for bulk transfer. It copies many files in parallel so it is much faster, verifies data end to end, transfers only changes on repeat runs, encrypts data in transit, and can run on a schedule. It turns a fragile scripting job into a managed task you can trust.

What is AWS DataSync?

DataSync is a managed data transfer service. It moves file and object data between storage locations, handling the speed, security, and reliability for you.

It moves data in these directions:

• From on premises storage, meaning storage in your own data center or office, into AWS
• Between AWS storage services, such as from one S3 bucket to another or into EFS
• To and from other cloud providers For on premises transfers, you run a DataSync agent, which is a small virtual machine you deploy on your own network. It reads the source data and sends it securely to AWS. For transfers that are already inside AWS, no agent is needed. Either way, you define a task that names a source, a destination, and options like verification and scheduling, then run it.

A real world problem

A company is closing its on site file server and moving everything to the cloud. There are millions of files and several terabytes, built up over years.

They started with a copy script over the weekend. It was slow, it choked partway, and they could not tell which files had made it. With a hard cutover date coming, that approach was too risky.

DataSync handles the move cleanly. It transfers the files in parallel, verifies each one arrived correctly, and on later runs copies only what changed, so the final cutover is a quick sync of recent edits rather than copying everything over again. The migration stops being a gamble.

Real world use cases

• Companies migrating file servers and network storage into S3, EFS, or FSx
• Teams moving large media, backup, or archive datasets into cloud storage
• Regular replication of on premises data into AWS for backup or disaster recovery
• Copying large datasets between AWS regions or accounts
• Feeding on premises data into AWS for analytics and machine learning
• Ongoing scheduled transfers that keep a cloud copy in sync with a source The pattern is moving a lot of data reliably, once or on a schedule, without hand built tooling.

Where it fits in AWS

For an on premises source, the DataSync agent runs on your network, reads your storage, and transfers data over the internet or a direct connection to the DataSync service, which writes it to your chosen AWS storage. For AWS to AWS transfers, DataSync talks to both storage services directly. You watch progress and results in CloudWatch, the AWS monitoring service.

flowchart LR
    A[On premises file server] --> B[DataSync agent]
    B -->|Secure transfer| C[DataSync service]
    C --> D[S3, EFS, or FSx]
    C -->|Progress and logs| E[CloudWatch]

DataSync is the mover. You tell it where data is and where it should go, and it handles getting it there safely.

How the workflow runs

If your source is on premises, you first deploy the agent and activate it. Then you create two locations, one for the source and one for the destination, such as an S3 bucket. You create a task that joins them and sets options like whether to verify data and whether to run on a schedule. You start the task, and DataSync copies the data in parallel, verifying as it goes. On later runs, it detects and transfers only what changed.

flowchart TD
    A[Deploy agent for on premises source] --> B[Create source and destination locations]
    B --> C[Create a task with options]
    C --> D[Run the task]
    D --> E[DataSync copies in parallel and verifies]
    E --> F[Later runs transfer only changes]

Setting it up in the AWS Console

This walkthrough copies data from one S3 bucket to another, which needs no agent and is the simplest way to see DataSync work. Have two S3 buckets ready, a source with some files and an empty destination.

1. Sign in to the AWS Console, search for DataSync, and open it. Check the region in the top right corner.
2. In the left menu choose Tasks, then click Create task.
3. For the source location, choose Create a new location, set the location type to Amazon S3, and select your source bucket. Let the console create the IAM role that allows DataSync to read that bucket.
4. For the destination location, again create a new location of type Amazon S3 and select your destination bucket, letting the console create the IAM role for writing to it.
5. Set task options. Leave verification on so DataSync checks the copied data, and leave the schedule off to run it manually this time. Give the task a name and create it.
6. Open the task and click Start. DataSync begins copying, and the task page shows files transferred and bytes moved as it works.
7. When the task finishes, open your destination bucket and confirm the files are there. Check the task's history tab to see that the transfer was verified and complete. Common mistakes: a task that fails on permissions means one of the IAM roles cannot read the source or write the destination, so check both roles. For on premises sources, if the agent will not activate, it usually cannot reach the DataSync service over the network, so check the agent's connectivity.

Using it from code

Once a task exists, you can trigger it from code, which is useful for scheduling transfers from your own systems. This starts a run of an existing task.

import boto3

datasync = boto3.client("datasync")

response = datasync.start_task_execution(
    TaskArn="arn:aws:datasync:us-east-1:123456789012:task/task-0abc123",
)

print("Started transfer:", response["TaskExecutionArn"])

You can also create locations and tasks in code with calls like create_location_s3 and create_task, so an entire transfer setup can be scripted and repeated.

Pricing

Item	Detail
Data transferred	A flat rate per GB copied by DataSync
Rate	About $0.0125 per GB (US)
Landing storage	You pay normal rates for the S3, EFS, or FSx you copy into
Other transfer costs	Standard network and cross region data charges may apply
Free tier	None specific to DataSync, but the per GB rate is low

The AWS data transfer family

AWS Data Transfer and Migration
├── DataSync         fast online transfer into and within AWS
├── Snowball         physical devices for huge offline transfers
├── Transfer Family  managed SFTP and FTP into S3 and EFS
├── Storage Gateway  hybrid storage linking on premises and AWS
└── DMS              database migration between engines

The main choice depends on size and connection. DataSync is right for online transfers over a decent network connection. Snowball is for when you have so much data, or such a poor connection, that shipping physical devices is faster. DMS is the one for moving databases specifically.

Wrapping up

DataSync replaces fragile copy scripts with a managed transfer that is fast, verified, and incremental. Point it at a source and a destination, and it handles moving your data safely, once or on a schedule. Next time a migration or a recurring data copy comes up, you know the tool that makes it routine.

Series progress

You are on stop nine of AWS Hidden Gems.

1. AWS Elemental MediaConvert
2. Amazon IVS
3. Amazon Rekognition
4. Amazon Personalize
5. AWS AppSync
6. Amazon Timestream
7. Amazon Textract
8. Amazon Kendra
9. AWS DataSync (you are here)
10. AWS IoT Core Next up is the final stop, AWS IoT Core, which connects and secures millions of devices and routes their data into AWS.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at [e-mail].