How I Migrate DigitalOcean Spaces to S3: 12 Million Objects, Two Hops, Zero Failures

Recently I had to migrate DigitalOcean Spaces to S3 for a fintech platform I run infra for. The source was a Space in an EU region and the destination was S3 in an APAC region: about 2.5 TiB across about 12 million objects. The final hop finished with every object succeeded and 0 failed.

That result is clean. The route to it wasn't. This is the version of the story with the mistakes left in, because the mistakes are where the useful lessons are.

I'm Nazmul Ahasan, and I lead Platform Engineering at NEXT Ventures. I write about AWS, cost and reliability work at nazmul-ahasan.pro.bd, and the business side of this move is in the object storage migration case study.

The shape of the problem

Look at the average object size before anything else: about 220 KB. This isn't a "move a few big files" job. It's a massive-small-file workload, which means the limit is requests per second, not bandwidth. Every object costs you a GET, a PUT, and possibly a HEAD or two. Multiply that by about twelve million.

The second fact is geography. EU to APAC is roughly 250–280 ms round trip. A copy tool running anywhere along that path pays that latency on every small object. You can throw concurrency at it, but you'll hit the source's rate limits long before you beat physics.

So I didn't do a direct copy. I did two hops:

1. Hop 1, same metro: rclone on an EC2 instance in the same EU metro, copying DO Spaces to an S3 staging bucket in <eu-region>. Low latency on both sides.
2. Hop 2, server-side: S3 Batch Operations copying staging to the production bucket in the APAC region. The bytes travel over the AWS backbone and never pass through a machine I manage.

If you take one thing from this post, take that one. Do the slow, chatty part of the transfer where latency is cheap, then let AWS move the bulk between its own regions.

Hop 1: rclone, DO Spaces to S3 in the EU

Config, and my first dumb mistake

Spaces keys come from the DigitalOcean control panel under API → Spaces Keys. The secret is shown exactly once. Copy it straight into your secret store.

The working config:

[dospaces]
type = s3
provider = DigitalOcean
endpoint = <do-eu-region>.digitaloceanspaces.com
region = <do-eu-region>
access_key_id = <SPACES_KEY>
secret_access_key = <SPACES_SECRET>

[s3eu]
type = s3
provider = AWS
env_auth = true
region = <eu-region>

My first version had the Space name baked into the endpoint host, my-space.<do-eu-region>.digitaloceanspaces.com, because that's the URL you see in the DO console. rclone lsd dospaces: came back empty. No error, just nothing. The endpoint should be the regional host, and the bucket goes in the path: dospaces:my-source-space. Five minutes lost, and a reminder that a silent empty result is worse than an error.

A smaller red herring: on the AWS side the instance role was scoped to specific buckets, so rclone lsd s3eu: returned a 403 on ListAllMyBuckets. That's harmless. You don't need to list every bucket in the account. Go straight to s3eu:my-staging-bucket-eu and move on. Don't widen the policy to make a cosmetic error go away.

Tuning: SlowDown, then crawling

My first serious run fired a lot of concurrency at Spaces and got 503 SlowDown back. That's DO telling you to back off. Two things were burning requests: raw parallelism, and rclone doing a HEAD on objects it didn't need to inspect.

Once the 503s stopped it was still crawling at about 150 objects per second. The cause was modtime comparison. To compare modification times, rclone HEADs the object on both sides. Two extra round trips per object for twelve million objects is a lot of waiting. For a one-shot migration into an empty bucket, size is a perfectly good comparison.

I raised the request rate step by step, --tpslimit 300 → 600 → 800, and watched for throttling at each step. This is the command I finished with:

rclone copy dospaces:my-source-space s3eu:my-staging-bucket-eu \
  --size-only \
  --transfers=96 --checkers=32 \
  --tpslimit=800 --tpslimit-burst=100 \
  --s3-no-head-object --s3-no-head \
  --s3-upload-concurrency=2 --s3-chunk-size=5M \
  --s3-no-check-bucket \
  --retries=10 --low-level-retries=20 \
  --stats=1m --stats-one-line \
  --log-file=rclone-hop1.log -v

Why the flags matter:

• --size-only removes the modtime HEADs. It's the biggest single speedup.
• --s3-no-head and --s3-no-head-object skip the HEAD before and after each upload. For small files, those HEADs are a big share of the total work.
• --s3-upload-concurrency=2 and --s3-chunk-size=5M fit objects of about 220 KB. Big multipart settings do nothing here except use memory.
• --s3-no-check-bucket stops rclone from checking or creating the bucket over and over.
• --tpslimit=800 is the number that matters. At 800 TPS I saw zero throttling.

It sustained about 400 objects per second, peaking around 145 MiB/s. Hop 1 took about a day, tuning included.

The six hours I lost

This was my most expensive mistake, and it had nothing to do with clever flags.

The instance started as a c6i.xlarge with 8 GB of RAM. rclone builds directory listings in memory, and a full listing of 12 million objects holds about 7 GB. The first real run ran out of memory and froze the box. Nothing crashed loudly. The process just stopped making progress, and I didn't notice for about six hours.

I also tried --fast-list, which is supposed to cut list calls by fetching everything in one go. On this bucket it sat idle for around 30 minutes building the listing before it copied anything. On a box that's already short of memory, that's the wrong trade.

The fixes:

• Resize to an r6i.4xlarge. Memory-optimised, because the listing is the workload.
• Work per top-level folder instead of the whole bucket at once.
• Monitor long runs. Tail --stats-one-line output, alert if the transferred count stops moving, watch memory. A watch loop on the log would have saved me most of a working day.

My opinion: any copy job expected to run longer than lunch needs a liveness check. "It's probably still going" doesn't count as monitoring.

Verifying hop 1

A full-bucket rclone check hit the same memory wall, so I checked per folder:

rclone check dospaces:my-source-space/Invoices s3eu:my-staging-bucket-eu/Invoices \
  --one-way --size-only \
  --missing-on-dst missing.txt --differ differ.txt

--one-way asks only one question: is everything at the source present at the destination? For a migration, that's the right question.

The gaps were real, and they were all explainable. The application was still live. New uploads under the current month's <YYYY/MM> prefix kept arriving after the bulk pass had listed them. Some files had been regenerated in place with a different size. Both kinds got targeted re-copies:

rclone copy dospaces:my-source-space s3eu:my-staging-bucket-eu \
  --files-from missing.txt --no-traverse --size-only

Feeding the list in with --files-from plus --no-traverse means rclone copies exactly those keys without listing the bucket again. For a sense of scale, the top-ups were a few thousand files in total, mostly certificates and invoices, plus a handful of agreements, photos and dispute files with size differences. This bucket holds user documents like invoices and certificates. "Mostly there" isn't acceptable for data like that.

Hop 2: S3 Batch Operations to migrate DigitalOcean Spaces to S3 in APAC

Once staging matched the source, the rest was meant to be the easy part. It was, eventually.

The IAM role

Batch Operations assumes a role that trusts batchoperations.s3.amazonaws.com:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "batchoperations.s3.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}

Permissions, split by purpose:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "SourceRead", "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectTagging", "s3:GetObjectVersion",
                 "s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::my-staging-bucket-eu", "arn:aws:s3:::my-staging-bucket-eu/*"] },
    { "Sid": "DestWrite", "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:PutObjectTagging", "s3:PutObjectAcl",
                 "s3:GetBucketLocation", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::my-prod-bucket-apac", "arn:aws:s3:::my-prod-bucket-apac/*"] },
    { "Sid": "ManifestAndReports", "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": ["arn:aws:s3:::my-prod-bucket-apac/batch-manifests/*",
                   "arn:aws:s3:::my-prod-bucket-apac/batch-reports/*"] }
  ]
}

Four failures before one success

I tried to be clever with the generated manifest first. The console and API can build the manifest for you from filters, so you don't write a CSV yourself. That's how the day went:

1. Instant failure from the EligibleForReplication filter. That filter only works on versioned buckets, and staging wasn't versioned. I removed it.
2. Missing PutInventoryConfiguration. The auto-generated manifest uses inventory machinery under the hood, and the role needs that permission.
3. 301 PermanentRedirect, 100% failure. An auto-manifest job runs in the source region. Writing cross-region to APAC from an EU job redirected every task.
4. CrossRegionOperationProhibited. When I tried to fix the region, Batch Operations made it clear it can't generate a manifest across regions.

The lesson: if source and destination are in different regions, build the manifest yourself and run the job in the destination region.

Building the manifest by hand

s5cmd is far faster than the AWS CLI at listing millions of keys. I piped its output through sed to strip it down to keys, then through a short Python step that URL-encodes each key:

s5cmd ls 's3://my-staging-bucket-eu/*' \
  | sed -E 's/^.{19}[[:space:]]+[0-9]+[[:space:]]+//' \
  | python3 -c '
import sys, urllib.parse
for line in sys.stdin:
    k = line.rstrip("\n")
    if k:
        print("my-staging-bucket-eu," + urllib.parse.quote(k, safe=""))
' > manifest.csv

wc -l manifest.csv   # should match the object count
aws s3 cp manifest.csv s3://my-prod-bucket-apac/batch-manifests/manifest.csv
aws s3api head-object --bucket my-prod-bucket-apac \
  --key batch-manifests/manifest.csv --query ETag --output text

Use safe=''. Encode everything, slashes included, and Batch decodes it correctly. Keys with spaces, plus signs, or non-ASCII characters are where hand-built manifests usually break. The file came to about 1.23 GB with one line per object, which matched the object count exactly. It went into the destination-region bucket next to the job.

The job

aws s3control create-job --region "$DST_REGION" \
  --account-id "$ACCOUNT_ID" \
  --operation '{"S3PutObjectCopy":{
      "TargetResource":"arn:aws:s3:::my-prod-bucket-apac",
      "MetadataDirective":"COPY",
      "StorageClass":"STANDARD"}}' \
  --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},
     "Location":{"ObjectArn":"arn:aws:s3:::my-prod-bucket-apac/batch-manifests/manifest.csv",
                 "ETag":"<manifest-etag>"}}' \
  --report '{"Bucket":"arn:aws:s3:::my-prod-bucket-apac","Prefix":"batch-reports",
     "Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \
  --priority 10 --role-arn "$BATCH_ROLE_ARN" \
  --confirmation-required --client-request-token "$(uuidgen)"

MetadataDirective: COPY keeps content types and user metadata. FailedTasksOnly keeps the report small. If it's empty, that's your proof.

The job went through the lifecycle you'd hope for. Preparing read every manifest row (about 12 million) with 0 errors. It then sat Suspended, waiting for my confirmation. I read the summary, then released it:

aws s3control update-job-status --region "$DST_REGION" \
  --account-id "$ACCOUNT_ID" --job-id "$JOB_ID" --requested-job-status Ready

Active, then Complete: every object succeeded, 0 failed. Always keep --confirmation-required on. Validating the manifest before twelve million PUTs is cheap insurance.

The mistake I'm a bit embarrassed by

Before I settled on Batch, I briefly ran an rclone S3-to-S3 copy on the EC2 box and thought it was a backbone transfer. It isn't. rclone always streams the bytes through the machine it runs on. Server-side copy within a single provider is a narrow case, and it doesn't cover cross-region at this scale with this tooling. If you want the bytes to stay inside AWS, use Batch Operations or replication, not a relay instance.

What hop 2 cost

Hop 2's bill comes down to three things: the Batch Operations job and per-object fees, about 12 million PUT requests in the destination region, and roughly 2.5 TB of inter-region transfer. All three were small. DigitalOcean egress and the EC2 time for hop 1 come on top. Staging in the EU isn't free, but it's cheap next to an engineer babysitting a latency-bound copy for a week.

Cutover and cleanup

The manifest is a snapshot. Anything written to Spaces after you list it isn't in it. So the cutover went like this:

1. Switch application writes to the APAC bucket, or briefly pause uploads.
2. Run a final delta sync of the recent prefixes directly from DO to the APAC bucket. It's small enough that latency doesn't matter. Use --checksum, because this is the last chance to catch an in-place change:

   rclone copy dospaces:my-source-space/<YYYY/MM> s3apac:my-prod-bucket-apac/<YYYY/MM> --checksum -v

1. Clean up: terminate the EC2 instance, put a lifecycle expiration rule on the staging bucket, delete the manifest and report prefixes, and remove the Batch role.
2. Rotate every credential used during the migration. Spaces keys, any access keys, all of it. Migration credentials are long-lived by accident and broad by design, so don't leave them lying around.

What I'd do differently

• Start on a memory-optimised instance and work per folder from minute one.
• Monitor from the first run. Six hours of a frozen box is the cost I resent most.
• Use --size-only and the no-HEAD flags immediately for small files into an empty destination.
• Skip the generated manifest for cross-region jobs. Build the CSV and run in the destination region.
• Remember that rclone is a relay, not a backbone.

Two hops, about a day and a half including mistakes, and zero failed objects.

I lead platform engineering for fintech. Read the full case study here or see how we can work together at Nazmul Ahasan.