DEV Community

Lucas Kuhn
Lucas Kuhn

Posted on

Syncing files from AWS S3 to local storage on Rails

I usually use Amazon S3 for storing images with Active Storage. There are some issues when pulling my database to work with it on localhost:

  • I don't want to accidentally delete something on S3
  • I don't want my localhost without images 🥲

I added this rake task to download files from S3 and format them in a way that works with local storage:

# lib/tasks/s3_to_local.rake 
desc "Sync files from S3 bucket to local storage folder"

task :s3_to_local do

  s3_bucket = "YOUR_S3_BUCKET"
  access_key_id = Rails.application.credentials.dig(:aws, :access_key_id)
  secret_access_key = Rails.application.credentials.dig(:aws, :secret_access_key)

  storage_folder = Rails.root.join('storage')

  system("AWS_ACCESS_KEY_ID=#{access_key_id} AWS_SECRET_ACCESS_KEY=#{secret_access_key} aws s3 sync s3://#{s3_bucket} #{storage_folder}")

  # Ignores sub_folders already created and .keep files
  images = { |file| file.file? && !file.empty? }

  # Formats the file path of each image so ActiveStorage understands them using :local storage
  images.each do |path_name|
    dir, basename = path_name.split
    file_name = basename.to_s
    sub_folders = dir.join(file_name[0..1], file_name[2..3])
    sub_folders.mkpath # Create the subfolder used by active_record
    path_name.rename(dir + sub_folders + basename) # Renames file to be moved into subfolder

Enter fullscreen mode Exit fullscreen mode

What makes this tricky is that S3 storage has everything in the root of the bucket, and when using local storage with ActiveStorage, everything should be under a subfolder.

Also, you need AWS CLI for this to work:

brew install awscli
Enter fullscreen mode Exit fullscreen mode

Basically copy this task into lib/tasks/s3_to_local.rake, add your bucket name, your AWS credentials, run rake s3_to_local and you're good to go

Top comments (1)

giridharans profile image

Nice !!
But i want to Sync Digital Ocean to local storage not 'AWS'
Is it possible ?