Using MinIO with Rails

#ruby #rails #minio #s3

I'm on a project where the file upload setup uses Active Storage with Amazon S3. But there's no configuration for me to download files from S3 and use them in development.

When I looked at config/storage.yml I saw the following:

local:
  service: Disk
  root: <%= Rails.root.join("storage") %>

I figured it would be simple, just bring everything inside the S3 Bucket into the app's app/storage folder.

I downloaded the AWS CLI and set it up with my credentials.

Then I synced the S3 files into my app/storage folder:

$ aws s3 sync s3://bucket-name ~/Projects/selected-project/storage

Started the app and it didn't work 🙄.

In ActiveStorage, when the configured service is Disk, after the blob generates the key, the file is saved inside two folders and then the blob key.

I checked the file path in the console and it returned:

> user = User.first
> ActiveStorage::Blob.service.send(:path_for, u.avatar.key)
=> "storage/jt/Y7/jtY7656jGPvfPMUUA8kX6Vb4"

I noticed those folders were the first 4 characters of the key.

I thought of a few solutions:

Write a script to generate those subfolders and place the files in them.

I didn't go with this. It might not work, and since I don't have experience with scripts that change the filesystem, I figured it would take too long.
Check how Active Storage generates path_for and tell it to generate directly without those subfolders.

I searched and didn't find anywhere I could do that.
Duplicate the bucket inside S3 and use it for development.

Didn't like the idea, I'd have to pay for storing 2 buckets.
Use MinIO.

This was the option I figured was fastest and had the best cost-benefit.

For those who don't know, MinIO is an Open Source project written in Go, designed from the start to be the standard for object storage in private cloud. It's a cloud-native object server with concurrent, scalable and lightweight performance (more info).

I need to set up MinIO like S3, since the app needs to think it's talking to S3. I downloaded MinIO from their site and set up the environment variables:

export MINIO_ACCESS_KEY="minio_storage_development"
export MINIO_SECRET_KEY="minio_storage_development"
export MINIO_REGION_NAME="us-east-1"

It's configured through environment variables. You can check all the settings in the docs. (MinIO docs)

Now just start it:

$ minio server ~/minio_storage

Endpoint:  http://xxx.xxx.xxx.xxx:9000  http://xxx.xxx.xxx.xxx:9000  http://127.0.0.1:9000
AccessKey: minio_storage_development
SecretKey: minio_storage_development
Region:    us-east-1

Browser Access:
  http://xxx.xxx.xxx.xxx:9000  http://xxx.xxx.xxx.xxx:9000  http://127.0.0.1:9000

Command-line Access: https://docs.min.io/docs/minio-client-quickstart-guide
  $ mc config host add myminio http://100.100.101.162:9000 minio_storage_development minio_storage_development

Note: I hid my IP with xxx.xxx.xxx.xxx because it's static and I don't want it exposed to the web.

There you go, a MinIO server running on your machine.

I accessed it through my browser at http://127.0.0.1:9000/ using the Access Key and Secret Key, and created a Bucket called rails_app_bucket.

In the Rails settings at config/storage.yml:

local:
  service: S3
  endpoint: http://127.0.0.1:9000
  access_key_id: minio_storage_development
  secret_access_key: minio_storage_development
  region: us-east-1
  bucket: rails_app_bucket
  force_path_style: true

I moved all the files I had downloaded from the S3 bucket into the MinIO bucket folder:

$ mv ~/Projects/selected-project/storage/* ~/minio_storage/rails_app_bucket

To sync files from S3 I now use:

$ aws s3 sync s3://bucket-name ~/minio_storage/rails_app_bucket

I also added MinIO to my Procfile.development so the server starts whenever I run the app through foreman:

# Procfile.development

server: bin/rails server
webpacker: bin/webpack-dev-server
redis: redis-server
sidekiq: bundle exec sidekiq
minio: minio server ~/minio_storage