DEV Community

loading...

AWS S3 as Docker volumes

chattes profile image Sourav Chatterjee ・1 min read

Attaching an S3 Bucket as NFS Volume in Docker

We can attach an S3 bucket as a mounted Volume in docker.

We need to use a Plugin to achieve this.

The plugin is

rexray/rexray

We will first install the plugin

docker plugin install rexray/s3fs:latest S3FS_REGION=us-east-2 S3FS_OPTIONS="allow_other,iam_role=auto,umask=000" --grant-all-permissions
Enter fullscreen mode Exit fullscreen mode

We will have to install the plugin as above ,as it gives access to the plugin to S3.

Once installed we can check using

docker plugin ls
Enter fullscreen mode Exit fullscreen mode

Now we can mount the S3 bucket using the volume driver like below to test the mount.

docker run -ti --volume-driver=rexray/s3fs -v ${aws-bucket-name}:/data ubuntu sleep infinity
Enter fullscreen mode Exit fullscreen mode

Thats it the Volume has been mounted from our S3 Bucket

We can inspect the container and check if the bucket has been mounted

"Mounts": [
            {
                "Type": "volume",
                "Name": "maps-openmaps-schools",
                "Source": "",
                "Destination": "/data",
                "Driver": "rexray/s3fs:latest",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
Enter fullscreen mode Exit fullscreen mode

We can also inspect the volume

$ docker volume inspect maps-openmaps-schools
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "rexray/s3fs:latest",
        "Labels": null,
        "Mountpoint": "",
        "Name": "maps-openmaps-schools",
        "Options": null,
        "Scope": "global",
        "Status": {
            "availabilityZone": "",
            "fields": null,
            "iops": 0,
            "name": "maps-openmaps-schools",
            "server": "s3fs",
            "service": "s3fs",
            "size": 0,
            "type": ""
        }
    }
]
Enter fullscreen mode Exit fullscreen mode

Discussion (3)

pic
Editor guide
Collapse
bonespiked profile image
bonespiked

Remember that s3 is NOT a file system, but an object store - while mounting IS an incredibly useful capability - I wouldn't leverage anything more than file read or create - don't try to append a file, don't try to use file system trickery (e.g. links, fs event listening, etc....).

Collapse
raphink profile image
Raphaël Pinson

Thanks for that post.

How are the perfs?

I remember making an s3fs-based system in Kubernetes some time ago, and the perfs were pretty bad...

Collapse
chattes profile image
Sourav Chatterjee Author

Will be keeping an eye on the performance. I am loading a pretty big file , have not tested as yet