DEV Community

Mount S3 Objects to Kubernetes Pods

Ant(on) Weiss on January 31, 2022

This post describes how to mount an S3 bucket to all the nodes in an EKS cluster and make it available to pods as a hostPath volume. Yes, we're awa...

Read full post

Eldad Assis • Jan 31 '22

Cool setup. Have you tested its speed?

Ant(on) Weiss • Jan 31 '22

No, speed wasn't a consideration here. Main motivation here was providing an easy and transparent way to upload files and make them accessible to pods. S3 gives users an easy and secure UI for that. Goofys is supposedly quite performant compared to other FUSE implementations (i.e s3fs). But we haven't benchmarked this ourselves.

Eldad Assis • Jan 31 '22

Thx! Would love to know numbers if you ever do try it :-)

Oleksii Smiichuk • Feb 22 '23

Hi, Can set multiple bucketName?
I need to interact with few s3 buckets for different tasks

Ant(on) Weiss • Feb 24 '23

hi @oleksiihead
no support for this right now.
to add this one would need to do smthng like:

modify the Dockerfile to replace the container startup command with an entrypoint script that mounts buckets in a loop.
modify the Helm chart to receive an dictionary of bucket names and mount points and pass these into the DaemonSet

If you get to do this - please submit a PR.

Lidor Ettinger • Feb 8 '22

Amazing approach!
Thx for sharing in details.

Big Bunny • Jul 25 '23 • Edited

I tried to use the sharing method to complete the entire demo, but unfortunately this didn't work. Because the goofys mount directory( /var/s3fs) in daemonset is not the same as the directory I want to share with the host(/var/s3fs:shared);

/otomato # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/nvme0n1p1           50.0G      4.7G     45.3G   9% /var/s3fs:shared
poc-s3goofys-source 1.0P         0      1.0P   0% /var/s3fs

Is there any configuration I missed?

Daemonset.yaml


apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: s3-mounter
  name: s3-mounter
  namespace: otomount
spec:
  selector:
    matchLabels:
      app: s3-mounter
  template:
    metadata:
      labels:
        app: s3-mounter
    spec:
      serviceAccountName: s3-mounter
      containers:
      - name: mounter 
        image: otomato/goofys
        securityContext:
          privileged: true
        command: ["/bin/sh"]
        args: ["-c", "mkdir -p /var/s3fs && ./goofys --region xxxxx -f poc-s3goofys-source /var/s3fs"]
        volumeMounts:
          - name: devfuse
            mountPath: /dev/fuse
          - name: mntdatas3fs
            mountPath: /var/s3fs:shared
      volumes:
        - name: devfuse
          hostPath:
            path: /dev/fuse
        - name: mntdatas3fs
          hostPath:
            path: /mnt/s3data

Ant(on) Weiss • Jul 30 '23

What's your node OS? Is mount propagation enabled in the container runtime? See this note here: kubernetes.io/docs/concepts/storag...

dirai09 • May 26 '23

I have tried this and the other similar option mentioned in this blog. blog.meain.io/2020/mounting-s3-buc.... In neither case, the mounting to hostPath was successful for the cluster managed by AWS EKS.

Ant(on) Weiss • May 28 '23

Hi @dirai09 , this was originally tested on AWS EKS. I haven't tested it since but it should in theory still work. What is the error you're getting when trying to mount the hostPath?
Also - can you share your config in a gist?

Randy Gupta • Jun 22 '22 • Edited

Nice approach. However, you might want to have a look at JuiceFS:

github.com/juicedata/juicefs

That has quite a good performance due to the combination with Redis and it is made with Kubernetes in mind.

Sarav AK • May 8 '23

Thanks for the wonderful suggestion @randy

behrooz hasanbeygi • Jan 31 '22

In high number of files its fail you due to nature of s3 api for small files the http response will be bigger than files.

I think mounting s3 is a bad idea, if you have enough developing resources its better to write a client for code to connect directly to s3 and cache list of s3 files ... For better performance.
But its a fun thing to do, also cephfs with rados gateway will give you better performance in kubernetes

Ant(on) Weiss • Feb 2 '22

good to know. not an issue in our case - we have a small number of large files there. And I agree it's not such a great idea in general - both performance wise and because of the hidden complexity. But it solved our specific itch and may help others solve it.

dirai09 • May 25 '23

Hi,
I don't think I am able to mount the volumes on the hostPath. Am I missing something here.

Jane • Mar 30

I ran into an issue where goofys doesn't reload the content of a small txt file. It updates the timestamp though. Do you know what could be wrong?
I have goofys run inside a container.