Software Delivery Futurist.
Chief Storyteller at https://perfectscale.io.
CEO and founder at https://otomato.io.
Co-founder at https://stagecentral.io.
No, speed wasn't a consideration here. Main motivation here was providing an easy and transparent way to upload files and make them accessible to pods. S3 gives users an easy and secure UI for that. Goofys is supposedly quite performant compared to other FUSE implementations (i.e s3fs). But we haven't benchmarked this ourselves.
Software Delivery Futurist.
Chief Storyteller at https://perfectscale.io.
CEO and founder at https://otomato.io.
Co-founder at https://stagecentral.io.
I tried to use the sharing method to complete the entire demo, but unfortunately this didn't work. Because the goofys mount directory( /var/s3fs) in daemonset is not the same as the directory I want to share with the host(/var/s3fs:shared);
/otomato # df -h
Filesystem Size Used Available Use% Mounted on
/dev/nvme0n1p1 50.0G 4.7G 45.3G 9% /var/s3fs:shared
poc-s3goofys-source 1.0P 0 1.0P 0% /var/s3fs
Software Delivery Futurist.
Chief Storyteller at https://perfectscale.io.
CEO and founder at https://otomato.io.
Co-founder at https://stagecentral.io.
I have tried this and the other similar option mentioned in this blog. blog.meain.io/2020/mounting-s3-buc.... In neither case, the mounting to hostPath was successful for the cluster managed by AWS EKS.
Software Delivery Futurist.
Chief Storyteller at https://perfectscale.io.
CEO and founder at https://otomato.io.
Co-founder at https://stagecentral.io.
Hi @dirai09 , this was originally tested on AWS EKS. I haven't tested it since but it should in theory still work. What is the error you're getting when trying to mount the hostPath?
Also - can you share your config in a gist?
In high number of files its fail you due to nature of s3 api for small files the http response will be bigger than files.
I think mounting s3 is a bad idea, if you have enough developing resources its better to write a client for code to connect directly to s3 and cache list of s3 files ... For better performance.
But its a fun thing to do, also cephfs with rados gateway will give you better performance in kubernetes
Software Delivery Futurist.
Chief Storyteller at https://perfectscale.io.
CEO and founder at https://otomato.io.
Co-founder at https://stagecentral.io.
good to know. not an issue in our case - we have a small number of large files there. And I agree it's not such a great idea in general - both performance wise and because of the hidden complexity. But it solved our specific itch and may help others solve it.
I ran into an issue where goofys doesn't reload the content of a small txt file. It updates the timestamp though. Do you know what could be wrong?
I have goofys run inside a container.
We're a place where coders share, stay up-to-date and grow their careers.
Cool setup. Have you tested its speed?
No, speed wasn't a consideration here. Main motivation here was providing an easy and transparent way to upload files and make them accessible to pods. S3 gives users an easy and secure UI for that. Goofys is supposedly quite performant compared to other FUSE implementations (i.e s3fs). But we haven't benchmarked this ourselves.
Thx! Would love to know numbers if you ever do try it :-)
Amazing approach!
Thx for sharing in details.
Hi, Can set multiple bucketName?
I need to interact with few s3 buckets for different tasks
hi @oleksiihead
no support for this right now.
to add this one would need to do smthng like:
If you get to do this - please submit a PR.
I tried to use the sharing method to complete the entire demo, but unfortunately this didn't work. Because the goofys mount directory( /var/s3fs) in daemonset is not the same as the directory I want to share with the host(/var/s3fs:shared);
Is there any configuration I missed?
Daemonset.yaml
What's your node OS? Is mount propagation enabled in the container runtime? See this note here: kubernetes.io/docs/concepts/storag...
I have tried this and the other similar option mentioned in this blog. blog.meain.io/2020/mounting-s3-buc.... In neither case, the mounting to hostPath was successful for the cluster managed by AWS EKS.
Hi @dirai09 , this was originally tested on AWS EKS. I haven't tested it since but it should in theory still work. What is the error you're getting when trying to mount the hostPath?
Also - can you share your config in a gist?
Nice approach. However, you might want to have a look at JuiceFS:
github.com/juicedata/juicefs
That has quite a good performance due to the combination with Redis and it is made with Kubernetes in mind.
Thanks for the wonderful suggestion @randy
In high number of files its fail you due to nature of s3 api for small files the http response will be bigger than files.
I think mounting s3 is a bad idea, if you have enough developing resources its better to write a client for code to connect directly to s3 and cache list of s3 files ... For better performance.
But its a fun thing to do, also cephfs with rados gateway will give you better performance in kubernetes
good to know. not an issue in our case - we have a small number of large files there. And I agree it's not such a great idea in general - both performance wise and because of the hidden complexity. But it solved our specific itch and may help others solve it.
Hi,
I don't think I am able to mount the volumes on the hostPath. Am I missing something here.
I ran into an issue where goofys doesn't reload the content of a small txt file. It updates the timestamp though. Do you know what could be wrong?
I have goofys run inside a container.