In high number of files its fail you due to nature of s3 api for small files the http response will be bigger than files.
I think mounting s3 is a bad idea, if you have enough developing resources its better to write a client for code to connect directly to s3 and cache list of s3 files ... For better performance.
But its a fun thing to do, also cephfs with rados gateway will give you better performance in kubernetes
Software Delivery Futurist.
Chief Storyteller at https://perfectscale.io.
CEO and founder at https://otomato.io.
Co-founder at https://stagecentral.io.
good to know. not an issue in our case - we have a small number of large files there. And I agree it's not such a great idea in general - both performance wise and because of the hidden complexity. But it solved our specific itch and may help others solve it.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
In high number of files its fail you due to nature of s3 api for small files the http response will be bigger than files.
I think mounting s3 is a bad idea, if you have enough developing resources its better to write a client for code to connect directly to s3 and cache list of s3 files ... For better performance.
But its a fun thing to do, also cephfs with rados gateway will give you better performance in kubernetes
good to know. not an issue in our case - we have a small number of large files there. And I agree it's not such a great idea in general - both performance wise and because of the hidden complexity. But it solved our specific itch and may help others solve it.