Sliplane looks very interesting - I'll give it a try. I have no end of sample apps with a Docker image.
As I mention in my profile, I'm Chief Technical Evangelist at Backblaze, and I noticed that you said that you're paying us more for API transactions than for storage, and:
There is probably some optimization potential there 🫠
There almost certainly is! There are a few ways that this can happen, often because of poorly written or poorly configured client software. In the Backblaze web UI, you can click into "Reports" and get some insight into what's happening. The culprit is usually one or more of "head object", in class B, or "list file names" or "list objects" in class C.
Many developers working with cloud object storage don't consider the fact that API calls are a limited/costly resource as they are developing their apps, and make repeated calls to head/list rather than caching the results. Also, layers of abstraction can mask what's happening under the covers. Repeated "stat" calls to get the attributes of what looks to the app like a local file can result in many redundant API calls.
Sometimes there's a simple fix - for example, Rclone has a --fast-list flag (you can set the RCLONE_FAST_LIST environment variable so you don't have to use the flag with every call) that lets it work more efficiently with cloud object storage such as B2 and S3, but it isn't the default, since it can result in higher memory use.
Earlier today I was on a call with a customer seeing lots of API call usage - their NAS sync software was making 15 head requests and about 1 list request per file each time it ran. Each list call can return up to 1,000 results, so there's no way this should be happening! I advised them to contact the NAS vendor to have them look into it.
Hah that’s cool, thanks for responding! Love backblaze btw, great product. The transactions all come from restic (backup tool, not sure if you’re familiar). Need to check if I can tune something there! But even if I can tune, the costs are still incredible low for now:)
Some comments have been hidden by the post's author - find out more
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Sliplane looks very interesting - I'll give it a try. I have no end of sample apps with a Docker image.
As I mention in my profile, I'm Chief Technical Evangelist at Backblaze, and I noticed that you said that you're paying us more for API transactions than for storage, and:
There almost certainly is! There are a few ways that this can happen, often because of poorly written or poorly configured client software. In the Backblaze web UI, you can click into "Reports" and get some insight into what's happening. The culprit is usually one or more of "head object", in class B, or "list file names" or "list objects" in class C.
Many developers working with cloud object storage don't consider the fact that API calls are a limited/costly resource as they are developing their apps, and make repeated calls to head/list rather than caching the results. Also, layers of abstraction can mask what's happening under the covers. Repeated "stat" calls to get the attributes of what looks to the app like a local file can result in many redundant API calls.
Sometimes there's a simple fix - for example, Rclone has a
--fast-listflag (you can set theRCLONE_FAST_LISTenvironment variable so you don't have to use the flag with every call) that lets it work more efficiently with cloud object storage such as B2 and S3, but it isn't the default, since it can result in higher memory use.Earlier today I was on a call with a customer seeing lots of API call usage - their NAS sync software was making 15 head requests and about 1 list request per file each time it ran. Each list call can return up to 1,000 results, so there's no way this should be happening! I advised them to contact the NAS vendor to have them look into it.
Hah that’s cool, thanks for responding! Love backblaze btw, great product. The transactions all come from restic (backup tool, not sure if you’re familiar). Need to check if I can tune something there! But even if I can tune, the costs are still incredible low for now:)