Good news everyone!
Reduct Storage Python SDK v0.3.0 has been released.
It supports the latest Reduct Storage HTTP API v0.6.0, but the most important change is data streaming. Let's see how it works with examples!
Prerequisites
This post isn't a tutorial, and I suppose you know how to launch an instance of Reduct Storage and create a bucket there. If you don't know, then a few links that can help:
Read Data By Chunks
In this release, we introduced a new method Bucket.read_by to read data by chunks with an asynchronous iterator:
with open("file.bin", "wb") as file:
async for chunk in bucket.read_by("entry-1", chunk_size=1024):
file.write(chunk)
It might be useful when you have a big record to download or need to read many records in the same time.
Write Data By Chunks
To write data by chunks, can be a bit more tricky. First, you have to pass an asynchronous iterator to the Bucket.write method. Then you have to specify the size of the data which you're writing. The storage engine requires the content-lenght header, and the user should know the size of written data in advance.
file_path = Path("./path/to/file")
async def sender():
with open(file_path, "br") as file:
while True:
chunk = file.read(1024)
if not chunk:
break
yield chunk
content_length = file_path.stat().st_size
await bucket.write("entry-1", sender(), content_length=content_length)
If you've got interested, learn more about the storage engine here
Cover Image Designed by roserodionova / Freepik
Top comments (0)