DEV Community

Cover image for Python: open().read()->str , but for big files

Python: open().read()->str , but for big files

Tai Kedzierski on July 28, 2022

Image (C) Tai Kedzierski We just had a use case where we needed to POST a file over to a server. The naive implementation for posting with request...
Read full post
Collapse
 
xtofl profile image
xtofl •

That's so terribly simple!

I was anticipating some multipart chunked transferring, but this makes excellent use of the machinery offered by the operating system.

Ever considered using contextlib?

@contextlib.contextmanager
def readmm(filename):
      _fh = open(file_name, mode)
      fsize = 0
     _mmap = mmap.mmap(self._fh.fileno(), fsize, access=mmap.ACCESS_READ)
     try:
        yield _mmap
     finally:
      _fh.close()
      _mmap.close()

with readmm("my_file.bin", "rb") as data:
  requests.post(url, data={"bytes": data})
Enter fullscreen mode Exit fullscreen mode
Collapse
 
taikedz profile image
Tai Kedzierski •

That does make it even more concise !

That said, I'm not sure how I feel about the enter/exit context being implicit behind this; as in, it reduces the amount of code, but I can feel like reading it back feels unintuitive.

Collapse
 
xtofl profile image
xtofl •

Mind, it's not implicit! It's extracted into the @contextmanager function.

I respect that you phrase it as unintuitive, since intuition is learned. Indeed, to (very) many, the extracted form of the code sandwich is intuitive. You can observe the movement from explicit sandwiches to extracted in many languages (e.g. Scope.Exit, using in C#).

The power it brings is that the developer cannot possibly forget to cleanup, so the reader doesn't have to wonder whether they did. Assuming code is read 10 times more than it is written, inner peace will be your part after growing this intuition.

Thread Thread
 
taikedz profile image
Tai Kedzierski •

Indeed. I guess it's something I just have to get used to - can be regarded as analogous to the with keyword which, unless you've learned and used it properly, can look oddly incomplete.