We are a team of 3 who built YUKI, a terminal editor with no dependencies. My contributions lay in the core, specifically storage and memory management.
The obvious way to read a file is f.read().decode().split("\n"). This makes each line in the file a new Python object in memory, but for large files this can easily fill up your RAM.
For the storage layer I did not use any external packages, relying on the stdlib. CompactLines uses a bytearray for compact line offsets, while MappedLines uses mmap for large read-heavy files.
But then there is correctness. Deletion. Offsets. Trailing lines. Slices. Undo. All of it had something to do with it. Differential testing against a Python list unearthed corruption bugs I would not have found otherwise.
Not having to rely on these packages taught me what they were doing for me.
The memory management was also written around explicit limits. At 8 MiB a file is considered large and falls into the large-file handling. Large read-heavy files can make use of MappedLines and mmap instead of allocating an editable representation right away. Once editing begins this can switch into CompactLines.
Another optimization, hidden deep within, is the undo mechanism. The undo stack is capped at 500 snapshots or 32 MiB of snapshot data, whichever comes first. If an individual snapshot would fly over this budget, it simply isn't saved.
These are practical limits. A fixed 8 MiB budget does not take into account how much memory the computer has or the memory footprint of everything else, and the undo system could potentially be optimized to use a more fine-grained or delta-based encoding that would save memory. These are areas I would like to iterate on and improve in the future.
So I had to think about not only the memory footprint of the document, but the memory footprint of the computer as a whole and how things like undo add to it.
Zero dependencies don't take away complexity, they force you to own it.`if file_size >= 8 * 1024 * 1024:
lines = MappedLines(path) # mmap for large files
else:
lines = CompactLines(data) # compact storage
Undo is bounded
if snapshot_size <= 32 * 1024 * 1024 and len(undo) < 500:
undo.append(snapshot)`
Top comments (0)