DEV Community: D

[Side B] Should a Binary-Only FS Support Text Mode? Redrawing the Architecture Boundary

Wed, 08 Apr 2026 17:14:51 +0000

From the Author:
D-MemFS was featured in Python Weekly Issue #737 (March 19, 2026) under Interesting Projects, Tools and Libraries. Being picked up by one of the most widely-read Python newsletters confirmed that in-memory I/O bottlenecks and memory management are truly universal challenges for developers everywhere. This series is my response to that interest.

🧭 About this Series: The Two Sides of Development

In Japan, I publish this series across two distinct platforms to serve different developer needs. To provide the complete picture here on Dev.to, I've brought them together as two "Sides":

Side A (Practical / originally on Qiita): Focuses on the "How". Implementation details, benchmarks, and concrete solutions for practical use cases.
Side B (Philosophy / originally on Zenn): Focuses on the "Why". The development war stories, design decisions, and how I collaborated with AI through Specification-Driven Development (SDD).

What the Design Document Said

D-MemFS's design principles include a clause in Article 3:

MFS is dedicated to "pure byte-sequence virtual hierarchy management and resource control." Text encoding, encryption, and physical persistence are delegated to upper-layer boundary controllers.

And in the open() mode specification:

Text mode (r, w, etc. — modes without the binary suffix b) shall raise ValueError.

In other words, text mode was explicitly excluded by design.

The reasoning is straightforward and consistent. A memory filesystem's job is to manage byte sequences. Encoding is an application-layer concern — the filesystem layer has no business managing it. Unix VFS doesn't know about text either. Introducing text mode would drag in encoding selection, newline translation, and multibyte boundary handling — a significant expansion of responsibility. There was no good reason to make that trade-off at the outset.

Design Correctness Doesn't Stop Reality From Punching Back

The design reviews were thorough. The implementation was solid. 346 test cases passed. Coverage exceeded 97%. We were in final polish mode before release.

Then I paused and asked myself:

"What will users actually think of this?"

Imagine being a user wanting to work with log files, config files (.ini, .json, .yaml) in memory with D-MemFS. No text mode. How do you read a config?

with mfs.open("/config/settings.json", "rb") as f:
    text = f.read().decode("utf-8")
    config = json.loads(text)

…It works. But am I really going to write .decode() every single time? And the writes too?

data = json.dumps(config, ensure_ascii=False).encode("utf-8")
with mfs.open("/config/settings.json", "wb") as f:
    f.write(data)

It works, but it's not natural.

Particularly for the ZIP-in-memory use case — one of D-MemFS's primary advertised strengths is "extract a ZIP into memory and process its contents." If every text file read after extraction requires an explicit .decode(), that felt like a lack of care toward the user.

"Just Use `io.TextIOWrapper`?" — Not That Simple

Python's standard library provides io.TextIOWrapper, which wraps a binary stream to provide text I/O — used everywhere under the hood in normal file operations.

"Just wrap the handle with TextIOWrapper(handle) — problem solved," I initially thought. But trying it revealed three distinct problems.

Problem 1: The `readinto()` Requirement

io.TextIOWrapper requires readinto() on its target for internal buffering. MemoryFileHandle was designed with a clean interface of read() / write() / seek() / tell() — no readinto().

Adding it is technically possible, but readinto() is tightly coupled to memoryview and the buffer protocol, which would ripple through the memory management design. Modifying the internal buffer architecture just to support text mode felt like the tail wagging the dog.

Problem 2: Buffering vs. Quota Incompatibility

TextIOWrapper maintains an internal buffer. Buffered writes are not reflected in the quota system until flush() is called. This creates a window where "the quota is technically exceeded, but writes are still succeeding."

D-MemFS has a hard quota. "Reject before writing" is the cornerstone of its design. Buffering-induced delayed accounting directly contradicts this principle.

Problem 3: The `seek()` Cookie Problem

TextIOWrapper's seek() uses opaque "cookies" — not byte offsets. You can only seek() to positions previously returned by tell(). This is fundamentally incompatible with MemoryFileHandle, which navigates freely by byte offset.

Solution: Build a Lightweight Wrapper — Without a Buffer

The conclusion was to build MFSTextHandle, a lightweight custom wrapper. Its design is guided by three principles:

No internal buffer. Every write() is immediately delegated to the binary handle. Quota checks are always immediate. The hard quota contract is never broken.

Encoding/decoding is the only responsibility. MFSTextHandle does exactly one thing: convert between strings and bytes. It holds no file operation logic, no lock management. Everything is delegated to the underlying MemoryFileHandle.

No close() responsibility. Handle lifecycle is managed by MemoryFileHandle's with statement. MFSTextHandle is a thin adapter — nothing more.

with mfs.open("/data/hello.txt", "wb") as f:
    th = MFSTextHandle(f, encoding="utf-8")
    th.write("Hello, world\n")

This is a helper used exclusively within a with mfs.open(...) scope. No separate lifecycle management of MFSTextHandle is required.

`read(size)` Is an Approximation in Characters

There is one limitation that deserves honest acknowledgment.

The size argument to read(size). In Python's standard text mode, size means character count. But because MFSTextHandle has no internal buffer, it passes size to handle.read(size) as a byte count.

For UTF-8 multibyte characters (e.g. Japanese), this means fewer characters than requested may be returned.

# In UTF-8, one Japanese character = 3 bytes
th.read(3)  # Reads 3 bytes → returns "こ" (1 character) — fine
th.read(1)  # Reads 1 byte → potential decode error mid-multibyte sequence

This limitation is explicitly documented in the docstring:

Note that this is an approximation in characters, not bytes.

It is not perfect. But for the most common use case — read() with no argument to read the entire file — it is not an issue. And readline(), which detects newlines byte by byte, is also unaffected by multibyte boundaries.

`readline()` and Three-Way Newline Handling

One more detail that required careful implementation: newline detection.

POSIX uses \n, old Mac used \r, Windows uses \r\n. MFSTextHandle.readline() recognizes all three.

The implementation is deliberately simple. Read one byte at a time, stop at \n or \r. If \r is found, peek at the next byte: if it's \n, consume it (treating \r\n as a single newline); otherwise, seek back.

if b == b"\r":
    next_b = self._handle.read(1)
    if next_b == b"\n":
        buf.extend(next_b)    # treat \r\n as a single newline
    elif next_b:
        self._handle.seek(self._handle.tell() - 1)  # put it back

Is reading one byte at a time slow? In theory, yes. But this is memory — not disk I/O. Reading one byte at a time from memory is a fundamentally different order of magnitude from disk. In practice, this straightforward implementation is not a bottleneck.

Did We Violate the Design Principle?

This is the question I thought hardest about.

Design principle Article 3 says "text encoding is delegated upward." MFSTextHandle handles text encoding. Isn't that a contradiction?

I concluded it isn't. Here's why.

MFSTextHandle is not a method of MemoryFileSystem. open() did not start accepting text modes. open() still accepts only rb, wb, ab, and r+b — pass a text mode and you get ValueError, same as before.

MFSTextHandle is an optional utility class. Users import it explicitly when they want it. It does not intrude into the filesystem layer's design at all.

In other words, the "MFS shall" scope of the design principle is fully preserved. Text processing was provided as a separate class outside of MFS. The design philosophy was not bent — the boundary was redrawn more precisely, and usability was added on top.

The Boundary Between Design Purity and Practicality

When building a library, there will always be a moment where "design purity" and "ease of use" collide.

Faithfulness to principle matters. But if following a principle forces unnecessary friction on users, it is worth re-examining where that principle applies — not whether it applies.

The key, I believe, is not to "bend" the principle but to "redraw the boundary accurately":

The filesystem layer (MemoryFileSystem) remains binary-only → maintained
Text conversion is provided as a separate class (MFSTextHandle) → added
No internal buffer; quota immediacy is preserved → consistent with design principles
open() mode specification is unchanged → API contract maintained

I did not change the principle written in the design document. I defined more precisely where that principle applies.

Closing

Adding MFSTextHandle is a small feature. _text.py is only 136 lines.

But the decision to add those 136 lines was where I spent the most time. Because it wasn't merely a feature addition — it was a dialogue with the design philosophy.

I wrote the design document. I reviewed it with AI repeatedly. I implemented, tested, and made 346 test cases pass. At the very end, I asked myself: "Am I violating the principles I've defended throughout this entire process?" And I needed to be able to answer — to myself — "No, here's why."

That ability to explain yourself is, I believe, the point of writing a design document in the first place.

Without one, this decision would have ended at "seemed useful so I added it." With one, I could ask "how does this align with the principles?" and actually answer it.

A design document isn't only useful for deciding what to include. It's useful for explaining why inclusion is justified.

🔗 Links & Resources

GitHub: https://github.com/nightmarewalker/D-MemFS
PyPI: https://pypi.org/project/D-MemFS/
Original Japanese Article: バイナリ専用 FS にテキストモードを足すべきか ― 設計原則とアーキテクチャ境界の再定義 ― D-MemFS 開発戦記４

If you find this project interesting, a ⭐ on GitHub would be the best way to support my work!

[Side A] SQLite's Shared In-Memory DB Disappears — and `deserialize()` Won't Save You

Wed, 08 Apr 2026 17:14:39 +0000

From the Author:
D-MemFS was featured in Python Weekly Issue #737 (March 19, 2026) under Interesting Projects, Tools and Libraries. Being picked up by one of the most widely-read Python newsletters confirmed that in-memory I/O bottlenecks and memory management are truly universal challenges for developers everywhere. This series is my response to that interest.

🧭 About this Series: The Two Sides of Development

In Japan, I publish this series across two distinct platforms to serve different developer needs. To provide the complete picture here on Dev.to, I've brought them together as two "Sides":

Side A (Practical / originally on Qiita): Focuses on the "How". Implementation details, benchmarks, and concrete solutions for practical use cases.
Side B (Philosophy / originally on Zenn): Focuses on the "Why". The development war stories, design decisions, and how I collaborated with AI through Specification-Driven Development (SDD).

Introduction

I recently released D-MemFS — a pure Python in-memory virtual file system with zero external dependencies. While integrating it into an application backend, an idea struck me: could D-MemFS help overcome a well-known weakness of SQLite's in-memory database (:memory:)?

It turned out there was an unexpected trap waiting for me.

First: The Limits of `:memory:` and `cache=shared`

SQLite's :memory: is a convenient way to use a database entirely in-process without touching the filesystem — perfect for testing and prototyping. However, it has one critical restriction: it cannot be shared across sessions (connections).

The cache=shared option exists to solve this. Using a named shared URI like file:my_db?mode=memory&cache=shared, you can run multiple named shared in-memory databases simultaneously — a genuinely useful pattern.

But even with cache=shared, an unavoidable volatility rule applies — just like with plain :memory::

In-memory DB volatility rule:
Even with shared caching enabled, the data vanishes without a trace the moment the last connection closes.

To prevent this data loss, some systems keep a dummy connection open permanently — a functional but inelegant hack.

My thought: what if we snapshot the database state just before the connection closes, and restore it cleanly when needed? This is where Python 3.11's serialize() / deserialize() and D-MemFS enter the picture.

But that's exactly where the trap was hiding.

"Easy with `deserialize()`, Right?" — Wrong.

Python 3.11+ sqlite3.Connection objects provide serialize() and deserialize(), which allow saving and restoring database state as raw bytes — no disk I/O required. I wrote what seemed like an obvious solution:

Snapshot: Call conn.serialize() to capture state as bytes, then save to D-MemFS.
Restore: Read from D-MemFS, reconnect using the shared cache URI, call deserialize().

import sqlite3

# Reconnect using the shared cache URI and load the data
new_conn = sqlite3.connect("file:my_db?mode=memory&cache=shared", uri=True)
new_conn.deserialize(snapshot_bytes)

# Looks good! Data restored!
print(new_conn.execute("SELECT * FROM users").fetchall())
# → [(1, 'Alice'), (2, 'Bob')]

So far, so good. "That was easy," I thought.

But the moment a worker thread (or another request) tries to connect to the same shared DB, it blows up.

# Connect to the same shared DB from elsewhere
worker_conn = sqlite3.connect("file:my_db?mode=memory&cache=shared", uri=True)
worker_conn.execute("SELECT * FROM users").fetchall()

# 💥 sqlite3.OperationalError: no such table: users

"Wait — why? Where's the table?"

new_conn could read the data just fine. But worker_conn, connecting to the exact same URI, sees nothing. I did a double-take.

The Tricky Behavior of `deserialize()`

Digging into the cause, I found a subtle but significant behavior of deserialize():

When deserialize() is called, the pager (SQLite's internal memory management) behind that connection gets swapped out for a completely private, independent in-memory DB containing the loaded data.
In other words, the moment deserialize() is called, that connection silently leaves the shared cache ring.

The connection that performed the restore holds the data. But any new connection joins the original shared cache — which is now empty — and gets no such table. That was the trap.

The Workaround: Using the `backup()` API

deserialize() cannot write directly into a shared cache connection. But we still need to restore from bytes. The solution: use a "courier".

Using SQLite's long-standing backup() API, the steps are:

Read: Retrieve the snapshot bytes from D-MemFS.
Load into courier: Create a temporary :memory: connection and deserialize() the data into it.
Copy to target: Use backup() to transfer everything from the courier into the real shared cache connection.
Discard: Close the courier.

# The real shared cache DB
shared_conn = sqlite3.connect("file:my_db?mode=memory&cache=shared", uri=True)

if snapshot_bytes:
    # Prepare the courier (temporary DB) and deserialize into it
    temp_conn = sqlite3.connect(":memory:")
    temp_conn.deserialize(snapshot_bytes)

    # Copy everything from the courier to the target
    temp_conn.backup(shared_conn)
    temp_conn.close()  # Courier has served its purpose

This slightly unglamorous workaround successfully restores state into a shared cache without corrupting it, across Python 3.11 through 3.14.

Honestly: Isn't a Plain Dict Good Enough?

Having explained all of this, the sharp-eyed reader is probably thinking:

"If you're just stashing serialized bytes, why bother with a virtual filesystem? A global Dict[str, bytes] would be simpler and faster."

You're absolutely right.

A plain dict requires no external library and keeps the code simple. A management class wrapping it would be even cleaner. The advantage of a memory filesystem is thin here.

So why did I use D-MemFS? Honestly — personal preference.

Comparison	Global Dict	D-MemFS
Speed / memory efficiency	⭕ Faster and lighter	🔺 FS overhead exists
State management model	🔺 Key-string-based self-management	⭕ Intuitive paths like `/snapshots/v1.db`
Snapshot size limit	🔺 Must implement yourself	⭕ Hard Quota prevents unbounded growth / OOM
Versioning / export	🔺 Mapping and serialization is tedious	⭕ Export entire directory tree in one call

In real application development, "using SQLite" usually implies eventual persistence — writing to a physical file at a checkpoint. When that moment arrives, managing raw bytes under dict keys feels mismatched with SQLite's mental model. Having a virtual path like /snapshots/v1.db — even if it only exists in memory — fits far more naturally. And D-MemFS's hard quota guarantees that unbounded snapshot growth can never cause an OOM kill.

Added as an official use case
The deserialize() pitfall and backup() workaround were interesting enough to formalize. I wrapped this pattern into an MFSSQLiteStore class and added it to D-MemFS's official README and examples.

Even if you manage snapshots with a plain dict, the SQLite deserialize() / backup() trick itself is worth keeping in the back of your mind.

And if managing a virtual filesystem on top of a database sounds oddly appealing — give D-MemFS a try.

🔗 Links & Resources

GitHub: https://github.com/nightmarewalker/D-MemFS
PyPI: https://pypi.org/project/D-MemFS/
Original Japanese Article: PythonでSQLiteの共有インメモリDBが消える仕様と、deserialize() の落とし穴を backup() で回避した話

If you find this project interesting, a ⭐ on GitHub would be the best way to support my work!

[Side B] Pursuing OSS Quality Assurance with AI: Achieving 369 Tests, 97% Coverage, and GIL-Free Compatibility

Wed, 01 Apr 2026 22:28:02 +0000

From the Author:
Recently, I introduced D-MemFS on Reddit. The response was overwhelming, confirming that memory management and file I/O performance are truly universal challenges for developers everywhere. This series is my response to that global interest.

🧭 About this Series: The Two Sides of Development

To provide a complete picture of this project, I’ve split each update into two perspectives:

Side A (Practical / from Qiita): Implementation details, benchmarks, and technical solutions.
Side B (Philosophy / from Zenn): The development war stories, AI-collaboration, and design decisions.

Testing is a "Contract between the Design Document and the Code"

Why do we write tests? "To prevent bugs," is correct, but I want to phrase it differently.

I believe tests are a contract between the design document and the code.

In the context of "Spec-First AI Development" that I wrote about in the previous article—a method I later learned is called SDD (Specification Driven Development)—testing is the embodiment of the specification. The design document says, "If the quota is exceeded, throw an MFSQuotaExceededError." The code implements that. Testing is a mechanism to eternally verify "is it really doing exactly that?" I think software is trustworthy only when the triangle of Design Document -> Test -> Code aligns consistently.

Precisely because strict specifications (SDD) exist, AI can propose exhaustive test cases, and a human can judge their validity (supervise) against those specifications. Without a specification, there is no standard to determine whether the test cases output by the AI are "correct."

In D-MemFS, I directly derived the vast majority of the test cases from the design document. Reading through each section of the design document, I kept asking, "What should be tested to verify this specification?" Tests that can be traced back to the design document like this are called traceability.

AI Gives Us Tons of Test Cases (But Needs Supervision)

If you have Claude Opus or GitHub Copilot write test cases, they will give you a massive amount, including edge cases humans wouldn't readily think of.

For example, when asking about a simple operation like "deleting a directory":

Delete an empty directory
Delete a directory with contents
Attempt to delete a non-existent directory
Attempt to delete the root directory
While deleting, another thread tries to write to the same path
Attempt to delete using symbolic link-like paths (paths with ..)

...exhaustive cases line up like this. There’s always one where you think, "Ah, I hadn't thought of that."

However, you must not blindly trust the tests an AI writes.

An AI writes "tests that look correct." But when its understanding of the specifications is wrong, it asserts incorrect expected values. In particular, we often found it reversing "cases that should error" and "cases that should run normally." Verifying them one by one against the design document is tedious, but if you don't, it leads to a false conviction that "the tests passed = it is correct."

AI provides speed. Humans verify the direction. This division of roles worked perfectly.

Thread-Safe Tests Look Like This

Since I touted support for GIL-free Python (PYTHON_GIL=0), I had to prove it doesn't break even in multithreading.

import threading
from dmemfs import MemoryFileSystem

def stress_test():
    mfs = MemoryFileSystem(max_quota=50 * 1024 * 1024)
    errors = []

    def worker(thread_id):
        for i in range(1000):
            path = f"/thread_{thread_id}/file_{i}.txt"
            try:
                mfs.mkdir(f"/thread_{thread_id}", exist_ok=True)
                with mfs.open(path, "wb") as f:
                    f.write(f"data-{thread_id}-{i}".encode())
                with mfs.open(path, "rb") as f:
                    data = f.read()
                assert data == f"data-{thread_id}-{i}".encode()
            except Exception as e:
                errors.append(e)

    threads = [threading.Thread(target=worker, args=(t,)) for t in range(50)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert not errors, f"Thread errors: {errors}"

50 threads × 1000 iterations. Running this under PYTHON_GIL=0 (free-threaded Python) and passing was a requirement.

Why is this important? In the free-threaded mode (PYTHON_GIL=0) introduced in Python 3.13, the "implicit lock" known as the GIL disappears. In normal Python, thanks to the GIL, inter-thread conflicts rarely surface. But when the GIL is removed, if your RW lock implementation has even the slightest flaw, a race condition occurs instantly.

D-MemFS's Read/Write locks are designed as explicit locks independent of the GIL. By executing the quota verification and reservation atomically under a single lock, the typical conflict of "another thread jumping in between verification and writing" was eliminated. This design was solidified around v8 of the design document when Claude Opus's review pointed out the potential for race conditions.

Indeed, in the initial implementation, a few problems emerged in the GIL-free environment, and I fixed them. GIL-free compatibility is currently a massive topic across the entire Python community. I believe whether a library can claim to be "GIL-free safe" will become a metric of reliability going forward.

Testing the Memory Guard Was a Battle Against "Environmental Dependencies"

The Memory Guard feature (memory_guard parameter), added in v0.3.0, checks the host machine's physical memory balance. If a Hard Quota manages the "budget within the virtual FS," the Memory Guard verifies "whether the machine has the luxury to pay that budget in the first place."

What made this testing troublesome was that the available physical memory differed for every test environment. Free memory on the CI Ubuntu runners was completely different from my local Windows machine. Thus, using unittest.mock.patch, I mocked get_available_memory_bytes and artificially created situations like "a state with only 100 bytes of free memory" to test it. Securing reproducibility independent of the environment—because the design document had stated "Memory Guard relies on OS memory info" from the start, I could design the tests presupposing a mock.

What is the "Remaining 3%"?

When I write 97% coverage, I know I'll be asked "Why not 100%," so I’ll answer that in advance.

The remaining roughly 3% consists of defensive code. Parts that are essentially "theoretically unreachable, but error handling placed just in case." For example:

# Fallback when internal state breaks
# A path that will absolutely not be traversed under normal usage
if self._internal_state is None:
    raise MFSInternalError("Internal state is corrupted (Possible bug)")

To let tests reach such defensive code, you'd have to directly break the object's internal state manually, which defeats the purpose of the test. Rather, it is code that is correct because it doesn't get reached.

Rather than obsessing over 100%, I feel it's much more honest to state 97% leaving clear what "the remaining 3%" is.

A Matrix of 3 OS × 3 Python Versions

I configured my CI like this:

OS	Python 3.11	Python 3.12	Python 3.13
Ubuntu	✅	✅	✅
Windows	✅	✅	✅
macOS	✅	✅	✅

I verified that all 369 tests pass across a total of 9 environments. Python 3.13 includes execution on the free-threaded build (PYTHON_GIL=0).

Setting this up myself was subtly tedious. Tests failed particularly over the nuances of path handling between Windows and macOS (path separators, case insensitivity/sensitivity). I was correct in setting "Multi-OS support" as an initial requirement rather than settling for "just getting it to pass in Linux CI."

The Moment All Tests Turned Green

When I fixed the final failing test, ran CI, and looked back at the screenshot of the moment all 369 tests turned green, I still feel a little emotional.

If I write "I built it hand-in-hand with AI," people might assume "So in the end, the AI did everything?" But the reality is different. AI offers speed and exhaustiveness. However, it was the human that judged what was correct.

Which specifications in the design document to implement. Which expected values in a test are correct. Judging whether a bug stems from a design issue or a code flaw. Deciding "not to implement this feature." All of these decisions could only be made because there was a standard called the design document.

Treating AI like an "incredibly fast but supervise-needing rookie." Treating the design document as the "Task Specification Handout given to that rookie." This combination functioned remarkably well.

Conclusion: With a Design Doc, I Feel We Can Trust AI a Bit More

Maintain the triangle: Design Document -> Test -> Code
Leave the speed and exhaustiveness to the AI
Judgment and philosophy lie with the human

This gave birth to D-MemFS. 369 tests, 97% coverage, and zero external dependencies.

As someone who used to solely work inside Azure DevOps, I published an OSS on GitHub for the very first time. I rewrote the design document 13 times, sparred hundreds of times with AI, and reviewed the design doc every time a test failed.

It all started as "just speeding up a small script," and before I knew it, I had built something like this.

🔗 Links & Resources

GitHub: https://github.com/nightmarewalker/D-MemFS
Original Japanese Article: AIと挑むOSS品質保証：369テスト・カバレッジ97%と、GIL-free対応の裏側 ― D-MemFS 開発戦記３

If you find this project interesting, a ⭐ on GitHub would be the best way to support my work!

[Side A] Completely Defending Python from OOM Kills: The BytesIO Trap and D-MemFS 'Hard Quota' Design Philosophy

Wed, 01 Apr 2026 22:27:44 +0000

From the Author:
Recently, I introduced D-MemFS on Reddit. The response was overwhelming, confirming that memory management and file I/O performance are truly universal challenges for developers everywhere. This series is my response to that global interest.

🧭 About this Series: The Two Sides of Development

To provide a complete picture of this project, I’ve split each update into two perspectives:

Side A (Practical / from Qiita): Implementation details, benchmarks, and technical solutions.
Side B (Philosophy / from Zenn): The development war stories, AI-collaboration, and design decisions.

Introduction

If you write in-memory processing in Python, you will eventually encounter this kind of failure:

Killed

Or on Windows, the process simply vanishes without a word. It's an OOM (Out of Memory) kill. Both io.BytesIO and dict will expand limitlessly until memory runs out. The process disappears without you even knowing "where" or "why" it crashed—this is one of the most troublesome pitfalls of Python in-memory processing.

In this article, I will dig into how the Hard Quota design of D-MemFS solves this problem, right from its core design philosophy.

The Problem: BytesIO and dict Swell Limitlessly

First, let's clarify the problem.

from io import BytesIO

buf = BytesIO()

# It won't stop no matter how much you write
# It continues to succeed until physical memory runs out
for i in range(100_000):
    buf.write(b"x" * 10_000)

print(buf.tell())  # 1,000,000,000 — 1 GiB

This write does not fail. It stubbornly continues succeeding until the OS kills the process.

The same applies to dict.

vfs: dict[str, bytes] = {}
for i in range(100_000):
    vfs[f"file_{i}.bin"] = b"x" * 10_000
# No errors until 1 GiB piles up

Soft Quotas Are Not Enough

An approach like "checking the size and warning after writing" is called a soft quota. But this has a fundamental flaw—the data has already been written.

# Pseudo-implementation of a soft quota (A bad example)
MAX_BYTES = 100 * 1024 * 1024  # 100 MiB
total = 0

def soft_write(buf: BytesIO, data: bytes) -> None:
    buf.write(data)        # <- Writes first
    global total
    total += len(data)
    if total > MAX_BYTES:  # <- Notices after writing
        raise MemoryError("quota exceeded")  # Too late

The moment the threshold is exceeded, the memory has already been consumed. Furthermore, rolling back the written data after throwing an exception is not easy.

D-MemFS's Hard Quota Design

The D-MemFS quota operates on a Central Bank model. Before a write is executed, it checks the remaining quota balance and immediately rejects the write if there isn't enough.

write(data) is called
    ↓
Requests a reservation of len(data) bytes from the Quota Manager
    ↓
Is the balance sufficient?
    YES -> Decreases balance and executes write
    NO  -> raises MFSQuotaExceededError (the write never happens)

Data is never written. The file is not polluted, and you can catch the exception and continue processing.

Code Example: Actually Using the Quota

Basic Quota Settings and Exception Handling

from dmemfs import MemoryFileSystem, MFSQuotaExceededError

# 10 MiB Hard Quota
mfs = MemoryFileSystem(max_quota=10 * 1024 * 1024)
mfs.mkdir("/data")

def safe_write(mfs: MemoryFileSystem, path: str, data: bytes) -> bool:
    """Can continue processing even if writing fails"""
    try:
        with mfs.open(path, "wb") as f:
            f.write(data)
        return True
    except MFSQuotaExceededError as e:
        print(f"[Warning] Skipped writing to {path} due to quota excess: {e}")
        return False

# Success Case
safe_write(mfs, "/data/small.bin", b"x" * (1 * 1024 * 1024))   # 1 MiB → OK

# Failure Case (Exceeds quota)
safe_write(mfs, "/data/big.bin", b"x" * (20 * 1024 * 1024))    # 20 MiB → Skipped

# The file is not polluted (opening 'wb' leaves an empty file, but no data was written)
st = mfs.stat("/data/big.bin")
print(st["size"])  # 0

Processing While Checking the Remaining Quota

from dmemfs import MemoryFileSystem, MFSQuotaExceededError

QUOTA = 64 * 1024 * 1024  # 64 MiB
mfs = MemoryFileSystem(max_quota=QUOTA)
mfs.mkdir("/chunks")

def process_stream(stream, chunk_size: int = 4 * 1024 * 1024):
    """Reads a stream into memory by chunks"""
    chunk_index = 0
    written_paths = []

    for chunk in iter(lambda: stream.read(chunk_size), b""):
        path = f"/chunks/chunk_{chunk_index:04d}.bin"
        try:
            with mfs.open(path, "wb") as f:
                f.write(chunk)
            written_paths.append(path)
            chunk_index += 1
        except MFSQuotaExceededError:
            print(f"Quota reached: Kept up to {chunk_index} chunks")
            break

    return written_paths

Node Count Limit: MFSNodeLimitExceededError

You can also set a limit on the number of files (nodes). This helps to quickly detect bugs that cause the file count to explode.

from dmemfs import MemoryFileSystem, MFSNodeLimitExceededError

# Max 100 files
mfs = MemoryFileSystem(max_nodes=100)
mfs.mkdir("/logs")

for i in range(200):
    try:
        with mfs.open(f"/logs/entry_{i:04d}.log", "xb") as f:
            f.write(f"log entry {i}\n".encode())
    except MFSNodeLimitExceededError:
        print(f"Node limit reached: Stopped at {i} files")
        break

Storage Backends: SequentialMemoryFile and RandomAccessMemoryFile

D-MemFS has two types of storage backends.

SequentialMemoryFile (Sequential)

Implemented internally as a chain of byte sequences (list[bytes]).

Fast appending and reading from the beginning
Slow random access (needs to traverse chunks)
High memory efficiency (fewer allocations)

RandomAccessMemoryFile (Random Access)

Implemented internally as a bytearray.

Fast seek + read/write
Because it pre-allocates buffers during writing, doing only sequential writing might result in wasted memory.

auto-promotion (Automatic Promotion)

When default_storage="auto" (the default), it observes the file access pattern and automatically switches backends.

File created -> Starts as SequentialMemoryFile
    ↓
Random access (seek) is detected
    ↓
Automatically promoted to RandomAccessMemoryFile

# You can also explicitly pin the backend
from dmemfs import MemoryFileSystem

mfs_seq = MemoryFileSystem(default_storage="sequential")    # Always sequential
mfs_ra  = MemoryFileSystem(default_storage="random_access") # Always random access
mfs_auto = MemoryFileSystem(default_storage="auto")         # Auto (default)

promotion_hard_limit: Suppressing Promotion of Giant Files

Auto-promotion entails copying data into a bytearray upon random access. If this happens with an extremely large file, memory usage temporarily doubles.

By setting promotion_hard_limit, files exceeding this size will not automatically promote.

from dmemfs import MemoryFileSystem

# Files 64 MiB or larger will not automatically promote
mfs = MemoryFileSystem(
    max_quota=512 * 1024 * 1024,
    promotion_hard_limit=64 * 1024 * 1024,
)

In pipelines handling massive data, this is an important parameter to prevent memory spikes. In enterprise batch processing or data pipelines, this parameter acts as a safety net purposefully designed to smooth out memory spikes. Being able to strictly control the upper limit of memory usage synergizes well with K8s memory limits and CI resource constraints, tying directly into operational stability.

Memory Accounting: What is Included in the Quota

The quota tracks more than just pure data bytes.

Quota consumption = Bytes of Actual Data + Chunk Overhead

Since SequentialMemoryFile retains data in chunks, the chunk header information is slightly added as overhead. Because of this, when configuring "Quota = 10 MiB", the actual memory usage will confidently stay under 10 MiB (the actual data will be slightly less due to the overhead).

This design prioritizes the guarantee that "the quota is absolutely never exceeded".

Thread-Safe Atomic Operations

Quota updates are handled atomically under locks.

from dmemfs import MemoryFileSystem
import threading

mfs = MemoryFileSystem(max_quota=10 * 1024 * 1024)
mfs.mkdir("/concurrent")

errors = []

def writer(thread_id: int):
    for i in range(50):
        try:
            path = f"/concurrent/t{thread_id}_f{i}.bin"
            with mfs.open(path, "xb") as f:
                f.write(b"x" * (100 * 1024))  # 100 KiB each
        except Exception as e:
            errors.append(e)

threads = [threading.Thread(target=writer, args=(i,)) for i in range(10)]
for t in threads: t.start()
for t in threads: t.join()

# Excess requests over the quota yield exceptions, but the file system isn't broken
quota_errors = [e for e in errors if "quota" in str(e).lower()]
print(f"Quota exceeded: {len(quota_errors)} times (Normal behavior)")
print(f"FS Corruption: None")

If the two steps of "checking the quota and writing" are separated, a race condition could occur where another thread cuts in between the check and the write to exhaust the quota. In D-MemFS, this verification and reservation are executed under a single RW lock, completely eliminating this conflict.

A World With Hard Quotas vs. Without

	No Quota (BytesIO / dict)	D-MemFS Hard Quota
Behavior on memory exceedance	Process is OOM killed	`MFSQuotaExceededError` rises
Detection timing	Unnoticed until OS kills it	Detected instantly before write
Catching the exception	Impossible (SIGKILL)	Recoverable with `try/except`
Rollback	Impossible	Unnecessary since write hasn't happened
File integrity	May be corrupted	Guaranteed
Logging / Monitoring	Often lost	Can be logged as an exception

Quota Configuration Guidelines

Here are practical guidelines regarding what values to set.

import os
import psutil

def recommended_quota() -> int:
    """
    Example of using a certain percentage of available memory as a quota.
    In production, a fixed value is more predictable.
    """
    available = psutil.virtual_memory().available
    return int(available * 0.25)  # 25% of available memory

# Rule of thumb for actual use cases
QUOTAS = {
    "unit_test":        32 * 1024 * 1024,   # 32 MiB — For testing
    "ci_pipeline":     256 * 1024 * 1024,   # 256 MiB — CI Pipeline
    "batch_processing": 2 * 1024 * 1024 * 1024,  # 2 GiB — Batch processing
}

Basic Principle: Estimate the worst-case input size and set the quota to 1.5 - 2 times that amount. If that exceeds the total memory budget of the process, reconsider the design.

Memory Guard: Detecting Physical Memory Depletion in Advance

While a Hard Quota manages the "budget within the virtual FS," there remains another problem—when the set quota exceeds the physical memory of the host machine.

For example, even if you set max_quota=4GiB, if the machine only has 2 GiB of free memory, the OS will execute an OOM kill before reaching the quota. Hard quotas alone cannot prevent this.

The Memory Guard introduced in v0.3.0 addresses these "OOMs occurring outside the quota."

3 Modes

Mode	Behavior
`"none"`	No checks (Default, backward compatible)
`"init"`	Checks if `max_quota` exceeds available memory at FS initialization
`"per_write"`	Checks physical memory balance on every write (interval specifiable)

from dmemfs import MemoryFileSystem

# Detect insufficient memory upon initialization (Recommended)
mfs = MemoryFileSystem(
    max_quota=4 * 1024 * 1024 * 1024,     # 4 GiB
    memory_guard="init",
    memory_guard_action="raise",           # If "warn", yields ResourceWarning
)

# Check per write (For stricter use cases)
mfs = MemoryFileSystem(
    max_quota=4 * 1024 * 1024 * 1024,
    memory_guard="per_write",
    memory_guard_action="warn",
    memory_guard_interval=1.0,             # Check interval in seconds
)

The Relationship Between Hard Quotas and Memory Guard

It might be easier to understand with an analogy of a house.

Hard Quota = The area of a room. A limit on how much baggage you can place.
Memory Guard = The building's load-bearing limit check. Confirming whether the building can withstand that weight in the first place.

Only when both are present is the safety of in-memory processing truly complete.

Design Ingenuity of "per_write" Mode

Since the "per_write" mode queries the OS for physical memory balance every time, there are concerns about performance impact. To address this, the memory_guard_interval parameter can control the check interval. The default is 1 second—if 1 second hasn't passed since the last check, it uses the cached value.

# Secures safety while maintaining performance even with high-frequency writes
mfs = MemoryFileSystem(
    max_quota=1 * 1024 * 1024 * 1024,
    memory_guard="per_write",
    memory_guard_action="raise",
    memory_guard_interval=2.0,   # Checks every 2 seconds
)

The guarantee of the Hard Quota that "it absolutely never exceeds the quota", and the guarantee of the Memory Guard that "it won't keep running while physical memory is lacking". This dual defense is the complete picture of D-MemFS's OOM countermeasures.

Behavior in free-threaded Python (GIL=0)

In the free-threaded mode (python3.13t) introduced in Python 3.13 onwards, there is no GIL, making thread conflicts more surface-level. D-MemFS has been tested in GIL=0 environments (369 tests × 3 OS × 3 Python versions), and quota atomicity is guaranteed by explicit locks irrelevant of the GIL.

# Testing in free-threaded Python
python3.13t -c "
from dmemfs import MemoryFileSystem
import threading

mfs = MemoryFileSystem(max_quota=5 * 1024 * 1024)
mfs.mkdir('/test')

def worker(n):
    for i in range(100):
        try:
            with mfs.open(f'/test/w{n}_{i}.bin', 'xb') as f:
                f.write(b'x' * 10240)
        except Exception:
            pass

threads = [threading.Thread(target=worker, args=(i,)) for i in range(20)]
for t in threads: t.start()
for t in threads: t.join()
print('Completed (No crashes)')
"

Conclusion

OOM is a failure that is immensely difficult to debug. Staff traces are rarely left behind, and it's hard to identify which code is the cause. By "proactively rejecting writes that don't fit in the budget," Hard Quotas convert this problem into a catchable exception.

D-MemFS's quota design is based on the philosophy of "No Surprises." Memory usage will never exceed the configured limit, exceptions can be handled with try/except, and the integrity of the file system is always maintained.

If you have ever experienced an OOM failure in in-memory processing, please do give it a try.

pip install D-MemFS

GitHub: https://github.com/nightmarewalker/D-MemFS
PyPI: https://pypi.org/project/D-MemFS/

🔗 Links & Resources

GitHub: https://github.com/nightmarewalker/D-MemFS
Original Japanese Article: PythonのOOMキルを完全防御する：BytesIOの罠とD-MemFS「ハードクォータ」の設計思想

If you find this project interesting, a ⭐ on GitHub would be the best way to support my work!

[Side B] Breaking Free from Vibe Coding Fatigue: A Practical Record of Building an OSS with 'Spec-First AI Development'

Wed, 25 Mar 2026 22:11:30 +0000

From the Author:
D-MemFS was featured in Python Weekly Issue #737 (March 19, 2026) under Interesting Projects, Tools and Libraries. Being picked up by one of the most widely-read Python newsletters confirmed that in-memory I/O bottlenecks and memory management are truly universal challenges for developers everywhere. This series is my response to that interest.

🧭 About this Series: The Two Sides of Development

To provide a complete picture of this project, I’ve split each update into two perspectives:

Side A (Practical / from Qiita): Implementation details, benchmarks, and technical solutions.
Side B (Philosophy / from Zenn): The development war stories, AI-collaboration, and design decisions.

Are You Realizing the Limits of "Vibe Coding"?

Having AI write code for us has become the norm. Throw a prompt at it, and it returns plausible code. It runs. The tests pass. It's convenient.

However, if you continue this way of "having AI generate code based on a vague vibe"—often called Vibe Coding—you will inevitably hit a wall at some point.

You don't understand the underlying principles of the code the AI generated.
Trying to fix a bug creates another bug.
You want to refactor, but have no standard for what must be protected.
In code reviews, you can't answer "Why is it done this way?"

This is Vibe Coding Fatigue. By continually accepting AI's output, the codebase eventually slips out of your control.

So, what should you do?

My answer was: "Write the design document before the code." Inspired by the idea "Write the specifications first," which I had heard somewhere last year, I arbitrarily called this "Spec-First AI Development." Before letting the AI write any code, we thoroughly iron out the specifications and design documents.

Why "Design Documents Before Code"?

When you suddenly let an AI write code, the following problems arise:

Problem 1: Reviewing AI-generated code is too difficult for humans.

AI yields hundreds of lines of code in an instant. But the human reviewing it must read and understand every single line. AI code frequently "works but the intention is unreadable." This is because the choices of variables and logic don't reflect human design intent.

On the other hand, reviewing a design document is overwhelmingly easier. If it says, "Throw an exception before writing if the quota is exceeded," you can judge the validity of that statement as plain language. What's more, you can even have the AI itself review the design document.

Problem 2: Code alone cannot guarantee "Design Philosophy."

Code states "what to do," but not "why it does so." When adding a new feature, AI might suggest an implementation that violates the existing design philosophy. If the philosophy is explicitly stated in the design document, you can instruct the AI to "think of a method based on the design philosophy." The design document becomes a guardrail.

For D-MemFS, I explicitly stated design philosophies like "Zero External Dependencies," "Safety First," and "Binary-Exclusive FS" in the design document. Even if AI suggested using a convenient external library, I could reject it with a single phrase: "It goes against the design philosophy."

Problem 3: Context Explosion.

Basic design documents, detailed design documents, test design documents—if you separate documents by phase or purpose, you don't have to make the implementing AI read all the documents. When implementing tests, you just need to hand over the test design document. The AI's context window is finite. If you flood it with irrelevant information, the accuracy of the crucial parts drops.

And the biggest advantage: Human understanding advances.

Through the process of bouncing ideas off the AI to refine the design document, the human's system comprehension deepens. "What should this class do?", "Where does this error occur?", "Is this operation thread-safe?"—you begin to hold the answers to these questions in your own mind. Consequently, the code output by the AI becomes remarkably easier to read. You can read it because you understand it.

Using Browser AIs as Sparring Partners

Let me talk about what I actually did to create the design documents.

The first thing I did was brush up on the idea. I uploaded the source file of the memory FS feature I had embedded in my personal app to both Gemini and ChatGPT. A sparring session started: "If I were to carve this out as a library, what kind of design would be best?"

I summarized the sparring contents into Markdown and handed it to another AI. Spurred on by that, we sparred some more. Of course, I relentlessly interjected with my own questions.

Once the ideas were somewhat aligned, I had one of the AIs write the design document. From here on, they naturally split into the role of modifying the design document and the role of reviewing.

:::details [Instruction to Gemini]
Based on the following ideas and requirements,
please write a design document for a Python in-memory file system library.
Requirements: Quota management, Hierarchical directories, Thread-safe, No external dependencies
[Paste the Markdown of ideas]
:::

Gemini outputs a design document (v1). I throw that straight into ChatGPT.

:::details [Instruction to ChatGPT]
Please critique the following design document.
List problems, omissions, and design inconsistencies as much as possible.
[Paste Design Document v1]
:::

ChatGPT lists the problems. I hand that over to Gemini to let it fix them. Then I evaluate it with ChatGPT again. I repeated this back-and-forth. Sometimes I hit Gemini's Pro usage limits.

And I myself also relentlessly kept putting in my two cents.

"Why does this API look like this?", "Are these error types sufficient?", "What happens during concurrent writes?", "When is the quota calculated?" If a vague answer returned, I kept pressing for "more specifics." I didn't let "something roughly like this" slide.

There is one thing to note. If you keep sparring over technical details, the context fills up rapidly, and the AI's responses gradually get weird. A dark descent, so to speak. Before that happens, you have the AI generate a handover message and switch to a new chat. I did this "chat switching" numerous times.

Claude Opus Overturned Everything

When the design document had somewhat come together (around v7-v8), I asked Claude Opus to review it.

What came back was a massive amount of feedback.

Problems I couldn't catch in sparring with Gemini and ChatGPT surfaced one after another. Issues with lock granularity, naive rollback design upon quota excess, omissions in path normalization rules, latent race conditions in the asynchronous wrapper design...

It was a moment I realized, "This is fundamentally much harder than I thought."

From here, we repeated: modify → Opus review → modify → Sparring → modify... and the design document eventually reached v13. Note that I started implementation around v11 or v12.

Never Bend the Design Philosophy

Along the way, the topic of "Wouldn't it be convenient to add this feature too?" came up many times. Every single time, I returned to this question:

"Does this align with the original design philosophy?"

The design philosophy for D-MemFS was set from the beginning:

Zero External Dependencies (stdlib only)
Safety First (No dangerous operations provided)
Not just a buffer, but a real FS with a hierarchical structure
Direct extraction support for Archives (ZIP)

Any proposal conflicting with these principles, no matter how convenient it seemed, was entirely dropped.

I insisted on zero external dependencies because I had seen the failure of pyfilesystem2. It stopped working due to a single change in setuptools. Relying on a library means adopting that library's risks wholesale. By using only the stdlib, it runs as long as Python runs.

"Safety First" concretely manifests in decisions like "Not providing operations to wipe the entire file system" and "Rejecting operations that could result in path traversal attacks by default." It's better not to possess convenient but dangerous features from the start.

Looking Back, This Was SDD

To summarize, this is what I did:

1. Spar ideas with AI and write design docs (use multiple AIs)
2. Human thoroughly reviews the design docs
3. Generate code ONLY after the design doc is solidified
4. If a problem is found, fix the design doc first
5. Only after fixing the design doc, fix the code

I learned recently that this is precisely the method known in the industry as SDD (Specification Driven Development). While I casually labeled it "Spec-First AI Development" earlier, it seems it already had a proper name. While Vibe Coding is "letting AI write code by vibe," SDD is "controlling AI through specifications." What I was doing was practicing this SDD in collaboration with AI.

The critical points are 4 and 5. When a problem is found during the implementation phase, it's easy to just fix the code first. But doing so causes the design document and the code to drift apart. The design document turns into "a dream we wrote down at the start," misaligned with reality.

Therefore, when a problem is found, return to the design document first. Fix the design document saying, "This specification should be changed like this," and then fix the code according to that modification. The design document is permanently the "source of truth," and the code follows it. Maintaining this sequence was vital.

By doing so, the evaluation criteria for the code written by AI transforms from "Does it run?" to "Does it correctly implement the intentions of the design document?" I believe this is the framework for "using AI while trusting it."

In my personal opinion, I even feel that the design document is more important than the code. As long as the design document is correct, the code can be rewritten infinitely. Whether you let an AI write it or write it by hand, having the standard of a design document allows you to evaluate "if it's correct." Conversely, without a design doc, the standard itself to judge whether the code is correct doesn't exist.

"It Got Bigger Than I Expected"

To be formal about it, I initially envisioned something much smaller. My intention was just to "tidy up the prototype a bit and turn it into a library."

But the more I wrote the design document, the more things I realized I had to consider. If I seriously consider thread safety, I realize I need RW locks. By implementing RW locks, I naturally want to check if it runs properly in GIL-free Python (PYTHON_GIL=0). To check that, I need tests, and writing tests exposes the flaws in the design...

Before I knew it, what emerged was a library running purely on the standard library, supporting hierarchical directories, equipped with quota management, RW-locked, harboring an asynchronous wrapper, supporting free-threaded Python, with 369 tests and 97% coverage.

In v0.3.0, even a Memory Guard was introduced. While Hard Quotas manage the "budget within the virtual FS," the Memory Guard checks the host machine's physical memory and rejects writes proactively if space cannot be secured. Given that an OS will execute an OOM kill before reaching the quota if the set quota exceeds the machine's free memory, this feature became a logical necessity. Having written in the design document that "the quota will absolutely not be exceeded," I could no longer ignore the possibility of death outside the quota — this too is a consequence of "Spec-First." (The technical details will be discussed in the 3rd article.)

The scary part of being Spec-First might be that if you try to do it right, you actually end up creating something truly solid.

🔗 Links & Resources

GitHub: https://github.com/nightmarewalker/D-MemFS
Original Japanese Article: バイブコーディング消耗からの脱却：「スペックファーストAI開発」でOSSを作った実践記録 ― D-MemFS 開発戦記２

If you find this project interesting, a ⭐ on GitHub would be the best way to support my work!

[Side A] Building a RAM Disk on Windows without Admin Privileges — Python I/O Acceleration Techniques

Wed, 25 Mar 2026 22:11:19 +0000

From the Author:
D-MemFS was featured in Python Weekly Issue #737 (March 19, 2026) under Interesting Projects, Tools and Libraries. Being picked up by one of the most widely-read Python newsletters confirmed that in-memory I/O bottlenecks and memory management are truly universal challenges for developers everywhere. This series is my response to that interest.

🧭 About this Series: The Two Sides of Development

To provide a complete picture of this project, I’ve split each update into two perspectives:

Side A (Practical / from Qiita): Implementation details, benchmarks, and technical solutions.
Side B (Philosophy / from Zenn): The development war stories, AI-collaboration, and design decisions.

Introduction

When you want to use a RAM disk on Windows, tools like ImDisk Toolkit or OSFMount are usually the first candidates that come to mind. However, they share a common restriction—they require administrative privileges, or a paid license. Often, you can't use them in CI environments or on shared development machines.

But if you slightly change your perspective, another approach emerges.

Isn't the memory of the Python process itself a RAM disk from the perspective of the Python code?

In this article, I will introduce how to leverage D-MemFS as a "Python-exclusive software RAM disk" that requires neither drivers nor administrative privileges.

The Conclusion First: Comparison of Methods

Let's first summarize "which method you should choose."

Comparison Item	`tempfile` on SSD	RAM Disk (ImDisk, etc.)	D-MemFS
Admin Privileges	Not Required	Required	Not Required
External Tools	Not Required	Required	Not Required (pip only)
Volatile (Auto-delete)	△ Manual Delete	△ Lost on Reboot	✅ Auto-collected by GC
Cross-Platform	✅	❌ Windows Only	✅ Win/Mac/Linux
Access from Other Processes	✅	✅	❌ Python Internal Only
Memory Limit Management	❌	❌	✅ Hard Quotas + Memory Guard
Sequential I/O	~1.9 GB/s	~2.0 GB/s	~1.9 GB/s
Random Access I/O	~1.4 GB/s	~1.3 GB/s	~1.4 GB/s
Usage in CI Environments	✅	❌ Permission Wall	✅

Conclusion: If it's pure Python processing without the need to call external commands, D-MemFS is the easiest and fastest choice.

💡 v0.3.0 New Feature: Memory Guard
While Hard Quotas manage the "budget within the virtual FS," the Memory Guard introduced in v0.3.0 checks the host machine's remaining physical memory. If there isn't enough memory, it rejects the write beforehand. Even if you set a quota of 4 GiB, it's meaningless if the machine only has 2 GiB free—Memory Guard prevents this issue. Details will be covered in Side A - Part 3: Design Philosophy of Hard Quotas.

Premise: Why Do We Want a RAM Disk?

The primary uses for a RAM disk are twofold:

High-speed temporary file processing — To speed up processing by eliminating disk I/O latency.
Avoiding writes to disk — To protect SSD write endurance, and to avoid leaving sensitive data behind.

Both of these are very common requirements in Python testing, CI, and data processing pipelines.

D-MemFS: Python-Exclusive Software RAM Disk

D-MemFS (dmemfs) is a pure Python in-memory file system library.

pip install D-MemFS

Standard library only (Zero external dependencies)
No administrative privileges required
Runs on Windows / macOS / Linux
Automatically deleted when the process terminates (volatile)

Replacing tempfile with D-MemFS

The most typical pattern is replacing tempfile.TemporaryDirectory().

Before Replacement (Writes to disk occur)

import tempfile
import os

def process_data(raw: bytes) -> bytes:
    with tempfile.TemporaryDirectory() as tmpdir:
        input_path = os.path.join(tmpdir, "input.bin")
        output_path = os.path.join(tmpdir, "output.bin")

        with open(input_path, "wb") as f:
            f.write(raw)

        # Some processing (an example assuming calling an external command)
        data = open(input_path, "rb").read()
        result = bytes(b ^ 0xFF for b in data)  # Dummy conversion

        with open(output_path, "wb") as f:
            f.write(result)

        return open(output_path, "rb").read()

After Replacement (Completely contained in memory)

from dmemfs import MemoryFileSystem

def process_data(raw: bytes) -> bytes:
    mfs = MemoryFileSystem()
    mfs.mkdir("/tmp")

    with mfs.open("/tmp/input.bin", "wb") as f:
        f.write(raw)

    with mfs.open("/tmp/input.bin", "rb") as f:
        data = f.read()

    result = bytes(b ^ 0xFF for b in data)

    with mfs.open("/tmp/output.bin", "wb") as f:
        f.write(result)

    with mfs.open("/tmp/output.bin", "rb") as f:
        return f.read()

Access to the disk becomes zero. When mfs goes out of scope, it is collected by the GC, so no clean-up is necessary.

Automatic Release via GC

MemoryFileSystem is collected by the GC when it exits the scope. You can use it similarly to TemporaryDirectory.

from dmemfs import MemoryFileSystem

def process():
    mfs = MemoryFileSystem(max_quota=64 * 1024 * 1024)
    mfs.mkdir("/work")

    with mfs.open("/work/data.csv", "wb") as f:
        f.write(b"id,value\n1,100\n2,200\n")

    with mfs.open("/work/data.csv", "rb") as f:
        print(f.read().decode())

process()
# When exiting the function, mfs is collected by GC, and all contents are wiped.

Benchmarks

The following are benchmark results comparing D-MemFS, tempfile on a RAM Disk, and tempfile on an SSD (repeat=5, warmup=1).

Measurement Environment: Windows, X:\TEMP (RAM Disk) / C:\TempX (SSD)
Throughput values are calculated from 512 MiB of write + read.

Mass read/write of small files (300 files × 4 KiB)

Method	Time (ms)
`io.BytesIO`	6
D-MemFS	51
tempfile (on RAM Disk)	207
tempfile (on SSD)	267

Stream read/write (16 MiB, 64 KiB chunks)

Method	Time (ms)
tempfile (on RAM Disk)	20
tempfile (on SSD)	21
`io.BytesIO`	62
D-MemFS	81

Random Access (16 MiB)

Method	Time (ms)
D-MemFS	34
tempfile (on SSD)	35
tempfile (on RAM Disk)	37
`io.BytesIO`	82

Large Capacity Stream read/write (512 MiB, 1 MiB chunks)

Method	Time (ms)
D-MemFS	529
tempfile (on RAM Disk)	514
tempfile (on SSD)	541
`io.BytesIO`	2 258

Random reading of numerous files (10,000 files × 4 KiB)

Method	Time (ms)
D-MemFS	1 280
tempfile (on RAM Disk)	6 310
tempfile (on SSD)	8 601

Key Points:

For small and numerous files, D-MemFS is overwhelmingly faster. Because the file open/close overhead is practically zero, it's 4 times faster with 300 files and over 5 times faster for random reads of 10,000 files.
For large capacity streams (512 MiB+, large chunks), D-MemFS and tempfile are equivalent. Because the memory bandwidth becomes the bottleneck, the difference in storage location hardly shows.
For small to medium capacity streams around 16 MiB, tempfile is faster. The per-chunk overhead is relatively higher for D-MemFS, so the finer the chunk size, the wider the gap.
io.BytesIO is the fastest for single-stream usage, but lacks file management features (paths, directories, quotas).

Using in Asynchronous Code (AsyncMemoryFileSystem)

For asyncio-based code, use AsyncMemoryFileSystem.

import asyncio
from dmemfs import AsyncMemoryFileSystem

async def main():
    mfs = AsyncMemoryFileSystem(max_quota=32 * 1024 * 1024)
    await mfs.mkdir("/data")

    # Asynchronous write
    async with await mfs.open("/data/result.bin", "wb") as f:
        await f.write(b"async result data\n")

    # Asynchronous read
    async with await mfs.open("/data/result.bin", "rb") as f:
        content = await f.read()
        print(content)

asyncio.run(main())

Internally, it incorporates exclusive control based on asyncio.Lock, making it safe to access simultaneously from multiple coroutines.

Example: Parallel Downloads and Immediate Processing

import asyncio
import aiohttp
from dmemfs import AsyncMemoryFileSystem

async def fetch_and_process(session, url: str, mfs, path: str):
    async with session.get(url) as resp:
        data = await resp.read()
    async with await mfs.open(path, "wb") as f:
        await f.write(data)

async def main():
    urls = [
        ("https://example.com/a.json", "/cache/a.json"),
        ("https://example.com/b.json", "/cache/b.json"),
        ("https://example.com/c.json", "/cache/c.json"),
    ]

    mfs = AsyncMemoryFileSystem(max_quota=64 * 1024 * 1024)
    await mfs.mkdir("/cache")

    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[
            fetch_and_process(session, url, mfs, path)
            for url, path in urls
        ])

    # Process all files while they are readily available in memory
    for _, path in urls:
        async with await mfs.open(path, "rb") as f:
            data = await f.read()
            print(f"{path}: {len(data)} bytes")

asyncio.run(main())

Limitations

While D-MemFS is extremely useful as a software RAM disk, it has fundamental limitations compared to a real RAM disk.

Limitation	Description
Restricted to within the Python Process	It cannot be accessed from other processes. It cannot be used for external commands or subprocess.
Volatile	When the process ends, all contents disappear (it does not persist until explicitly written out).
Cannot be Mounted	It cannot be exposed to the OS as a drive letter or mount point.
Not compatible with `os.PathLike` (Intentional)	A design decision to prevent confusing virtual paths with host paths.

Use Cases

Use Case	Description
Testing	For tests involving the file system using `pytest`. As a replacement for the `tmp_path` fixture.
CI	Speeding up builds on GitHub Actions or Azure Pipelines by reducing disk I/O.
Data Processing Pipelines	Handling all intermediate files entirely in memory.
Archive Extraction/Processing	Extracting zip/tar archives into memory and processing the internal files without writing to disk.
Dealing with Sensitive Data	Temporary processing where you don't want to leave any footprints on disk.
Mocking without External Dependencies	Intercepting file system access during testing.

Extracting Archives into Memory

This is an example of extracting a zip file without writing it to disk and processing the internal files.

import io
import zipfile
from dmemfs import MemoryFileSystem

def process_zip(zip_bytes: bytes) -> dict[str, int]:
    """Extract contents of the zip to MFS in memory, returning the size of each file."""
    mfs = MemoryFileSystem(max_quota=256 * 1024 * 1024)
    mfs.mkdir("/extracted")

    with zipfile.ZipFile(io.BytesIO(zip_bytes)) as zf:
        for name in zf.namelist():
            data = zf.read(name)
            with mfs.open(f"/extracted/{name}", "wb") as f:
                f.write(data)

    # Process files on MFS (here returning a list of sizes)
    result = {}
    for entry in mfs.listdir("/extracted"):
        path = f"/extracted/{entry}"
        result[entry] = mfs.stat(path)["size"]
    return result

You can do similar things with tar.gz.

import io
import tarfile
from dmemfs import MemoryFileSystem

def process_targz(targz_bytes: bytes) -> list[str]:
    """Extract tar.gz into MFS in memory and return the file list."""
    mfs = MemoryFileSystem(max_quota=512 * 1024 * 1024)
    mfs.mkdir("/extracted")

    with tarfile.open(fileobj=io.BytesIO(targz_bytes), mode="r:gz") as tf:
        for member in tf.getmembers():
            if member.isfile():
                f = tf.extractfile(member)
                if f is not None:
                    # Reproduce directory hierarchy
                    parts = member.name.split("/")
                    for depth in range(1, len(parts)):
                        d = "/extracted/" + "/".join(parts[:depth])
                        try:
                            mfs.mkdir(d)
                        except FileExistsError:
                            pass
                    with mfs.open(f"/extracted/{member.name}", "wb") as out:
                        out.write(f.read())

    return mfs.listdir("/extracted")

The key point is that writing to disk is zero. Compared to the traditional pattern of extracting downloaded archives to a temporary directory before processing, you won't wear down the write endurance of your SSD.

Using as a pytest Fixture

import pytest
from dmemfs import MemoryFileSystem

@pytest.fixture
def mfs():
    """Provides an independent in-memory FS for each test."""
    return MemoryFileSystem(max_quota=32 * 1024 * 1024)

def test_write_and_read(mfs):
    mfs.mkdir("/work")
    with mfs.open("/work/test.txt", "wb") as f:
        f.write(b"test data")
    with mfs.open("/work/test.txt", "rb") as f:
        assert f.read() == b"test data"

def test_quota_is_isolated(mfs):
    """Each test has an independent quota."""
    mfs.mkdir("/data")
    with mfs.open("/data/file.bin", "wb") as f:
        f.write(b"x" * 1024)
    st = mfs.stat("/data/file.bin")
    assert st["size"] == 1024

Conclusion

The problem of needing administrative privileges for RAM disks on Windows can be completely avoided within Python code by using D-MemFS. Just by running pip install D-MemFS and replacing TemporaryDirectory, you can gain a dramatic speed improvement over using tempfile on an SSD.

For pure Python processing that doesn't rely on external commands, D-MemFS is the easiest and fastest choice.

GitHub: https://github.com/nightmarewalker/D-MemFS
PyPI: https://pypi.org/project/D-MemFS/

🔗 Links & Resources

GitHub: https://github.com/nightmarewalker/D-MemFS
Original Japanese Article: 管理者権限不要でWindowsにRAMディスクを構築する ― Python環境のI/O高速化手法

If you find this project interesting, a ⭐ on GitHub would be the best way to support my work!

[Side B] Have you ever wanted to extract a ZIP file in memory? I have.

Tue, 17 Mar 2026 16:48:24 +0000

Author's Note:
I am a software engineer based in Japan. This article is an English translation (with AI assistance for clarity) of a development chronicle originally written for the Japanese developer community platform, Zenn.
Recently, I introduced my project, D-MemFS, on Reddit (r/Python), where it sparked intense architectural discussions. This response confirmed that in-memory I/O bottlenecks and OOM crashes are truly universal pain points for developers everywhere. Therefore, I decided to cross the language barrier and share these insights globally.

🧭 About this Series: The Two Sides of Development

In Japan, I publish this series across two distinct platforms to serve different developer needs. To provide the complete picture here on Dev.to, I've brought them together as two "Sides":

Side A (Practical / originally on Qiita): Focuses on the "How". Implementation details, benchmarks, and concrete solutions for practical use cases.
Side B (Philosophy / originally on Zenn): Focuses on the "Why". The development war stories, design decisions, and how I collaborated with AI through Specification-Driven Development (SDD).

15 to 20 Minutes. Every Single Time.

There is a Python desktop application I built for business use. I developed it about three years ago, and I have been modifying and using it ever since. One of its features is a process to "download a ZIP from the internet, extract it, and place the contents in appropriate locations."

The problem was the test waiting time that occurred every time I touched this app.

When you modify code, naturally, you test it. A single trial of that test took 15 to 20 minutes. Whether it was adding a feature, fixing a bug, or changing a specification, I had to wait over 15 minutes for every single operation check. Modify, wait, check the results, fix again, wait again. During development, this waiting time occurred over and over.

The bottleneck was obvious without even investigating.

Extracting the ZIP and copying files.

The downloaded file is saved to storage, extracted on that storage, and then copied to the appropriate location (including overwrites). It's entirely disk I/O. Of course it takes time.

"Wouldn't this be a lot faster if it were all done in memory?"

I Tried PyFilesystem2

The very first thing that came to mind was pyfilesystem2. The name itself suggests "doing file system-like things in Python," and it’s supposed to have memory FS features.

I installed it, wrote the code, and executed it.

UserWarning: pkg_resources is deprecated ...

A mysterious warning appeared. I looked into it and found that the deprecation of pkg_resources in setuptools 82 caused pyfilesystem2 to stop working properly. Looking at Issue #597, it has been left open since late 2024. It seems maintenance has stalled.

It worked when I started development, but before I knew it, it was broken. This is a classic external library story.

I Looked Elsewhere, But Nothing Clicked

I checked other libraries too.
However, including pyfilesystem2, they either had excessively too many features for what I was looking for, or lacked necessary features.

The thing that bothered me the most was the lack of quotas (capacity limits). Running in memory means that if it runs out of control, the process will immediately die from OOM (Out-of-Memory). Since I was developing this app with the intention of eventually letting end-users use it directly, having it "eat up all the memory and crash when fed invalid data" was out of the question. I wanted it to throw an error with a limit like "Up to 100MB max." I couldn't find a library that had that kind of feature.

So I Decided to Build It Myself (Just Out of Curiosity)

I thought, "Then I'll just build a prototype and embed it directly into the app." Just out of curiosity.

Within a few hours, I had something that looked the part. The flow was: receive the ZIP with BytesIO, extract it into an in-memory directory structure, and read from there.

I ran it.

It was fast.

It finished before I could even finish drinking a cup of coffee. Calculating the time it took from the logs, it now fit within under 2 minutes. From 15-20 minutes down to less than 2 minutes. For the exact same process. Waiting 15 minutes for every test during development now ended in 2 minutes. This alone drastically changed the development experience.

I slightly suspected, "Isn't this cheating?" but thinking about it, it's obvious: it gets faster when you eliminate the round trips of writing to disk and reading it back again.

"Why Not Release It as a Library?"

After being satisfied embedding it in my app, I suddenly remembered.
"Wait, pyfilesystem2 is broken right now. Aren't there other people struggling with the exact same thing?"

Looking again from the perspective of a memory FS, there are a few memory FS-related libraries on PyPI, but they are all unmaintained or have limited features. If I build this properly and publish it, there might be demand for it.

Before I knew it, I started writing a design document, and I found I had rewritten it 13 times.

Here's What It Looks Like To Use

When you use the actually published D-MemFS (PyPI: dmemfs), it looks like this. The core MemoryFileSystem class implementation is in dmemfs/_fs.py, so feel free to check that out as well if you're interested.

import zipfile
import io
from dmemfs import MemoryFileSystem

# The byte sequence of a ZIP (assuming received from the network)
zip_bytes = b"..."  # requests.get(...).content, etc.

# Extract in memory (import_tree automatically creates parent directories)
mfs = MemoryFileSystem(max_quota=100 * 1024 * 1024)  # 100MB limit
with zipfile.ZipFile(io.BytesIO(zip_bytes)) as zf:
    tree = {f"/{name}": zf.read(name) for name in zf.namelist() if not name.endswith("/")}
    mfs.import_tree(tree)

# You can read it just like a normal file system
with mfs.open("/data/report.csv", "rb") as f:
    for line in f.read().decode("utf-8").splitlines():
        print(line)

If the quota is exceeded, an MFSQuotaExceededError is raised. This prevents "accidentally eating up all the memory."

from dmemfs import MemoryFileSystem, MFSQuotaExceededError

mfs = MemoryFileSystem(max_quota=1024)  # 1KB limit (for testing)
try:
    with mfs.open("/huge.bin", "wb") as f:
        f.write(b"x" * 2048)
except MFSQuotaExceededError as e:
    print(f"Capacity exceeded: {e}")

And So, My First OSS Release

When I built the prototype, my intention was merely to "just embed it in my own app." And yet, I rewrote the design document countless times, wrote an insane amount of tests, and even tested it on a Mac in CI which I had never used before...

For someone who had managed code in private repositories all along, placing what I had built myself into a "sort of public square" like GitHub takes a fair amount of courage.

Having been a systems engineer for a long time, this was my somewhat delayed OSS debut.
It's quite deeply moving.

🔗 Links & Resources

GitHub: https://github.com/nightmarewalker/D-MemFS
Original Japanese Article: インメモリで ZIP を展開したいと思ったことはありますか？私はあります。― D-MemFS 開発戦記１

If you find this project interesting, a ⭐ on GitHub would be the best way to support my work!

[Side A] Why BytesIO Isn't Enough — Building a Python In-Memory FS Library

Tue, 17 Mar 2026 16:47:56 +0000

Author's Note:
I am a software engineer based in Japan. This article is an English translation (with AI assistance for clarity) of a practical technical guide originally written for the Japanese developer community platform, Qiita.
Recently, I introduced my project, D-MemFS, on Reddit (r/Python), where it sparked intense discussions. This response confirmed that in-memory I/O bottlenecks and OOM crashes are truly universal pain points for developers everywhere. Therefore, I decided to cross the language barrier and share these concrete solutions globally.

🧭 About this Series: The Two Sides of Development

In Japan, I publish this series across two distinct platforms to serve different developer needs. To provide the complete picture here on Dev.to, I've brought them together as two "Sides":

Side A (Practical / originally on Qiita): Focuses on the "How". Implementation details, benchmarks, and concrete solutions for practical use cases.
Side B (Philosophy / originally on Zenn): Focuses on the "Why". The development war stories, design decisions, and how I collaborated with AI through Specification-Driven Development (SDD).

Introduction

I kept running into the exact same problem across different projects.

Whether I was trying to mock a file system for testing, or handle temporary files in a CI pipeline without touching the physical disk, my first instinct was always to reach for io.BytesIO. It feels like the standard, "Pythonic" way to handle in-memory data.

But let's be honest: the moment you try to do something even slightly complex—like handling multiple files or directories—BytesIO quickly shows its limitations and starts making your life miserable.

In this article, I will categorize exactly why BytesIO falls short for these use cases, and introduce D-MemFS—a pure Python in-memory file system library I built to solve these exact frustrations.

The Limitations of BytesIO

It's Just a Single Buffer

io.BytesIO provides a read/write interface for a single mutable byte sequence. While it feels like handling a file, it is ultimately just "a single buffer."

from io import BytesIO

buf = BytesIO()
buf.write(b"hello, world")
buf.seek(0)
print(buf.read())  # b'hello, world'

Where this falls short is when you want to handle multiple files.

Attempting to Substitute with `dict[str, BytesIO]`...

When developers want to handle multiple files in memory, a common stopgap measure is using a dictionary. I've seen this workaround in countless codebases (and yes, I'm fully guilty of writing it myself in the past).

from io import BytesIO

# A simple in-memory "file system"
vfs: dict[str, BytesIO] = {}

def vfs_write(path: str, data: bytes) -> None:
    buf = BytesIO(data)
    vfs[path] = buf

def vfs_read(path: str) -> bytes:
    buf = vfs[path]
    buf.seek(0)
    return buf.read()

vfs_write("config/settings.json", b'{"debug": true}')
vfs_write("data/input.csv", b"id,name\n1,Alice\n")
print(vfs_read("config/settings.json"))

This looks like it works at first glance, but problems arise immediately.

Desired Feature	Situation with `dict[str, BytesIO]`
Directory creation & listing	Must pseudo-implement using key prefixes
Deleting a sub-tree	Must manually implement `{k: v for k, v in vfs.items() if not k.startswith(prefix)}`
Memory limits (Quota)	None. Memory piles up infinitely
Thread-safe reads/writes	None. Must write your own locks
File stat (size, modified time)	Must manage it manually
Append mode	Must manually `buf.seek(0, 2)`

If you seriously try to implement a directory structure, you end up in a state where you are essentially "building your own file system." Moreover, whoever does this will inevitably introduce subtle bugs.

Another Pitfall: Memory Swells Infinitely

When processing massive amounts of data in memory, BytesIO cannot set a memory limit. You won't notice until the process crashes with an Out-Of-Memory (OOM) error. It becomes a breeding ground for troubles that don't reproduce in the test environment but only crash in production.

I Built D-MemFS

To solve all of the above problems entirely, I built D-MemFS.

Zero external dependencies (standard library only)
Hierarchical directory structure
Hard Quotas (rejects writes before memory is allocated)
Thread-safe via RW locks
Asynchronous wrappers (AsyncMemoryFileSystem)

pip install D-MemFS

Basic Usage

mkdir / open / write / read

from dmemfs import MemoryFileSystem

mfs = MemoryFileSystem()

# Create a directory
mfs.mkdir("/work/data")

# Write to a file
with mfs.open("/work/data/hello.txt", "wb") as f:
    f.write(b"Hello, D-MemFS!\n")

# Read from a file
with mfs.open("/work/data/hello.txt", "rb") as f:
    print(f.read())  # b'Hello, D-MemFS!\n'

# stat information
st = mfs.stat("/work/data/hello.txt")
print(st["size"])         # 16
print(st["is_dir"])       # False
print(st["modified_at"])  # Unix timestamp (float)

Directory Operations

# List directory contents
for entry in mfs.listdir("/work/data"):
    print(entry)  # 'hello.txt'

# Delete an entire sub-tree
mfs.rmtree("/work")

# Check existence
print(mfs.exists("/work"))  # False

Handling Text Files

By default it's binary mode only, but you can use MFSTextHandle for text I/O.

from dmemfs import MemoryFileSystem, MFSTextHandle

mfs = MemoryFileSystem()
mfs.mkdir("/logs")

# Write text
with mfs.open("/logs/app.log", "wb") as f:
    # Wrap the binary handle with the text wrapper
    th = MFSTextHandle(f, encoding="utf-8")
    th.write("Started\n")
    th.write("Processing completed\n")

# Read text
with mfs.open("/logs/app.log", "rb") as f:
    th = MFSTextHandle(f, encoding="utf-8")
    for line in th:
        print(line, end="")

Preventing Runaways with Quotas

By simply passing the max_quota parameter, you can restrict memory usage before writing.

from dmemfs import MemoryFileSystem, MFSQuotaExceededError

# Set a 1 MiB quota
mfs = MemoryFileSystem(max_quota=1 * 1024 * 1024)

mfs.mkdir("/data")

try:
    with mfs.open("/data/big.bin", "wb") as f:
        # Raises an exception here if it tries to exceed 1 MiB
        f.write(b"x" * (2 * 1024 * 1024))
except MFSQuotaExceededError as e:
    print(f"Quota exceeded: {e}")
    # -> Quota exceeded: quota exceeded (limit=1048576, used=0, requested=2097152)

MFSQuotaExceededError occurs before the write is executed. Files won't be polluted in a half-written state.

You can also set a limit on the maximum number of files (nodes).

from dmemfs import MemoryFileSystem, MFSNodeLimitExceededError

mfs = MemoryFileSystem(max_nodes=4)
mfs.mkdir("/data")

mfs.open("/data/a.txt", "xb").close()
mfs.open("/data/b.txt", "xb").close()

try:
    mfs.open("/data/c.txt", "xb").close()
except MFSNodeLimitExceededError as e:
    print(f"Node limit exceeded: {e}")

Batch Operations with import_tree / export_tree

It also features functionalities to import and export files all at once.

import zipfile
import io
from dmemfs import MemoryFileSystem

mfs = MemoryFileSystem()

# Import in bulk using dict[str, bytes] format
mfs.import_tree({
    "/snapshot/config.json": b'{"debug": true}',
    "/snapshot/data.csv": b"id,name\n1,Alice\n",
})

# Processing...
with mfs.open("/snapshot/config.json", "rb") as f:
    config = f.read()

# Export all at once and write to a ZIP file
exported = mfs.export_tree("/snapshot")
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as zf:
    for path, data in exported.items():
        zf.writestr(path.lstrip("/"), data)
with open("snapshot.zip", "wb") as f:
    f.write(buf.getvalue())

It naturally supports the pattern: "Do all processing in memory, and export only at the very end."

Comparison Summary

Feature	`BytesIO`	`dict[str, BytesIO]`	D-MemFS
Single-file I/O	✅	✅	✅
Hierarchical directories	❌	△ Pseudo-implementation	✅
Directory listing	❌	△ Manual implementation	✅
Subtree deletion	❌	△ Manual implementation	✅
Append mode	△ Manual seek	△ Manual seek	✅
Hard Quota	❌	❌	✅
stat (size, time)	❌	❌	✅
Thread safety	❌	❌	✅
External dependencies	None	None	None (stdlib only)

Installation

pip install D-MemFS

Python 3.11+ is required. Zero external dependencies.

Conclusion

BytesIO is excellent as a "single buffer," but it is not a substitute for a file system. If you try to build your own with dict[str, BytesIO], you will step on the same bugs every time.

D-MemFS is designed to be used wherever an in-memory file system is needed—testing, CI, temporary data processing pipelines (limited to processing within the Python process). First, pip install D-MemFS and try replacing TemporaryDirectory with it.

PyPI: https://pypi.org/project/D-MemFS/

🔗 Links & Resources

GitHub: https://github.com/nightmarewalker/D-MemFS
Original Japanese Article: BytesIO じゃダメな理由 — Python インメモリ FS ライブラリを作った話

If you find this project interesting, a ⭐ on GitHub would be the best way to support my work!

DEV Community: D

[Side B] Should a Binary-Only FS Support Text Mode? Redrawing the Architecture Boundary

🧭 About this Series: The Two Sides of Development

What the Design Document Said

Design Correctness Doesn't Stop Reality From Punching Back

"Just Use io.TextIOWrapper?" — Not That Simple

Problem 1: The readinto() Requirement

Problem 2: Buffering vs. Quota Incompatibility

Problem 3: The seek() Cookie Problem

Solution: Build a Lightweight Wrapper — Without a Buffer

read(size) Is an Approximation in Characters

readline() and Three-Way Newline Handling

Did We Violate the Design Principle?

The Boundary Between Design Purity and Practicality

Closing

🔗 Links & Resources

[Side A] SQLite's Shared In-Memory DB Disappears — and `deserialize()` Won't Save You

🧭 About this Series: The Two Sides of Development

Introduction

First: The Limits of :memory: and cache=shared

"Easy with deserialize(), Right?" — Wrong.

The Tricky Behavior of deserialize()

The Workaround: Using the backup() API

Honestly: Isn't a Plain Dict Good Enough?

🔗 Links & Resources

[Side B] Pursuing OSS Quality Assurance with AI: Achieving 369 Tests, 97% Coverage, and GIL-Free Compatibility

🧭 About this Series: The Two Sides of Development

Testing is a "Contract between the Design Document and the Code"

AI Gives Us Tons of Test Cases (But Needs Supervision)

Thread-Safe Tests Look Like This

Testing the Memory Guard Was a Battle Against "Environmental Dependencies"

What is the "Remaining 3%"?

A Matrix of 3 OS × 3 Python Versions

The Moment All Tests Turned Green

Conclusion: With a Design Doc, I Feel We Can Trust AI a Bit More

🔗 Links & Resources

[Side A] Completely Defending Python from OOM Kills: The BytesIO Trap and D-MemFS 'Hard Quota' Design Philosophy

🧭 About this Series: The Two Sides of Development

Introduction

The Problem: BytesIO and dict Swell Limitlessly

Soft Quotas Are Not Enough

D-MemFS's Hard Quota Design

Code Example: Actually Using the Quota

Basic Quota Settings and Exception Handling

Processing While Checking the Remaining Quota

Node Count Limit: MFSNodeLimitExceededError

Storage Backends: SequentialMemoryFile and RandomAccessMemoryFile

SequentialMemoryFile (Sequential)

RandomAccessMemoryFile (Random Access)

auto-promotion (Automatic Promotion)

promotion_hard_limit: Suppressing Promotion of Giant Files

Memory Accounting: What is Included in the Quota

Thread-Safe Atomic Operations

A World With Hard Quotas vs. Without

Quota Configuration Guidelines

Memory Guard: Detecting Physical Memory Depletion in Advance

3 Modes

The Relationship Between Hard Quotas and Memory Guard

Design Ingenuity of "per_write" Mode

Behavior in free-threaded Python (GIL=0)

Conclusion

🔗 Links & Resources

[Side B] Breaking Free from Vibe Coding Fatigue: A Practical Record of Building an OSS with 'Spec-First AI Development'

🧭 About this Series: The Two Sides of Development

Are You Realizing the Limits of "Vibe Coding"?

Why "Design Documents Before Code"?

Using Browser AIs as Sparring Partners

Claude Opus Overturned Everything

Never Bend the Design Philosophy

Looking Back, This Was SDD

"It Got Bigger Than I Expected"

🔗 Links & Resources

[Side A] Building a RAM Disk on Windows without Admin Privileges — Python I/O Acceleration Techniques

🧭 About this Series: The Two Sides of Development

Introduction

The Conclusion First: Comparison of Methods

Premise: Why Do We Want a RAM Disk?

D-MemFS: Python-Exclusive Software RAM Disk

Replacing tempfile with D-MemFS

Before Replacement (Writes to disk occur)

"Just Use `io.TextIOWrapper`?" — Not That Simple

Problem 1: The `readinto()` Requirement

Problem 3: The `seek()` Cookie Problem

`read(size)` Is an Approximation in Characters

`readline()` and Three-Way Newline Handling

First: The Limits of `:memory:` and `cache=shared`

"Easy with `deserialize()`, Right?" — Wrong.

The Tricky Behavior of `deserialize()`

The Workaround: Using the `backup()` API

Attempting to Substitute with `dict[str, BytesIO]`...