Decompression in C/C++: Virtual Files and Memory I/O Principles and Techniques

In this article, we will analyze the techniques for managing buffers and virtual files in memory during decompression operations, with practical examples in C/C++.

Understanding Memory vs Disk Decompression

Decompression can occur either from files stored on disk or directly from data already in memory. While high-level libraries often provide functions for extracting data directly into memory, understanding how virtual files and custom I/O techniques work gives developers the flexibility to adapt more file-centric libraries to buffer-based scenarios.

This approach can improve performance, increase security, and reduce the need for temporary files. In the following examples, we'll show how to implement these techniques and integrate them with common C/C++ libraries.

How Decompression Libraries Handle Buffers Internally

Before delving into practical techniques, it is important to understand how decompression libraries manage data internally. Whether dealing with ZIP, LZMA, or Cabinet formats, most libraries operate on the principle of streaming data through internal buffers rather than loading entire archives into memory. This design allows them to efficiently handle large files and reduce memory footprint, but it also means that the raw buffers used internally are typically not exposed to the developer.

Even high-level libraries that provide convenient APIs for extraction often maintain one or more internal buffers:

• Input buffer: stores chunks of the compressed data read from disk or another source.
• Decompression buffer: temporarily holds decompressed data before it is written to disk or returned to the caller.
• Cache or lookahead buffer: some algorithms require additional buffering to support seeking, partial reads, or integrity checks.

For libraries that support direct memory extraction, the API abstracts these buffers, allowing developers to read or write directly to their own memory structures. The internal buffering is managed transparently, ensuring efficient data flow and correct decompression without requiring manual intervention.

However, many low-level, "file-centric" C libraries assume the presence of a file on disk. They do not expose their internal buffers and expect standard file I/O operations (read, write, seek, etc.) to access the data. In these scenarios, developers must implement custom I/O callbacks to simulate file behavior using an in-memory buffer. Typical callbacks include:

• Read: to fetch data from the memory buffer instead of a disk file.
• Write: to store decompressed output into memory rather than writing to a file.
• Seek/Skip: to emulate moving the file pointer, required by libraries that perform random access.
• Tell/Size: to report the current position or the total size of the virtual file.

Understanding this distinction is crucial for several reasons:

1. Performance: Relying solely on disk I/O can introduce significant overhead, especially for repeated reads or small decompression operations.
2. Flexibility: Memory-based I/O allows processing data that may come from network streams, encrypted archives, or other non-file sources.
3. Security: Avoiding temporary files reduces the risk of sensitive data being written to disk.
4. Compatibility: Knowledge of internal buffering and callback mechanisms enables developers to adapt libraries that do not natively support memory extraction.

In essence, whether a library provides high-level memory APIs or requires custom callbacks, the underlying principle is the same: all decompression operates on buffers, and understanding how these buffers are managed is key to implementing efficient and flexible memory-based decompression in C/C++.

High-Level Libraries with Native Memory Extraction Support

Some modern decompression libraries provide built-in support for extracting data directly into memory, so you don’t need to implement custom I/O callbacks. Notable examples include:

• libzip:

  • zip_fread(): Reads a file from the archive directly into a memory buffer.
  • zip_source_buffer_create(): Creates a ZIP source from an existing memory buffer.
  • zip_open_from_source(): Opens a ZIP archive using a zip_source_t*, allowing all operations in memory.

• libarchive:

  • archive_read_data(): Extracts the current archive entry into a memory buffer.
  • archive_write_data(): Writes data directly from a memory buffer into an archive.

• zlib:

  • uncompress(): Decompresses a memory buffer containing compressed data into another buffer.
  • inflateInit(), inflate(): Stream-based API for in-memory decompression.

• minizip (part of zlib):

  • unzOpenMemory(): Opens a ZIP archive stored entirely in memory.
  • unzReadCurrentFile(): Reads the current file directly into memory.

Even though the libraries mentioned above provide high-level APIs to extract data directly into memory, under the hood they perform essentially the same operations as manual I/O callbacks. Each library manages internal buffers, reads compressed data in chunks, decompresses it, and writes the output to memory. The difference is that these processes are encapsulated within the API, so developers don’t need to implement their own callbacks manually. Understanding this helps to bridge the conceptual gap when transitioning to lower-level, file-centric libraries that do require explicit callback implementations.

Low-Level Libraries Without Direct Memory Extraction

Unlike high-level libraries, many low-level or file-centric decompression libraries do not provide native APIs for extracting data directly into memory. Examples include:

• LZMA SDK (7-Zip)
• Windows Cabinet APIs (FCICreate, FDICopy, etc.)

These libraries are designed to operate on files stored on disk. They expect standard file I/O operations (read, write, seek, etc.), and do not expose internal buffers for direct memory access.

To work with data already in memory, developers need to implement custom I/O callbacks that simulate file operations on buffers. Typical callbacks include:

• Read: Fetch data from a memory buffer instead of disk.
• Write: Store decompressed output in memory.
• Seek / Skip: Move the file pointer to emulate random access.
• Tell / Size: Report the current position or the total size of the virtual file.

One common approach is to wrap a memory buffer in a structure (e.g., MemoryStream) that keeps track of the current position and size. Each callback operates on this structure, allowing the library to process the buffer as if it were a regular file.

This approach requires more boilerplate code compared to high-level libraries, but provides full control over memory usage, enables working with non-disk data sources (network streams, encrypted archives), and avoids creating temporary files on disk.

Why Low-Level Libraries Need Custom Callbacks

Low-level, file-centric decompression libraries don't natively support extracting data into memory. To make them work with in-memory buffers, they essentially need to be "tricked."

This is where custom I/O callbacks come in. When the library requests data—thinking it's reading from a file—your callbacks provide it from a memory buffer instead. Conceptually, the process works like this:

1. You have the compressed archive loaded into a memory buffer.
2. The library performs its usual read, seek, or write operations.
3. Your callbacks intercept these operations and feed the library the requested data from your buffer.
4. The library copies chunks of this data into its internal buffers and processes it as if it were reading from a real file.

In other words, the callbacks gradually transfer the existing archive data into the library's own buffers, allowing it to decompress directly in memory without touching the disk.

Example: Custom Callbacks for In-Memory Decompression

To make low-level libraries work with memory buffers, we implement callbacks that emulate file operations. A common approach is to encapsulate the current archive buffer in a structure like MemoryStream:

typedef struct
{
    const BYTE* data;      // pointer to memory buffer
    size_t size;           // total size of buffer
    size_t pos;            // current position
} MemoryStream;

Read Callback

size_t Read(MemoryStream* ms, void* dst, size_t bytes) 
{
    if (ms->pos >= ms->size) 
    {
        return 0;
    }

    if (ms->pos + bytes > ms->size) 
    {
        bytes = ms->size - ms->pos;
    }

    memcpy(dst, ms->data + ms->pos, bytes);

    ms->pos += bytes;
    return bytes;
}

Skip Callback

bool Skip(MemoryStream* ms, size_t offset) 
{
    if (offset > ms->size) 
    {
        return false;
    }

    ms->pos = offset;
    return true;
}

Seek Callback

enum SeekOrigin { SEEK_SET, SEEK_CUR, SEEK_END };

bool Seek(MemoryStream* ms, long offset, SeekOrigin origin) 
{
    size_t newPos = 0;

    switch (origin) {
        case SEEK_SET:
            newPos = offset;
            break;
        case SEEK_CUR:
            newPos = ms->pos + offset;
            break;
        case SEEK_END:
            newPos = ms->size + offset;
            break;
    }

    if (newPos > ms->size) 
    {
        newPos = ms->size; // prevent overflow
    }

    ms->pos = newPos;
    return true;
}

Tell Callback

size_t Tell(MemoryStream* ms) 
{
    return ms->pos;
}

Write Callback

size_t Write(MemoryStream* ms, const void* src, size_t bytes) 
{
    // if writing to a pre-allocated output buffer
    memcpy((uint8_t*)ms->data + ms->pos, src, bytes);
    ms->pos += bytes;
    return bytes;
}

Understanding the Callbacks

Each of these callbacks serves a specific purpose when simulating a file in memory:

• Read: Provides the library with the next chunk of data from the memory buffer. Essential for any decompression operation.
• Write: Stores the decompressed output back into a memory buffer. Only needed when the library writes decompressed data.
• Seek / Skip: Adjusts the current read/write position in the buffer, emulating file pointer movement. Not all libraries require random access; some only read sequentially.
• Tell: Reports the current position in the buffer. Useful for libraries that need to track offsets.

Depending on the library and the operation you need:

• A simple sequential read may only require Read and Write.
• If the library expects random access within the archive, Seek / Skip and Tell become necessary.
• Some libraries may not use Write at all if you only need to inspect or decompress data to temporary structures.

The flexibility of these callbacks allows low-level, file-centric libraries to operate entirely in memory, while giving the developer control over how and where data is stored.

Callbacks as a Starting Point

The callbacks shown above are intended as a starting template. Each low-level decompression library may have slightly different expectations for function signatures, parameters, or behavior. For example, some libraries might pass an opaque pointer instead of a MemoryStream*, or require additional flags in the callbacks.

The key idea is that these functions illustrate the principle of adapting a memory buffer to behave like a file. From this foundation, you can customize and extend the callbacks to match the specific API requirements of libraries like LZMA SDK, Windows Cabinet APIs, or other file-centric decompression tools.

Low-Level Library Callbacks: LZMA SDK vs Windows CAB SDK

When adapting low-level decompression libraries to work directly with in-memory buffers, the number and type of callbacks required can vary significantly.

LZMA SDK

• ISeqInStream (simple sequential read):
  • Read → reads data from the buffer
• ILookInStream (advanced / random-access read):
  • Look → peek at upcoming bytes without advancing the position
  • Skip → skip a number of bytes
  • Read → read and advance the cursor
  • Seek → move the cursor to a specific position

⚠️ An important note:

Even though technically only the Read and Seek callbacks would be sufficient
to read from a full in-memory buffer, SzArEx_Open does not directly accept
a struct pointing to a local ISeekInStream object. Therefore, an adapter
(LookAdapter) implementing the ILookInStream interface is needed.

This adapter wraps the MemoryStream so that SzArEx_Open can use Read,
Seek, Look, and Skip callbacks uniformly, allowing the archive to be
parsed directly from memory without requiring a physical file.

In total, a full ILookInStream implementation requires 4 callbacks, while a simple ISeqInStream only requires 1. These callbacks let the library incrementally copy data from the user-provided buffer into its internal structures.

Windows Cabinet (CAB) SDK

• Requires only two main callbacks:
  1. File operations – Read, Write, Seek are implemented in functions like CabRead, CabWrite, and CabSeek. These emulate file I/O on memory buffers.
  2. Notify – CabNotify informs the application of operations such as file creation, closure, or moving to the next cabinet.

This comparison shows how different low-level libraries require different levels of manual I/O adaptation to work with memory buffers. LZMA SDK offers more granular control over reading and seeking, while CAB SDK simplifies the interface with fewer callbacks.

Comparative Summary

Library	Memory Extraction Support	Internal Buffers	Need for Custom Callbacks	Implementation Complexity	Notes / Advantages
libzip	✅ Yes	Input, decompression, lookahead	❌ Not needed	Low	High-level API: zip_fread(), zip_source_buffer_create(), zip_open_from_source() allow full memory-based operations.
libarchive	✅ Yes	Input, decompression, cache	❌ Not needed	Low	Supports multiple archive formats (ZIP, TAR, etc.), archive_read_data() and archive_write_data() work directly with buffers.
zlib	✅ Yes	Input, output	❌ Not needed	Low/Medium	uncompress() and inflate() APIs allow memory buffer decompression; suitable for streaming.
minizip (part of zlib)	✅ Yes	Input, output	❌ Not needed	Medium	unzOpenMemory(), unzReadCurrentFile() enable handling ZIP archives entirely in memory.
LZMA SDK (7-Zip)	❌ No native support	Input, lookahead	✅ Required	High	Requires implementation of callbacks: ISeqInStream (1 callback) or ILookInStream (4 callbacks) for memory buffers; offers granular control over data access.
Windows Cabinet (CAB) APIs	❌ No native support	Input, output	✅ Required	Medium	Core callbacks: Read, Write, Seek, plus CabNotify; simpler memory adaptation than LZMA SDK.

Memory vs Disk Decompression: Library Comparison

When working with compressed data in C/C++, understanding how different libraries handle memory and disk I/O is crucial. Some libraries provide high-level APIs for direct memory extraction, while others are file-centric and require custom callbacks to work with buffers. This table compares popular decompression libraries in terms of memory support, internal buffering, and implementation complexity, helping developers choose the right approach for efficient and secure memory-based workflows.

Why Extract Archives from Memory? A Practical Perspective

There are several compelling reasons to decompress archives directly in memory instead of writing them to temporary files:

• Better performance
  Avoiding disk I/O means accessing data much faster. For scenarios where speed is critical, such as embedded systems, streaming, or large data sets, this can make a real difference.

• Improved security
  When sensitive data is never written to disk, the risk of leaving traces or exposing it to accidental inspection is reduced. In memory-only workflows, the archive and its contents remain transient.

• Single-File Deployment
  By following approaches such as embedding binary data in a Win32 executable (as seen in embed my other article on embedding methods), you can load an archive from within the executable, unpack it directly in memory, and avoid deploying external files. This simplifies deployment and reduces the attack surface.

• Flexible Data Sources
  The unpacked data can come from network streams, an embedded resource, or another custom buffer. In-memory extraction means you're not bound by the file system: the unpacking logic works with any buffer provided.

• Cleaner Resource Management
  The absence of temporary files means less cleanup, fewer permissions issues, and less overhead in managing file paths or locations.

In short, direct in-memory decompression aligns well with modern requirements for resource embedding, streaming workflows, and small-footprint applications. It sets the stage for the techniques discussed in this article, where we'll show how to implement memory-based decompression and also adapt low-level libraries to this model.

Conclusion

In this article we explored how decompression libraries handle buffers, how to extract archives directly into memory, and what adaptation is needed for both high-level and low-level APIs. We saw that while libraries like libzip, libarchive, zlib, and minizip offer straightforward memory-friendly functions, others like LZMA SDK and the Windows Cabinet SDK require custom callback implementations to simulate file I/O on buffers.

By understanding these mechanisms of buffer management, read/write/seek operations, and the rationale behind in-memory extraction—such as performance gains, increased security, flexible deployment, and cleaner resource management—you gain the tools to choose the appropriate library and integration method for your use case.

Whether you are embedding binary data into a Win32 executable, streaming archives from a network, or working in an environment where disk I/O is undesirable or unavailable, the techniques discussed here give you the flexibility to load, decompress, and work with archive data entirely in memory.

As a next step, you can apply these patterns in your own projects: use high-level APIs when available, and fall back to custom callbacks and virtual file layers when required. With this understanding, you’re better equipped to build efficient, flexible and secure decompression workflows in C++.