Optimizing Disk I/O in NumPy: Implementing a Fast LZ4 Compression Algorithm via C-Extensions

numpy-cache · PyPI

High-performance LZ4 cache for NumPy arrays

pypi.org

macht1212 / numpy-cache

High-performance LZ4 cache for NumPy arrays

NumPy Cache – Fast LZ4-Based Caching for NumPy Arrays

High‑performance, lightweight disk cache for NumPy arrays with LZ4 compression – now with configurable compression speed.

📌 The Problem

When dealing with large NumPy arrays, developers face a classic trade‑off:

Method	Speed (100 MB)	Issue
np.save() / np.savez()	~45 ms	Huge storage, slow network transfer
np.savez_compressed()	~3.6 s	Single‑threaded DEFLATE (zlib) is too slow
numpy_cache	~100 ms	Best of both worlds ✅

There is a clear gap: no lightweight, specialised solution combines np.save() speed with good compression – until now.

🚀 Features

• ✅ Blazing fast – 20 × faster than np.savez_compressed()
• ✅ Good compression – 2 × smaller than np.save()
• ✅ Pure C extension – minimal overhead, maximum performance
• ✅ Configurable speed – acceleration parameter (1–16) lets you trade compression ratio for speed
• ✅ NumPy integration – works with all numeric dtypes (int, uint, float, bool)
• ✅ Multi‑dimensional – supports up…

View on GitHub

When processing large volumes of numerical data in Python, developers regularly face a trade-off between disk I/O speed and storage space. Standard tools in the NumPy ecosystem cover the extremes of this spectrum but leave a gap for scenarios requiring simultaneous efficiency in both parameters.

In this article, we will analyze the limitations of standard approaches and explore the architecture of a lightweight solution based on C-extensions and the LZ4 algorithm. This solution allows caching arrays with latency comparable to an uncompressed memory dump, while achieving a compression ratio that matches or exceeds standard zlib.

Analysis of Standard Approaches and Their Limitations

• np.save(): Performs a direct binary data dump. It has minimal latency (less than 1 ms per megabyte) but performs no compression. When working with large datasets, this leads to inefficient use of disk space and network bandwidth.
• np.savez_compressed(): Uses the deflate (zlib) algorithm. It provides good compression, but the compression process is a CPU-bound operation. On a 100 MB array, the write operation can take over 3 seconds, which is unacceptable for model training loops or real-time systems.
• HDF5 (via h5py) or Zarr: Powerful tools for storing multidimensional arrays. However, they require pulling in heavy dependencies, learning a specific API, and configuring chunking, which is excessive for the simple task of fast caching of intermediate ndarray states.

Solution Architecture

To eliminate this bottleneck, there is a solution that combines the speed of direct memory access with the efficiency of LZ4. Key architectural decisions include:

• C-extension (Python C-API): The critical execution path is offloaded to C. This avoids Python interpreter overhead and the Global Interpreter Lock (GIL) during serialization.
• Direct Memory Access (Buffer Protocol): Using PyArray_DATA allows obtaining a direct pointer to the contiguous memory block of the array, minimizing copy operations (a zero-copy approach where the data structure permits it).
• LZ4 Algorithm: The LZ4_compress_fast function is selected, allowing flexible control over the balance between speed and compression ratio via an acceleration parameter.

API Usage Example

The library interface is intentionally kept minimal to integrate into existing code with minimal changes.

import numpy as np
from numpy_cache import save, load

# Initialize an array of ~100 MB
arr = np.random.randn(5000, 5000).astype(np.float32)

# Save with default parameters (acceleration=4)
# The library automatically determines the data type and shape
save(arr, 'cache_data.npc')

# Tuning the balance: increasing acceleration to 16 
# reduces the compression ratio but maximizes write speed
save(arr, 'cache_data_fast.npc', acceleration=16)

# Load with full metadata restoration (dtype, shape)
loaded_arr = load('cache_data.npc')

Explanation: The save function accepts an ndarray object and passes the retrieved data to the C-module, where compression and header writing occur. The file extension can be anything (.bin, .cache, .npy), as parsing relies exclusively on the byte structure of the header.

Binary Serialization Structure

To ensure data portability between architectures (e.g., x86_64 and ARM64), a simple and predictable structure is used: a fixed header followed by a compressed data block.

// The pragma pack directive ensures no field alignment padding,
// guaranteeing a consistent header size (96 bytes) across all platforms.
#pragma pack(push, 1)
struct CacheHeader {
    uint64_t uncompressed_size; // Original data size in bytes
    uint64_t compressed_size;   // Size of the compressed block
    uint64_t shape[8];          // Array dimensions (supports up to 8 dimensions)
    uint32_t magic;             // Magic number 0x4C5A4E43 ("LZNC") for validation
    uint32_t version;           // Structure version (current: 1)
    uint32_t ndim;              // Actual number of dimensions
    uint32_t dtype;             // Internal NumPy data type identifier
};
#pragma pack(pop)

Using a magic number allows quickly rejecting files with corrupted structures or incorrect formats during loading, without attempting to decompress invalid data. Support for non-contiguous slices (strided arrays) is implemented by pre-creating a contiguous copy of the data in memory before compression, if the source array is not C-contiguous.

Benchmarks and Performance Analysis

Testing was conducted on float32 arrays of ~1 MB. The goal of the tests was to record the overhead of serialization and deserialization compared to standard methods.

Write Latency

Method	Intel i5-1235U (Ubuntu)	Apple M1 (macOS)
np.save	880 μs	548 μs
np.savez	1,083 μs	554 μs
np.savez_compressed (zlib)	28,922 μs	32,211 μs
NumPy Cache (accel=4)	756 μs	635 μs

Observation: The write speed of the LZ4-based solution is comparable to uncompressed np.save and demonstrates an order-of-magnitude reduction in execution time compared to zlib.

Read Latency

Method	Intel i5-1235U (Ubuntu)	Apple M1 (macOS)
np.save	92 μs	77 μs
np.savez	401 μs	225 μs
np.savez_compressed (zlib)	4,859 μs	2,877 μs
NumPy Cache (accel=4)	421 μs	670 μs

Observation: During reading, there are expected overheads for LZ4 decompression (~300–500 μs) compared to direct np.save mapping. However, this time remains an order of magnitude lower than zlib, making this compromise well-justified for most caching tasks.

Compression Ratio

For the test dataset:

• np.save / np.savez: ~1.0 MB
• np.savez_compressed: ~0.5 MB
• NumPy Cache: ~0.4 MB (depending on the acceleration parameter)

In certain scenarios with numerical data, the LZ4 algorithm shows better compression than zlib due to its dictionary handling specifics and the absence of redundant checks inherent to deflate. However, the challenge of caching purely random data remains.

Managing the acceleration Parameter

The acceleration parameter (range 1–16) is passed directly to LZ4_compress_fast. It determines how aggressively the algorithm searches for matches in the dictionary:

• 1–4: Maximum match exploration. Recommended for archival storage where write time is not critical.
• 5–10: Balance. A value of 4–8 is optimal for most intermediate result caching tasks.
• 11–16: Minimal match search. Used in systems with strict write latency requirements (real-time), where a slight increase in file size is acceptable.

Conclusion

The presented approach demonstrates that caching NumPy arrays does not always require resorting to heavy formats like HDF5. Using C-extensions combined with LZ4 allows achieving latencies close to uncompressed dumps while significantly saving disk space.

The project's source code is available under the Apache 2.0 license. Implementation details of the C-module and scripts for reproducing the benchmarks can be found on PyPI.