Muhammed Shafin P

Posted on Nov 23

Introducing Qeltrix: A Content-Derived, Parallel Streaming Obfuscation Container

#hejhdiss #chacha20 #filecontainer #parallelcompression

Project Repository: github.com/hejhdiss/Qeltrix

We're excited to introduce Qeltrix (.qltx), a proof-of-concept command-line utility that explores innovative approaches to file encryption and compression. Qeltrix demonstrates how content-derived cryptography, parallel processing, and streaming architectures can work together to create secure, efficient data containers.

What is Qeltrix?

Qeltrix is an experimental file packaging format that combines several cryptographic and data processing techniques into a single container. At its core, it solves an interesting problem: how do you encrypt a file without needing to remember, store, or transmit a separate password or key? The answer lies in deriving the encryption key directly from the file's content itself.

Key Features

Content-Derived Encryption

The most distinctive feature of Qeltrix is its content-derived keying mechanism. Instead of requiring users to manage passwords or keys, Qeltrix generates encryption keys from the file content itself:

Two-Pass Mode: Derives the key from the SHA256 hash of the entire compressed data stream, providing maximum entropy and security
Single-Pass Mode: Derives the key from the first N bytes of the raw file, enabling fast, memory-bounded operations

This approach creates a unique security model where the file content becomes its own key material.

Modern Cryptography Stack

Qeltrix implements industry-standard cryptographic primitives:

ChaCha20-Poly1305 AEAD: Provides authenticated encryption with associated data
HKDF Key Derivation: Properly derives cryptographic keys from content hashes
SHA256 Checksums: Ensures data integrity at both block and container levels
Per-block Authentication: Each encrypted block includes its own Poly1305 authentication tag

Parallel Processing Architecture

Built for multi-core systems, Qeltrix uses Python's ProcessPoolExecutor to compress and encrypt blocks in parallel, dramatically improving throughput on large files. The implementation carefully manages in-flight tasks to balance memory usage with performance.

Streaming Design

Qeltrix is designed to handle files of any size through:

Disk-backed temporary storage for intermediate compressed blocks
Streaming write operations that don't require loading entire files into memory
Block-based architecture with configurable chunk sizes

Multi-Layer Data Transformation

Each data block undergoes several transformation stages:

LZ4 Compression: Fast, efficient compression of raw data
Deterministic Permutation: Content-seeded byte shuffling for additional obfuscation
XOR Keystream: Additional diffusion layer
AEAD Encryption: Final authenticated encryption with ChaCha20-Poly1305

This is a Proof of Concept

It's important to emphasize that Qeltrix is a proof of concept exploring interesting ideas in content-derived cryptography and parallel data processing. As a PoC, there is no guarantee that this project will receive future updates or evolve further.

This is entirely open to the community, a for-the-community, by-the-community type project. The author has simply introduced their original concept, and the implementation and its evolution are in your hands. The concept itself is licensed under CC BY-SA 4.0 (Creative Commons Attribution-ShareAlike 4.0 International), ensuring that derivative works remain open and attributed.

Anyone is welcome and encouraged to take this foundation and evolve it into something bigger if they find it valuable. The implementation is intentionally documented and structured to make experimentation and extension accessible. Whether you want to add one feature or completely reimagine the approach, the code is yours to build upon.

The "future possibilities" listed below are not a roadmap, they're inspiration. They represent areas where we (the community and the author collectively) see potential, but whether any of them materialize depends entirely on who picks up the torch.

Potential Enhancements

Seekability: Currently, unpacking requires processing blocks sequentially. Future versions could implement random-access seeking to specific blocks, enabling faster partial file extraction and streaming playback of media files.

Parallel Decryption: While packing is fully parallelized, unpacking currently processes blocks serially. Multi-threaded decryption could significantly improve extraction performance on multi-core systems.

Compression Algorithm Options: The current implementation uses LZ4 for its speed. Adding support for other algorithms (Zstd, Brotli, LZMA) would allow users to optimize for different size/speed trade-offs.

Incremental Updates: The format could be extended to support modifying individual blocks without reprocessing the entire file, enabling efficient updates to large containers.

Deduplication: Block-level deduplication could identify and eliminate redundant data across multiple packed files or within a single large file.

Stream Cipher Alternatives: While ChaCha20 is excellent, experimenting with other modern ciphers or custom stream generation methods could be interesting.

Metadata Extensions: The metadata block could be enhanced to store filesystem attributes, timestamps, permissions, compression ratios, or custom user tags.

Error Correction: Adding Reed-Solomon or other error correction codes could make containers more resilient to storage media corruption.

Hybrid Key Derivation: Combining content-derived keys with optional user passwords or external key files could provide additional security layers.

Format Versioning: More sophisticated version negotiation could allow backward compatibility as the format evolves.

Compression Level Tuning: Exposing compression level parameters and providing presets (fastest, balanced, maximum compression) for different use cases.

Progress Reporting: Adding hooks for progress callbacks during pack/unpack operations for better user experience in GUI applications.

Encryption Algorithm Variants: Experimenting with AES-GCM as an alternative AEAD cipher, or exploring custom cipher combinations.

Sparse File Support: Optimizing handling of files with large runs of zeros or repeated patterns through sparse block encoding.

Multi-file Containers: Extending the format to package multiple files into a single container with an internal directory structure.

Signature Support: Adding optional digital signatures to verify author authenticity alongside content integrity.

Block Size Auto-tuning: Automatically selecting optimal block sizes based on file characteristics and available system resources.

Streaming Encryption: Supporting encryption of data streams where the total size isn't known in advance.

Cross-platform Testing: Ensuring consistent behavior across different operating systems and Python implementations.

Feel free to add your own ideas to this list—or better yet, implement them and share your results!

Use Cases and Applications

While experimental, Qeltrix-style containers could be useful for:

Content Verification: The content-derived nature makes the container tamper-evident
Archival Storage: Parallel compression and strong encryption for backup systems
Data Distribution: Self-validating containers that don't require separate checksums
Research: Exploring alternative approaches to data protection and compression

Technical Implementation

Qeltrix is implemented in Python 3 and leverages:

lz4 for high-speed compression
cryptography library for ChaCha20-Poly1305 AEAD operations
ProcessPoolExecutor for parallel task management
Temporary file-backed processing for memory efficiency

The file format includes:

A clearly defined header with magic bytes (QLTX) and version information
JSON metadata containing salt, block parameters, and mode information
Sequential encrypted data blocks
A footer index with block offsets and the global checksum
A delimiter ensuring reliable footer parsing

Getting Started

Installation is straightforward:

pip install lz4 cryptography

Basic usage examples:

# Pack a file (two-pass mode, maximum security)
python3 qeltrix.py pack myfile.dat output.qltx

# Pack with single-pass mode (faster, bounded memory)
python3 qeltrix.py pack large.mp4 fast.qltx --mode single_pass_firstN

# Unpack a container
python3 qeltrix.py unpack output.qltx recovered.dat

Security Considerations

As a PoC, Qeltrix demonstrates concepts but should not be used for production security-critical applications without thorough review and hardening. Some considerations:

Content-derived keys mean identical files produce identical containers (deterministic)
The security depends entirely on content uniqueness and entropy
No password recovery mechanism exists by design
The permutation layer provides obfuscation but not cryptographic strength

Contributing and Feedback

Qeltrix is open source under GPLv3, and that's intentional. Since there's no guarantee of continued development by the original author, the future of this project is in the hands of anyone who finds it interesting. We—meaning the community and the author collectively—welcome contributions, suggestions, and experimental extensions.

Whether you're interested in implementing parallel decryption, adding new compression algorithms, exploring alternative key derivation methods, or taking the project in an entirely new direction, your involvement matters. This is deliberately structured as a foundation that others can build upon, modify, or use as inspiration for their own projects.

If you create something interesting based on Qeltrix—whether it's a minor improvement or a major fork—consider sharing it back with the community. The possibilities listed above are just starting points; your imagination and needs might reveal entirely different directions worth exploring.

Visit the repository at github.com/hejhdiss/Qeltrix to explore the code, open issues, submit pull requests, or fork it for your own experiments.

Conclusion

Qeltrix represents an exploration of content-derived cryptography and parallel data processing techniques. As a proof of concept, it demonstrates interesting possibilities but comes with no guarantees of future development or updates.

This is a for-the-community, by-the-community project. The author has introduced the original concept (licensed under CC BY-SA 4.0), and now it's entirely in the hands of those who find it valuable. The extensive list of future possibilities, from seekability and parallel decryption to alternative compression algorithms and metadata flexibility, isn't a promise of what will come, but rather a map of what could be explored by anyone motivated to do so.

Whether Qeltrix remains a small PoC or becomes the basis for something more substantial depends entirely on who chooses to pick it up and run with it. We (the community and the original author) are excited to see where these ideas might lead, even if that direction is entirely different from what was originally envisioned.

The door is open. The code is yours. Build something interesting.

DEV Community