DEV Community: Artem

Building a Fault-Tolerant File Storage over JPEG Images

Artem — Wed, 01 Jul 2026 15:42:58 +0000

TL;DR: The idea is to use a set of ordinary JPEG images as a distributed medium for an encrypted container. The data is split into redundant fragments and distributed across the images, allowing the container to be recovered even if some of the JPEG files are lost. We will not examine the code; instead, we will focus only on the general principle behind the approach.

Where the idea came from

Most steganographic systems follow a "one image — one file" model. If the image is lost or damaged, the embedded data is lost with it. The idea was to distribute the data container across multiple JPEG images while preserving the ability to recover the data even if some of those images are lost.

I should point out right away that this is not steganography in the traditional sense. The additional data is written to JPEG images after the EOI (End Of Image) marker, where it can be easily detected by even the simplest analysis. The goal is not to completely hide the presence of the data, but to use ordinary JPEG images as a distributed medium for an encrypted container.

At the same time, in everyday use the JPEG images remain fully functional, look like ordinary photographs, and do not attract attention. Even if the additional data is detected, it does not reveal whether it is an encrypted container, write artifacts, or simply an arbitrary sequence of bytes.

Why JPEG

First, JPEG is the most widely used image format. Photographs in this format naturally accumulate on computers and phones and are shared between people.

Second, this format has one important property: a JPEG file ends with a special EOI marker. Everything written after it is ignored during image decoding, so the additional data does not affect how the image is displayed. This is the area we will use:

      JPEG     JPEG     Additional
      data     EOI      data
       │        │        │
...░░░░░░░░░░░░░▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒

At first glance, the solution looks extremely simple: just split the encrypted container into fragments and write one fragment to the end of each JPEG file.

However, several questions immediately arise. How do we determine which JPEG files belong to the same container? How do we determine the order of the fragments during reconstruction? Where do we store the encryption keys? How do we ensure that the loss of any single JPEG file does not make the container unrecoverable? And how do we make the entire JPEG tail indistinguishable from a random sequence of bytes?

Beyond the EOI marker

It naturally follows that, in addition to the container fragment itself, each JPEG file must contain two auxiliary blocks: an encryption key block and a metadata block. This immediately leads to several design requirements.

First, the user password must not be used to encrypt the container or the metadata. They should be encrypted with a randomly generated master key, while the master key itself should be encrypted with the user's password.

Second, the master key cannot be stored in only one JPEG file. Otherwise, losing that file would make it impossible to recover the container. Therefore, every JPEG file must contain its own encryption key block.

Third, every JPEG file must also contain its own metadata block. This makes it possible to determine which JPEG files belong to the same container, which fragment is stored in each file, and in what order the fragments should be reconstructed.

Fourth, none of the service data should stand out from the rest of the data. Therefore, both the encryption keys and the metadata must be stored only in encrypted form.

Thus, when creating a container, we first ask for the user's password and generate a random master key. The container and the metadata are encrypted with the master key, while the master key itself is encrypted with the user's password. After the EOI marker, we sequentially write the encryption key block, the metadata block, and the encrypted container fragment:

      JPEG     JPEG      Encryption     Metadata     Container
      data     EOI       keys           block        fragment
       │        │         │              │            │
...░░░░░░░░░░░░░▓░░░░░░░░░░░░░░░░░░▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░░░

The encryption key block and the metadata block must always have a fixed size. This makes it possible to open the container using any JPEG file with a valid tail: decrypt the first block with the user's password to recover the master key, then use that key to decrypt the second block and obtain the information required to locate and reconstruct the container.

Encryption keys

Although each JPEG file contains the same master key, the encryption key blocks must be different. Otherwise, identical byte sequences would be repeated across all JPEG files.

If we simply encrypt the master key in every JPEG file using the same password, the key blocks will be identical. Therefore, we use AEAD encryption with random salt and nonce values generated separately for each JPEG file. As a result, each encryption key block has a different encrypted representation.

We will use a 32-byte master key. After AEAD encryption, its size increases to 48 bytes due to the 16-byte authentication tag. Together with the salt and nonce, the total size of the key block is 76 bytes:

salt                  16 bytes
key_nonce             12 bytes
encrypted_master_key  48 bytes
------------------------------
                      76 bytes

The master key provides another advantage: to change the user's password, it is enough to re-encrypt the master key with the new password and update the encryption key block in each JPEG file. The entire container does not need to be re-encrypted.

Metadata

After recovering the master key, we need to know which JPEG files belong to the container, which fragment is stored in each file, and which recovery threshold was used when splitting the container into fragments:

container_uuid         16 bytes  - container identifier (UUID)
container_generation    4 bytes  - container generation (uint32)
container_threshold     2 bytes  - recovery threshold for reconstruction
shard_index             2 bytes  - index of the current fragment
shard_total             2 bytes  - total number of fragments in the container
-------------------------------
                       26 bytes

The recovery threshold defines the minimum number of container fragments required for successful reconstruction (specified when the container is initialized). The container identifier is used to determine which JPEG files belong to which container (if files from multiple containers are accidentally mixed together). The container generation is used to select the latest recoverable version of the container in case failures occur while modifying its contents.

Thus, the total size of the metadata in plaintext is only 26 bytes. As with the key block, we use AEAD encryption. A separate random nonce is generated for each JPEG file, so even identical metadata produces a different encrypted representation.

After encryption, a 16-byte authentication tag is added to the 26 bytes of metadata, increasing the size of the encrypted metadata to 42 bytes. Together with the nonce, the total size of the block is 54 bytes:

metadata_nonce      12 bytes
encrypted_metadata  42 bytes
----------------------------
                    54 bytes

JPEG tail

We have covered the format of the service data. It remains to decide how the user files themselves will be stored. Once that is defined, the structure of the JPEG tail will be complete.

Fragmentation algorithms operate on a single stream of bytes rather than on a collection of individual files. Therefore, before encryption, the user files (the ones that will be stored inside the container) must be combined into a single binary object. At the same time, it must be possible to obtain the list of stored files without extracting their contents. There are several possible approaches:

A custom binary object format and a manifest file. It provides full control over the data structure but requires a manual implementation.
A TAR archive. It packs the files into a single binary object, but a separate manifest file is required to quickly obtain information about the contents.
A ZIP archive without compression. It packs the files into a single binary object while also storing all the necessary information about the contents. This is the approach we will use.

In other words, the user files are packed in memory into a ZIP archive, which is then encrypted and split into redundant fragments. Each container fragment is written after the encryption key block and the metadata block:

       Encryption Key Block            Metadata Block    Container Fragment
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
└─────────┘└───────┘└─────────────┘└───────┘└───────────┘└────────────────┘
 16 bytes  12 bytes    48 bytes    12 bytes   54 bytes        N bytes
 random    random      encrypted   random     encrypted       encrypted
 salt      nonce       master key  nonce      metadata        data shard

The container fragment occupies all the remaining space until the end of the file. Its size is not fixed and depends on the size of the encrypted container, the number of JPEG files, and the selected recovery threshold.

Redundancy and recovery

From this point on, the scheme resembles a RAID array: the container fragments (together with the keys and metadata) are distributed across multiple independent JPEG files with a specified recovery threshold. As a result, the container can be recovered from any set of JPEG files of the current generation, provided the number of valid fragments is not below the threshold:

                      Container
                      Shards: 5
┌──────────────────────────────────────────────────┐
▒▒▒▒▒▒▓▓   ▒▒▒▒▒▒▓▓   ▒▒▒▒▒▒▓▓   ▒▒▒▒▒▒▓▓   ▒▒▒▒▒▒▓▓
  JPEG       JPEG       JPEG       JPEG       JPEG
└────────────────────────────┘   └─────────────────┘
        Threshold: 3/3              Redundancy: 2

If some JPEG files are lost, but the number of remaining files is not below the recovery threshold, the container can not only be reconstructed but also restored to its original level of redundancy. To do this, it is enough to add new JPEG files and redistribute the container across all available media. In the project implementation, this operation is provided as a separate command that rebuilds the container and restores full redundancy relative to the current set of JPEG files.

For a Python implementation, the zfec library is an excellent choice. It provides an erasure coding mechanism — a practical implementation of the Reed–Solomon algorithm.

Container initialization

Now that the storage format is fully defined, we can move on to container initialization. To get started, we need a set of JPEG files, a user password, and a recovery threshold.

First, we verify that the files are valid JPEG files (they start with SOI), are not already being used as carriers for another container, and that the selected recovery threshold does not exceed the number of files.

Next, we generate the container identifier and the master key, create an empty ZIP archive in memory, encrypt it, and split it into fragments using erasure coding.

After that, for each JPEG file independently, we generate an encryption key block and a metadata block, append the corresponding container fragment, and write the resulting JPEG tail after the EOI marker.

The write operation is performed in two stages. First, temporary copies are created for all JPEG files, and the JPEG tails are added to them. Each temporary file is written and synchronized to disk. Only after all temporary files have been successfully prepared are they used to replace the original files one by one using POSIX semantics (an atomic rename/replace). The directory is then synchronized.

This approach does not make the group of replacements a single transaction, but it eliminates the possibility of an individual JPEG file being left partially written. After a failure, each carrier will be either in the old state or in the new state.

Container removal from the carrier files of the current recoverable generation follows the same principle: temporary versions of all JPEG files are first prepared without the JPEG tails, after which they atomically replace the original carriers.

Modifying the contents

After the container has been created, working with the files inside it reduces to updating the internal ZIP archive. All intermediate operations are performed in memory — data is written to disk only after the new version of the container has been created.

Regardless of whether we are adding a new file, deleting an existing one, or modifying its contents, the sequence of operations remains the same. First, we recover the master key using the user's password, decrypt the metadata, reconstruct the container from its fragments, and decrypt it. After making the necessary changes, we encrypt the container again, split it into a new set of fragments, update the metadata, and write the new JPEG tails using the same two-phase write procedure as during container initialization.

Each write creates a new container generation (the container_generation value in the metadata is incremented by one). This allows us to distinguish the current version of the container from previous ones in the event of an interrupted operation and restore consistency, if possible.

Inconsistent state and redundancy recovery

Suppose the container is distributed across four JPEG files, and the recovery threshold is set to two fragments:

               Generation 1
               Shards: 4
┌───────────────────────────────────────┐
░░░░░░▓▓   ░░░░░░▓▓   ░░░░░░▓▓   ░░░░░░▓▓
  JPEG       JPEG       JPEG       JPEG
└─────────────────┘
  Threshold: 2/2

Now suppose that a modification operation is interrupted after the second fragment has been overwritten. After the failure, the available JPEG files will contain two container generations at the same time:

    Generation 1          Generation 2
    Shards: 2             Shards: 2
┌─────────────────┐   ┌─────────────────┐
░░░░░░▓▓   ░░░░░░▓▓   ▒▒▒▒▒▒▓▓   ▒▒▒▒▒▒▓▓
  JPEG       JPEG       JPEG       JPEG
└─────────────────┘   └─────────────────┘
  Threshold: 2/2        Threshold: 2/2

Since both generations still satisfy the minimum recovery threshold, either of them can be reconstructed. This is where the generation number becomes important: when opening the container, the generation with the highest number is selected as the most recent one. After opening the container, it is enough to run the recovery command, which redistributes the container across all available JPEG files, restores full redundancy for the current set of available JPEG files, and creates a new container generation.

Now consider a less fortunate scenario. Suppose the modification operation is again interrupted after the second fragment has been overwritten, but one of the second-generation fragments is also lost for external reasons (for example, the corresponding JPEG file is accidentally overwritten in an image editor):

    Generation 1          Generation 2
    Shards: 2             Shards: 1
┌─────────────────┐   ┌─────────────────┐
░░░░░░▓▓   ░░░░░░▓▓   ▒▒▒▒▒▒▓▓   ▒▒▒▒▒▒▒▒
  JPEG       JPEG       JPEG       JPEG
└─────────────────┘   └─────────────────┘
  Threshold: 2/2        Threshold: 1/2

In this case, the latest generation can no longer be reconstructed: only one of the required two fragments is available. However, the previous generation is still fully recoverable because it still has enough fragments. Therefore, container reconstruction will roll back to that generation. After opening the container, it is again sufficient to run the recovery command to restore the lost redundancy.

Unfortunately, recovery after failures is not always possible. If no generation has the minimum required number of fragments available, the container cannot be recovered.

For this reason, the recovery threshold should be chosen with some margin. The lower the threshold, the more losses the container can tolerate, but the more redundant data must be stored. It is always a trade-off between fault tolerance and the amount of redundancy (I usually set the threshold slightly below half of the total number of JPEG files).

Problems and limitations

Although the described approach works, it has a number of important limitations.

First, as I mentioned at the very beginning, the project is not a steganographic system. The additional data after the EOI marker can be easily detected using any editor. However, the images themselves remain unchanged, and the JPEG tail is a cryptographically random sequence of bytes from which its purpose cannot be determined.

Second, image editors and some online services completely rewrite the JPEG file, discarding the entire JPEG tail. Therefore, this container is intended primarily for local storage and file transfer without re-encoding.

Third, all operations are performed entirely in memory. When working with large containers, the amount of memory used during an operation may significantly exceed the size of the container itself. If there is not enough available RAM, the operating system may start using swap space, causing parts of the decrypted data to be temporarily written to disk.

Conclusion

The result is a small open source project implementing the idea described above. Of course, it is certainly not meant to be a serious data protection system or a steganographic tool. But as an engineering experiment, I found it quite interesting.

I tried to keep the implementation as simple as possible. The project has only two external dependencies: cryptography and zfec. You can take a look at the code or try the project yourself:

GitHub: github.com/artabramov/jpegfs
PyPI: pypi.org/project/jpegfs

Beyond Encryption: Reducing Metadata Leakage with Fragmentation

Artem — Mon, 22 Jun 2026 05:29:36 +0000

TL;DR: This article is a design exploration rather than a cryptographic proposal. It examines whether splitting encrypted files into independently stored fragments can reduce metadata leakage and make encrypted storage harder to analyze. The approach adds complexity and does not strengthen encryption itself.

1. The one-file-one-object problem

When people talk about file encryption, they usually focus on protecting file contents. If an attacker obtains a disk or a backup, the documents remain unreadable without the encryption key.

However, encryption does not necessarily hide everything about the storage.

Many encrypted storage systems follow a simple model: one original file becomes one encrypted object. The contents are protected, but the overall structure often remains visible. An observer may still be able to see how many objects exist, their approximate sizes, when new objects appear, and which objects change over time.

None of this reveals the plaintext directly. Still, it can provide useful clues about how the storage is organized and how it evolves.

This type of information is commonly referred to as metadata leakage. The exact amount of exposed metadata depends on the storage system, but the general problem remains the same: encryption protects data, while some information about that data may still be observable.

This raises an interesting question.

If encryption hides the contents of files, can we also make the structure of encrypted storage harder to analyze?

2. Fragmenting encrypted files

One possible way to reduce metadata exposure is to stop treating each encrypted file as a single storage object.

The file is encrypted as usual, but the resulting ciphertext is then split into multiple fragments that are stored independently.

Instead of a storage system containing a collection of encrypted files, the storage now contains a collection of encrypted fragments. An observer can still see the objects themselves, but determining which fragments belong together becomes less straightforward.

At first glance, this appears to improve privacy. The data remains protected by encryption, while the storage structure becomes less obvious.

It is important to keep the goal of fragmentation in perspective, however.

Fragmentation does not strengthen encryption, make key recovery harder, or replace proper key management. The confidentiality of the data still depends entirely on the underlying cryptographic mechanisms.

Its purpose is much narrower: to reduce the amount of structural information that can be inferred from the storage itself.

The question, therefore, is not whether fragmentation can replace encryption. It cannot.

The real question is whether fragmentation can make an already encrypted storage system more difficult to analyze.

3. Fragmenting files into a fixed number of parts

The simplest fragmentation strategy is to always split a file into the same number of fragments. For example, every encrypted file could be divided into 8, 16, or 32 parts regardless of its size.

This removes the direct one-file-one-object relationship, but it introduces a different problem.

When the number of fragments is fixed, fragment size becomes directly linked to the size of the original file. Larger files produce larger fragments, while smaller files produce smaller ones.

For example, splitting an 80 MB file into eight parts produces fragments of roughly 10 MB each. The same approach applied to an 8 MB file produces fragments of about 1 MB.

As a result, size-related metadata does not disappear. It simply moves from the file level to the fragment level. Fragments belonging to the same file tend to have similar sizes, making them easier to group and analyze.

This creates an interesting tradeoff. The direct relationship between a file and a storage object is removed, but a new relationship emerges between the file and the size of its fragments.

For that reason, splitting files into a fixed number of parts is best viewed as a first step rather than a complete solution. It obscures the storage structure, but still leaves patterns that can be observed through fragment sizes.

4. Fragmenting files into fixed-size parts

The limitations of the previous approach suggest a natural alternative: instead of fixing the number of fragments, fix their size.

For example, all encrypted data could be split into 1 MB blocks. Most fragments would then have the same size regardless of the original file.

This avoids the direct relationship between file size and fragment size, but it introduces a different form of metadata leakage.

With fixed-size fragments, file size is reflected in the number of fragments produced. A 400 MB file generates roughly 400 objects, while a 40 KB file may still occupy a single fragment.

In other words, the storage no longer reveals information through fragment sizes, but it still reveals information through fragment counts.

This becomes particularly noticeable for small files. A file that fits entirely within a single block effectively falls back to the familiar one-file-one-object model.

There is also a practical tradeoff when choosing a block size. Small blocks improve fragmentation but can generate huge numbers of objects for large files. Large blocks reduce object counts but provide little benefit for smaller data.

Compared to splitting files into a fixed number of parts, this approach hides metadata more effectively. However, the relationship between file size and storage structure remains visible, just in a different form.

5. Adaptive fragmentation

The previous approaches share a common weakness: they apply the same rule to every file.

When the number of fragments is fixed, file size is reflected in fragment size. When fragment size is fixed, file size is reflected in fragment count.

A more flexible approach is to choose the fragment size from a predefined set.

KB = 1024
MB = 1024 * KB

FRAGMENT_SIZES = [
    1 * KB,
    2 * KB,
    4 * KB,
    8 * KB,
    16 * KB,
    32 * KB,
]

Instead of treating every file identically, the system selects a size that produces a reasonable number of fragments.

For example, the goal might be to keep most files within a range of roughly 8–32 fragments.

TARGET_MIN = 8
TARGET_MAX = 32

for fragment_size in FRAGMENT_SIZES:
    count = math.ceil(file_size / fragment_size)

    if TARGET_MIN <= count <= TARGET_MAX:
        return fragment_size

This avoids the extremes of the previous strategies. Large files no longer explode into thousands of fragments, while smaller files are less likely to remain represented by a single object.

An additional benefit is that many unrelated files end up using the same fragment sizes. The storage gradually fills with large numbers of similar-looking objects, making size-based analysis less reliable.

The choice of fragment sizes, however, introduces its own tradeoff.

A large number of size options allows more precise tuning but also creates more distinct size categories within the storage. Fewer size options reduce the amount of information available to an observer, but make fragmentation less adaptable.

In practice, a relatively coarse scale is often sufficient:

FRAGMENT_SIZES = [
    1 * KB,
    4 * KB,
    16 * KB,
    64 * KB,
    256 * KB,
    1 * MB,
    4 * MB,
    16 * MB,
]

Even with adaptive sizing, deterministic behavior can still reveal patterns. A file of a particular size will usually be fragmented in exactly the same way every time.

One way to reduce this predictability is to introduce a small amount of randomness.

Instead of selecting a single "best" size, the system can choose from several nearby candidates.

best_index = FRAGMENT_SIZES.index(best_size)

candidates = FRAGMENT_SIZES[
    max(0, best_index - 1):
    min(len(FRAGMENT_SIZES), best_index + 2)
]

fragment_size = random.choice(candidates)

For example, if 1 MB is considered optimal, the system might choose 512 KB, 1 MB, or 2 MB. All options remain reasonably close to the target fragmentation level, but the resulting storage layout becomes less predictable.

As a result, two files of the same size may be fragmented differently, and even uploading the same file twice can produce different fragment layouts.

The relationship between file size and storage structure does not disappear entirely. However, it becomes significantly less direct than in either of the fixed-rule approaches.

6. The reconstruction map

At this point, a practical question becomes unavoidable.

If a file has been split into dozens or even hundreds of encrypted fragments, how does the system know which fragments belong together?

The answer is that fragmentation requires an additional metadata structure that describes how files can be reconstructed. We can call it a reconstruction map.

In its simplest form, it might contain information such as:

file_id
original_name
original_size
fragment_ids[]
fragment_order[]
checksum/hash

The reconstruction map stores the relationships between files and fragments, allowing the system to locate the required objects and reassemble them in the correct order.

Without it, the fragments are simply a collection of unrelated encrypted objects.

This observation reveals an important property of fragmentation.

Fragmentation does not eliminate metadata. Instead, it relocates the most valuable metadata into a dedicated structure.

In fact, the reconstruction map contains much of the information that fragmentation attempts to hide from an external observer. It knows which fragments belong together, how many fragments exist, and how they should be reassembled.

For that reason, storing such information alongside the fragments in plaintext would defeat much of the purpose of fragmentation. In practice, the reconstruction map would need to be protected separately, whether as an encrypted database, an encrypted manifest, or a dedicated secrets store.

This leads to an important conclusion.

The structure of the storage never truly disappears. Fragmentation merely makes that structure less visible and concentrates the knowledge required to reconstruct it into a separate component that must be protected just as carefully as the data itself.

7. Correlating fragments by creation time

Even if fragments contain no identifying information and are assigned completely random identifiers, one source of metadata still remains: time.

When a file is written to storage, all of its fragments are typically created within a very short period. As a result, they often share similar timestamps and appear in the filesystem as part of the same operation.

For an observer, this can become another useful signal. A group of objects created at nearly the same moment is likely to be related, even if their contents and identifiers reveal nothing.

The issue extends beyond timestamps themselves. Most filesystems maintain additional metadata associated with each object. For example, ext4, XFS, and UFS store information such as creation times, modification times, sizes, and filesystem-specific records.

None of this metadata directly reveals which fragments belong together. However, it can provide enough clues to support correlation and analysis.

One possible way to weaken these signals is to periodically recreate some existing fragments.

Instead of leaving fragments untouched after their initial creation, the system can occasionally select a subset of them, generate new identifiers, and write replacement objects containing the same encrypted data.

The contents remain unchanged, but the associated filesystem metadata is refreshed.

Over time, this makes creation timestamps less reliable as an indicator of when data originally entered the system. Fragments that were once created together gradually become harder to distinguish from fragments produced by unrelated operations.

This does not eliminate time-based metadata, but it can make temporal analysis significantly less useful.

8. The reshuffling algorithm

Once fragments begin to be recreated over time, the next question becomes obvious: how many should be reshuffled during each operation?

Recreating every fragment would be the most effective way to break temporal relationships, but it would also be prohibitively expensive. Even a minor change could trigger a rewrite of the entire storage.

At the opposite extreme is doing nothing at all, leaving timestamp-based correlations intact.

A practical solution lies somewhere in between.

One approach is to let the amount of reshuffling grow much more slowly than the storage itself. For example:

shuffle_count = ceil(log2(total_fragments + 1))

This produces roughly the following behavior:

100 fragments      → 7 recreations
1,000 fragments    → 10 recreations
10,000 fragments   → 14 recreations
100,000 fragments  → 17 recreations

In practice, minimum and maximum limits are usually desirable:

shuffle_count = min(
    MAX_SHUFFLE,
    max(
        MIN_SHUFFLE,
        math.ceil(math.log2(total_fragments + 1))
    )
)

With this approach, temporal relationships gradually become less reliable without forcing the system to constantly rewrite large portions of the storage.

The goal is not to erase the history of the data completely. It is simply to make that history less obvious and less useful for analysis.

9. The cost of fragmentation

By this point, fragmentation may seem like an attractive way to reduce metadata exposure. However, these benefits come at a cost.

The most obvious consequence is the growing number of storage objects. In a traditional system, one file usually corresponds to one object. With fragmentation, the same file may be represented by dozens or even hundreds of fragments.

This increases the amount of filesystem metadata, the number of objects that must be tracked, and the overall complexity of storage operations.

The reconstruction map introduces additional overhead as well. Every fragmented file requires metadata describing which fragments belong together and how they should be reassembled. As the number of fragments grows, so does the amount of information needed to manage them.

File operations also become more expensive.

Reading a file requires locating all relevant fragments and reconstructing them in the correct order. Writing a file requires encryption, fragmentation, metadata updates, and, potentially, reshuffling of existing fragments.

Storage scalability becomes another consideration.

If files are fragmented aggressively, object counts can increase much faster than the amount of stored data. A storage system containing 100,000 files may easily end up managing millions of fragments.

Modern filesystems can handle such workloads, but large object counts can affect indexing, backup operations, directory traversal, and general maintenance tasks. In filesystems such as ext4, every object also requires its own inode, making metadata consumption a practical concern rather than a purely theoretical one.

Perhaps the most significant cost, however, is architectural complexity.

A fragmented storage system requires reconstruction maps, fragment management, reassembly logic, reshuffling mechanisms, consistency guarantees, and recovery procedures. Over time, it begins to resemble a second filesystem built on top of the existing one.

For that reason, fragmentation is difficult to describe as a free security improvement.

It can reduce the amount of metadata available to an observer, but it achieves this by introducing additional overhead, greater implementation complexity, and new operational challenges.

Conclusion

Encryption protects the contents of data. Fragmentation can make the structure of that data more difficult to analyze.

By breaking the direct relationship between files and storage objects, a fragmented storage system can reduce some forms of metadata leakage and make storage layouts less transparent to an observer.

At the same time, fragmentation does not strengthen encryption and does not eliminate metadata entirely. Much of the structural information is simply moved into a reconstruction mechanism that must be protected separately.

The approach also introduces significant costs. More objects must be managed, additional metadata must be maintained, and the overall architecture becomes considerably more complex.

As a result, fragmentation is best viewed as a complementary technique rather than a replacement for encryption. It addresses a different problem: not protecting the contents of data, but reducing the amount of information that can be inferred from the structure of the storage itself.

Sorting Encrypted Strings with a Leaked-Order Index

Artem — Sat, 13 Jun 2026 06:27:09 +0000

TL;DR: This is not a cryptographic construction. It is a pragmatic engineering compromise for applications where encrypted storage is required but approximate alphabetical ordering is still useful. I sort encrypted strings using an external index: the sum of weighted Unicode code points for the first N characters with exponential positional weights, followed by quantization. Monotonicity is preserved, but accuracy predictably degrades after the first few characters. Not a cryptographic scheme; some ordering information leaks by design.

The problem

Some time ago, while implementing a project, I ran into the problem of sorting encrypted data in a database. I’d like to share the solution.

I won’t go into detail describing the entire application. I’ll just say that, according to the required architecture, almost all data in the database must be stored exclusively in encrypted form: usernames, file names, tags, comments, dates, and so on (with the exception of identifiers and some system fields). That is, the table structure should be open, but the contents should not be. The encryption is symmetric: the same key is used for both encryption and decryption. This means that without the encryption key, even with a full database dump an attacker should not obtain any original data. And this is where two problems immediately arose: searching by the data without fully decrypting it, and sorting encrypted data.

The first problem, with some caveats, is solved fairly simply. To search encrypted data, it is enough to additionally store hashes of the original values you plan to search on. This allows exact-match lookups (for example, users by login or files by tag) without storing the original values in plaintext. Yes, this won’t allow pattern searches, but it’s quite acceptable for the project’s goals.

But sorting encrypted data turned out to be significantly more difficult.

The solution

A quick search showed that the problem is far from new, but there are no standard approaches to solving it. In other words, each specific task is solved individually. Yes, there are some clever algorithms that allow sorting directly on encrypted data, but they are anything but simple. In a nutshell, here’s what exists at the moment:

OPE (Order-Preserving Encryption) — the order of ciphertexts matches the order of plaintexts. Convenient for ORDER BY, OPE leaks significant ordering and distribution information, making statistical attacks practical in many scenarios.
ORE (Order-Revealing Encryption) — the order does not match directly, but there is a comparison procedure for ciphertexts. Safer than OPE, but still reveals relative ordering.
SSE (Searchable Encryption) and FHE (Fully Homomorphic Encryption) — look serious and could theoretically fit (though almost certainly with colossal computational overhead).

I rejected the first option myself. The remaining options were either too complex to deploy or required substantial cryptographic infrastructure: there were no simple solutions for PostgreSQL (which I was using at the time), and diving that deep into cryptography—let alone writing my own database extension—was not in my plans. So I started looking for a simpler alternative.

So, we need to sort encrypted strings in the database without decrypting them. The solution should be simple to implement and acceptable from a security standpoint. Small sorting inaccuracies are allowed.

The only option that came to mind was to compute some numeric indices for the data we intend to sort by and store them separately. That is, instead of sorting by the data itself, we can sort by their indices. The index values should correlate with the original strings but should not allow straightforward reconstruction of them. The approach seemed reasonable, so I started thinking along these lines.

First, I decided it would be most convenient to convert strings into numbers whose magnitude would be relevant to the content of the original strings. Unicode code points can be used for the conversion.

Second, for indexing, you can consider only the first N characters rather than the entire string. Thus, each of these N characters should contribute to the final index, with the first character given the highest weight and each subsequent one progressively less.

Initially, I tried positional scaling with base 10. That is, I multiplied the character code points by 10 raised to a power decreasing from the start of the substring, then summed the result. That base turned out to be insufficient: due to carry propagation, many sequences of code points produced the same sums (collisions were rampant). But increasing the base fixed this.

As a result, the weight of an individual character is computed as the product of its code point and a constant—the positional weight—raised to the power of its position from the end of the substring:

WEIGHT = CODEPOINT × (WEIGHT_FACTOR ^ (N - 1 - POSITION))

Before the calculation, you need to canonicalize the string so that visually and lexically equivalent character variants have the same weight (with additional special-case handling for some alphabets).

With the right choice of parameters, the weights of the higher positions exponentially dominate the lower ones. You just need the final index to comfortably fit into a 64-bit BIGINT for database storage.

At first glance, the solution seemed acceptable. But it turned out that this index is too informative: the large difference in character weights and the limited alphabet allow a dictionary attack (you can recover a substring by sequentially subtracting precomputed weights from the index). We need to “smear” the result to increase the number of collisions while preserving deterministic sort order. This is solved by quantization: it is enough to divide the index by a fixed divisor and discard the remainder:

FINAL_WEIGHT = WEIGHT_SUM // (2 ^ COARSE_SHIFT)

This preserves monotonicity but turns the set of exact values into a finite number of buckets. Substrings that fall into the same bucket will be considered equal (the database will resolve their relative order on its own).

After a bit of experimentation, I settled on N = 6, WEIGHT_FACTOR = 40, COARSE_SHIFT = 17. Essentially, this is an engineering compromise that provides deterministic sorting (heavily biased toward the initial characters) regardless of the internal representation (UTF-16/UTF-32). The result is resistant to clumping on the first few characters but does allow attacks using a constrained dictionary. In practice, with the parameters above, the first few characters can often be inferred with high probability, while the remaining positions are obscured by quantization. At the same time, sorting error beyond the second character inevitably increases. The code:

from typing import Union
import unicodedata

INDEX_LENGTH = 6
WEIGHT_FACTOR = 40
COARSE_SHIFT = 17

# Precomputed position weights
_WEIGHTS = tuple(WEIGHT_FACTOR ** i for i in range(
    INDEX_LENGTH - 1, -1, -1))

# Precomputed quantization divisor
QUANT_DIVISOR = 2 ** COARSE_SHIFT

# Unicode normalization form used during canonicalization
NORMALIZATION_FORM = "NFC"


def _canonicalize(value: str) -> str:
    """Case-insensitive canonicalization of a string."""
    return unicodedata.normalize(NORMALIZATION_FORM, value.casefold())


def get_index(value: Union[str, int]) -> int:
    """Compute a quantized sorting index for ORDER BY clauses."""
    if isinstance(value, int):
        value = str(value)

    s = _canonicalize(value)

    idx = 0
    for pos, ch in enumerate(s[:INDEX_LENGTH]):
        idx += ord(ch) * _WEIGHTS[pos]

    return idx // QUANT_DIVISOR

Thus, the original data in the database remain hidden, but they can be sorted by the first six characters. This is not intended to provide cryptographic security of the sort key itself. The goal is to keep the original values encrypted while exposing only enough information to support approximate alphabetical ordering.

Example

Using the parameters described above (N = 6, WEIGHT_FACTOR = 40, COARSE_SHIFT = 17), the following strings produce sortable indices:

Value	Index
Alan	77939
Albert	77939
Alice	77943
Brian	78841
Brittany	78841
Charles	79423
Daniel	80074

Notice that Alan and Albert fall into the same bucket after quantization. The same happens with Brian and Brittany. From the database's perspective, each pair has the same sort index.

At the same time, the index is influenced by more than just the first character. For example, Alice receives a different index than Alan and Albert, even though all three names begin with A.

This is intentional. Quantization reduces precision and introduces collisions, making exact reconstruction of the indexed prefix more difficult. At the same time, useful ordering information is preserved: names with similar prefixes tend to receive nearby indices, while lexicographically later names generally receive larger indices.

The result is not perfectly alphabetical ordering. Instead, it is a deterministic approximation that preserves useful ordering information while intentionally sacrificing precision.

Limitations

This approach has several important limitations:

It is not a cryptographic sorting scheme.
Prefix information leaks by design.
Sorting accuracy decreases after the first few characters.
Large numbers of similar prefixes may collapse into the same bucket.
The approach is unsuitable when order leakage is unacceptable.
The index should be treated as auxiliary metadata rather than encrypted data.

For my use case, these trade-offs were acceptable. The goal was not perfect secrecy of the sort key, but rather practical ordering of encrypted data without introducing complex cryptographic infrastructure.