When you change an encryption key for a database, new writes can use the new key straight away. But the data already on disk still needs the old key. Updating that data is a separate job called re-encryption.
Apache Ignite can do this while the cluster serves requests. Cache-group key rotation and background re-encryption were added in Ignite 2.10.0.
In my test with Ignite 2.18.0, removing the rate limit barely changed the time to finish but application throughput dropped sharply. These notes explain how re-encryption works and how to control its speed.
Which key are we changing?
With persistence and cache encryption enabled, Ignite encrypts stored cache data and its write-ahead log (WAL).
A cache group is a set of caches that share storage structures. Encrypted caches in the same group share a data key. A master key protects those data keys. The TDE documentation describes this setup.
Ignite stores data in fixed-size blocks called pages. Changing a cache group's data key means re-encrypting these pages.
Changing the master key re-encrypts the data keys themselves. Here, I'm focusing on the cache-group data key and the pages it protects.
When is re-encryption useful?
Two common reasons to change a data key are:
- Regular key changes. Your security policy sets a period after which the key must change.
- Possible exposure. You suspect a leak, or an employee who had access to the data key has left the company.
For scheduled changes, NIST's key management guide gives up to two years as one example for data keys that encrypt smaller amounts of data. Larger amounts may need a shorter period.
This is the time a key is used to encrypt new data. You may still need it to read older data.
The cache must be encrypted before you use this feature. Re-encryption updates the data in the running cluster. It does not update backups made earlier: they still need the old key.
What happens to the existing pages?
There are two main steps:
- Ignite gives the cache group a new key and starts using it for new writes.
- A background process goes through the existing pages so they can be written with the new key.
The old key stays available for pages that still need it. A page's key identifier tells Ignite which key to use when reading it. The key-rotation guide describes this process.
In Ignite 2.18.0, the background process goes through the stored pages one by one. It loads each page into memory if needed and marks it for writing with the new key. This is the scanning step.
Checkpoints save these pages from memory to disk. Re-encryption is complete only after these writes finish.
On each node, Ignite removes the old key after re-encryption is complete and no WAL segments need it. If a node restarts during re-encryption, the process continues after the restart.
Starting the process and checking progress
For an active cluster with an encrypted cache group, you can use control.sh:
# Example limit: 10 MB/s per node.
control.sh --encryption reencryption_rate_limit 10
# Replace my-encrypted-group with your cache group name.
control.sh --encryption change_cache_key my-encrypted-group
# Check background re-encryption progress.
control.sh --encryption reencryption_status my-encrypted-group
MB/s means megabytes per second. A limit of 10 MB/s lets each node scan about 10 MB of pages per second. Ignite calculates the amount from the page count and page size.
Some pages may already be in memory. Checkpoints write pages to disk later, so the disk's actual read and write rates can differ from the scan rate.
A value of 0 means unlimited and is the default. You can also suspend and resume background re-encryption through the CLI.
A limit set through control.sh is not permanent. After a restart, the node uses its configured value. Save your chosen limit in each server's configuration if you want to keep it. The guide covers the rate limit and restart behavior.
What changed when I raised the limit?
I tested seven limits on a three-server Ignite 2.18.0 cluster. Each server held about 8.6–8.8 GiB of stored data.
During the tests, YCSB sent a mix of 50% reads and 50% updates, with a target of 2,800 operations/s. Each limit had three valid runs.
At lower settings, a higher limit helped. Moving from 5 to 10 MB/s cut re-encryption time from about 40 minutes to 23.
Median application throughput stayed close to the target in both cases. Here, throughput means the number of completed operations per second.
The graph shows all seven settings:
Each point is the median of three runs. The lines show the lowest and highest measured values, not confidence intervals. Application throughput includes the measurement periods before, during, and after re-encryption.
At the higher settings, the gain became much smaller:
| Re-encryption limit | Time to finish, seconds | Whole-run throughput, operations/s |
|---|---|---|
| 25 MB/s | 823.9 | 2,150.3 |
| Unlimited | 818.9 | 1,593.2 |
Removing the limit changed median completion time by only 0.6%, about five seconds. Median application throughput was 26% lower.
The time ranges overlapped, and the test script checked completion only once every ten seconds. These measurements gave me no clear reason to remove the limit.
I did not find the exact bottleneck. Disk I/O and checkpoint waits are possible causes, but the saved data cannot confirm either one. The full setup, measurements, and limits of the experiment are below.
Choosing a limit for your cluster
Re-encryption keeps the cluster available, but it still uses resources that the application needs. A higher scan limit does not guarantee that the job will finish sooner.
Choose a limit with two goals in mind: when re-encryption must finish, and what response times and throughput the application needs while it runs.
To compare limits on your cluster:
- Use traffic similar to your normal workload.
- Try several limits and record how long re-encryption takes.
- Compare application throughput and response times for each limit.
- Raise the limit only if the shorter completion time is worth the effect on requests.
The useful limit may be different on another cluster. These results came from one small test, so use them as a reason to measure your own setup.
Each run started with a separate 180-second warm-up. Measurement then began with a 120-second baseline period before the key change. It continued through re-encryption and for 60 seconds after it finished. The controller checked for completion every ten seconds. If scanning had finished but a checkpoint was still needed, it forced one. It checked the collected files and waited 300 seconds before the next run. The seven limits were tested in three groups: 5/25/unlimited, 10/15/20, and a separate group for 17.5 MB/s. Each limit had three valid runs. In groups with several limits, the order was random, with a fixed seed so the order could be repeated. Pilot runs and a group with a different target rate were left out of the results. Before each group of tests, I checked which target rates the cluster could support. A target passed if the cluster reached at least 95% of it. All three checks passed at 4,000 operations/s and failed at 5,000. The experiment used 70% of the highest passing target: 2,800 operations/s. READ p99 means that 99% of recorded read times are at or below this value. Intended latency starts at an operation's planned start time. It includes any delay before the client starts the call. Actual latency starts when the call really begins. Both measurements end when the call finishes. YCSB records both. The saved summaries contain only intended-latency percentiles. They don't contain comparable actual-latency or baseline-latency percentiles. So these results cannot show whether actual response times met the application's service-level agreement (SLA). The table below shows the median of three runs. Duration starts after the key-change command returns and ends when the controller first sees that re-encryption is complete. This includes checkpoint waits and the time between status checks. YCSB reported throughput at regular intervals before, during, and after re-encryption. I took the median of these samples for each run, then the median across the three runs. Warm-up was excluded. For p99, I also took the median of the three run values. At 25 MB/s, the measured time was between 767.5 and 830.1 seconds. Without a limit, it was between 806.3 and 831.3 seconds. The throughput and intended p99 ranges also overlapped. I did not run statistical significance tests. The periods before and after re-encryption had fixed lengths. Re-encryption took longer at lower limits. In those runs, more throughput samples came from the re-encryption period. This makes comparisons across limits harder. The 25 MB/s and unlimited runs had similar lengths, so this matters less for that pair. Separate results for each period would be better. Each YCSB worker sends one request at a time. Intended latency uses the requests' planned start times. It does not show how many requests are waiting on the server or how much work remains at the end of a run. The summaries for each run and the combined tables were saved. The raw histograms, measurements over time, and logs were lost when the cloud VMs and disks were deleted. Without them, I cannot study latency in each phase or find the exact resource limit. The experiment used one cloud setup, one dataset size, one workload profile, and one target rate. Below are the saved values for all 21 runs. Test setup, what the measurements mean, and all 21 runs
Test setup
Component
Configuration
Ignite
2.18.0; persistence; encrypted partitioned cache; one backup;
FULL_SYNC writes
Cluster
Three server VMs and a separate YCSB VM in one Yandex Cloud availability zone
Each VM
8 vCPU, AMD EPYC Genoa under KVM; 16 GiB RAM; 150 GiB network disk
Server memory and WAL
2 GiB JVM heap; data region up to 8 GiB; WAL mode
LOG_ONLY
Software
Ubuntu 24.04; OpenJDK 17; YCSB 0.17.0 with an Ignite 2.18-compatible binding
Dataset
12 million records; ten 100-byte fields per record; about 8.6–8.8 GiB of stored data per server
Workload
YCSB A: 50% reads, 50% updates; uniform keys; 64 worker threads
Test procedure
Reading the metrics
Configured limit
Duration, s
Whole-run throughput, operations/s
Intended READ p99, s
5 MB/s
2,412.8
2,799.8
28.1
10 MB/s
1,400.5
2,799.5
46.0
15 MB/s
1,061.5
2,748.2
65.4
17.5 MB/s
983.2
2,680.8
136.1
20 MB/s
895.2
2,640.0
200.1
25 MB/s
823.9
2,150.3
253.9
Unlimited
818.9
1,593.2
369.6
Limits of this experiment
All 21 runs
0 means unlimited. The values are rounded. Intended latency is shown in seconds instead of microseconds.
limit_mb_s,duration_s,throughput_ops_s,intended_read_p99_s
5,2412.819,2799.8,28.082
5,2390.390,2799.8,28.606
5,2451.715,2799.4,25.625
10,1400.500,2799.5,79.561
10,1359.447,2799.6,45.515
10,1445.509,2782.3,46.039
15,1061.458,2762.6,65.438
15,1064.814,2748.2,89.981
15,1051.501,2676.8,57.016
17.5,993.935,2706.4,129.827
17.5,983.240,2680.8,136.053
17.5,960.698,2522.0,192.807
20,922.917,2754.9,149.684
20,852.925,2640.0,200.147
20,895.219,2623.1,240.386
25,830.100,2313.1,253.886
25,767.505,1501.4,394.002
25,823.871,2150.3,253.624
0,806.252,1471.6,375.652
0,831.256,1593.2,369.623
0,818.876,1795.7,355.729
Have you seen similar issues with backups, migrations, or other background jobs?
This article is based on my talk at the 20th Saint Petersburg International Conference “Regional Informatics (RI-2026)” on 29 October 2026.
My paper, “Experimental Evaluation of the Impact of Background Re-encryption Rate on Apache Ignite Performance,” appears on pages 182–184 of the conference proceedings. It is in Russian, with an English abstract (ISBN 978-5-00182-210-3).
Top comments (0)