DEV Community: Valentin Balaschenko

Memory in an XRPL Node: Explicit Heap Trimming after Cache Sweeps

Valentin Balaschenko — Thu, 05 Nov 2026 16:05:46 +0000

For operators

If you measure your node's memory, read the xrpld process, its RSS from /proc/<pid>/status or a process exporter, not the systemctl status figure, which includes the kernel's page cache.
If you run xrpld 3.3.0 or later, your node already trims the heap after every cache sweep. There is nothing to configure.
If you build or run xrpld with an allocator other than glibc's, the trim does not apply.
If your memory graph has become a fine sawtooth since upgrading, that is the trim working, not a fault.

Background

The previous analysis identified the cause of memory growth as allocator behaviour rather than a memory leak. The node's caches released memory as expected at regular intervals, but the process's memory footprint continued to increase. This was due to glibc's allocator retaining freed memory instead of returning it to the operating system. The application was aware of the timing of large memory releases, but the allocator was not informed of these events.

glibc provides the malloc_trim function, which instructs the allocator to return unused memory to the kernel. This post describes the implementation of explicit heap trimming, the criteria for invoking malloc_trim, the trigger points considered, why allocator-level alternatives were not pursued, associated performance costs, and the observed impact during controlled testing and production deployment on Mainnet validators.

The path from finding to release took most of a year. Candidate trigger points were tested on a lab validator in November 2025, the change was merged into the development branch in February 2026 and measured in the performance lab, and it then ran on a Mainnet node through the 3.2.0 release-candidate period. It was held back from 3.2.0 as a precaution while a consensus-stability question was investigated, so 3.2.0 as released does not contain it although its release notes initially listed the change. The investigation found no effect from trimming, and the change shipped in 3.3.0 in August 2026.

The change was introduced in xrpld version 3.3.0 and is now active on all nodes running that release or later. On Ripple's Mainnet validators it lowered steady-state resident memory by 13% to 23%. The following sections detail its operational effects.

Allocator retention heuristics and explicit heap trimming

glibc's memory allocator is designed for efficient repeated allocations by placing freed memory onto free lists, which avoids system calls for similar-sized requests. To reduce lock contention, glibc implements multiple independent heaps called arenas. Each active thread is assigned an arena, and on 64-bit systems, the allocator can create up to eight arenas per processor core (see mallopt(3)). In multi-threaded applications, this results in free memory being retained across many arenas simultaneously.

Left to itself, glibc returns freed memory to the operating system in two scenarios:

Large allocations. Allocations at or above the M_MMAP_THRESHOLD are given their own memory mapping from the kernel with mmap, and that mapping is removed as soon as they are freed.
Free space at the top of an arena. Smaller allocations are handled within arenas. An arena releases memory only if contiguous free space at the top exceeds M_TRIM_THRESHOLD, which defaults to 128 KiB. Free memory located elsewhere in the arena is retained regardless of its size. Both thresholds are dynamic. By default, glibc increases the mmap threshold when a block larger than the current threshold is freed, raising it to the size of that block, up to a maximum of 32 MiB on 64-bit systems. The trim threshold is set to twice the mmap threshold. As a result, if a process occasionally frees large blocks, the allocator becomes less aggressive in trimming memory. There is no automatic adjustment in the other direction. The kernel cannot signal memory pressure to the allocator, and the allocator does not revisit previously retained memory.

xrpld's allocation pattern is poorly handled by these heuristics. The heap is dominated by small objects, such as tree nodes and ledger entries, allocated in large numbers into caches with varying lifetimes. When caches evict, freed chunks are interleaved with live allocations, so the top of each arena is rarely free and automatic trimming is infrequent. Freed memory therefore remains resident across many arenas, and the process grows until the retained memory matches the application's allocation churn, the rate at which it allocates and frees short-lived objects. This retained volume was the growth measured in the previous post.

malloc_trim is the allocator's explicit release path. Since glibc 2.8 it iterates over every arena and returns every whole page of free memory to the kernel, wherever that page sits within the heap, using the madvise system call to release pages, or sbrk to shrink the top of the main heap (malloc_trim(3)). It is not limited to the top of the heap, which is what makes it effective where the automatic path is not. Its single argument, pad, reserves an amount of free space at the top of the main heap. Only the main heap honours it. The per-thread arenas ignore it.

There are two main limitations to malloc_trim:

It does not move live data. Any page containing at least one live allocation remains resident, so the memory cost of fragmentation is reduced but not eliminated.
It is specific to glibc. Other allocators, such as jemalloc or tcmalloc, do not implement this function, and instead manage memory release according to their own policies. Two alternatives were not pursued in this work. Limiting the number of arenas with MALLOC_ARENA_MAX reduces how many places free memory is retained, at the cost of more lock contention, but it does not return free pages inside an arena, which is where the growth was. Replacing the allocator with jemalloc or tcmalloc changes memory behaviour for every build and deployment at once. An explicit trim at a single call site was the least intrusive change, could be evaluated in isolation, and left the allocator untouched for operators who build differently. An evaluation of jemalloc is under way separately.

malloc_trim is available and can release memory that the automatic trimming path cannot reach. The primary cost is a scan of the heap and the page faults that occur when released pages are subsequently reused. The remaining question is when to call it.

Trigger selection

malloc_trim should be invoked only after the application has released a significant amount of memory. Invoking it at other times pays for a heap scan with little to show for it, and frequent calls compound this cost. The selection criterion was defined as points in xrpld where the application knows that a large release has just completed. Five candidate locations were instrumented in PR #6022, with each call tagged by its trigger and the process Resident Set Size (RSS) logged before and after it. The sweep trigger and the two online-delete hooks ran together for 40 hours on a lab validator connected to Mainnet. The order-book and sync-completion triggers were judged by how often they fire, and the per-cache variant was superseded by the application-level call before the run.

Inside each cache sweep. The first implementation called malloc_trim from TaggedCache::sweep, once per cache. Several caches sweep in close succession, so this version needed a rate limiter with a five-second minimum interval and a re-entrancy guard to prevent overlapping calls.
After the application sweep. Application::doSweep executes all cache sweeps sequentially as a background job, with the interval determined by node_size. Invoking malloc_trim once after the final sweep captures all memory releases in the cycle (commit). This trigger accounted for the majority of memory reclaimed, executed at each sweep interval, and reclaimed memory on nearly every call.
After online-delete rotation. Two hooks in SHAMapStoreImp were evaluated: one following ledger cache clearance and another after old ledger deletion (same commit). Both triggers were infrequent due to the rarity of rotation events, and most memory released during rotation was already reclaimed by the subsequent sweep.
After the full order-book rebuild. OrderBookDB::update walks the ledger state every 25,600 ledgers, a constant in the code that works out to roughly once a day, and only on nodes with pathfinding enabled (commit). It cannot provide regular reclamation.
On sync completion. The transition into the full operating mode in NetworkOPs (commit), which happens once after start and then only when the node loses and regains synchronisation. Too rare for regular reclamation, and with a drawback raised in review: a node that has just regained sync is about to begin its heaviest catch-up work, so trimming at that moment means paying to release memory that will be reallocated immediately.

RSS before and after every malloc_trim call, by trigger. Lab validator connected to Mainnet, 11–13 November 2025, 40 hours. Green candles are calls after which RSS was lower, red candles calls after which it was higher. Top: the sweep trigger, firing at every sweep interval. Middle and bottom: the two online-delete triggers, firing a few dozen times over the run.

The decision documented in the PR was based on two factors: the sweep-based trigger achieved the highest observed memory reclamation, and it operates within a background job context, keeping heap scan overhead off the consensus and RPC paths. Only the sweep-based trigger was retained.

The best-performing trigger also determined where the call should be made from. Invoking malloc_trim within each cache's sweep required throttling for safety. A single invocation after Application::doSweep completes all cache sweeps does not require throttling, as the call frequency is limited by the sweep interval and each invocation addresses the full release cycle. The final implementation includes a single call in Application::doSweep, with the rate limiter and re-entrancy guard removed.

Trim padding

The pad argument of malloc_trim reserves free space at the top of the heap so that the next allocations do not have to grow the heap again. A non-zero reserve could in principle reduce the cost of refilling the caches after each sweep. In practice, only the main heap honours pad, and the per-thread arenas that serve xrpld's worker threads ignore it, so a reserve could protect at most the main heap's share of the memory.

Four 12-hour test runs were executed on the lab validator connected to Mainnet. Trimming was invoked from Application::doSweep with pad set to 0, 256 KiB, 1 MiB, and 16 MiB. RSS was sampled at 10-second intervals, and the duration of each trim operation was logged. To eliminate the impact of the initial ramp-up, RSS statistics are reported from the second hour of each run onward.

`pad`	Mean RSS (GiB)	P95 RSS (GiB)	Trim calls	Mean trim (ms)	P95 trim (ms)
0	15.64	16.22	670	163	393
256 KiB	15.91	16.50	706	171	373
1 MiB	16.01	16.65	645	179	416
16 MiB	15.77	16.72	621	166	372

The differences are small. Mean RSS varies by 0.4 GiB across the four settings, under 3%, and mean trim duration by 16 ms, about 10%. Each setting was run once, in sequence, against whatever Mainnet traffic arrived during its twelve hours, and variation between runs from load alone is of the same order. The comparison can therefore rule out a large effect but cannot rank the settings. A small effect is also what the semantics predict, since only the main heap honours the reserve.

The default was kept. pad is zero, and it is a compile-time constant rather than a configuration setting, so that the change adds no tuning surface to the node.

Memory results

The change was measured in three tests: a lab machine connected to Mainnet, a closed performance network under a fixed transaction load, and two of Ripple's Mainnet validators over the summer.

Lab machine

The lab validator connected to Mainnet ran two builds on the same hardware and settings: the unmodified binary, and a binary with the trim triggers as they stood in November 2025, which called malloc_trim from the application sweep and from the two online-delete points, of which only the sweep call shipped. RSS and Referenced memory were sampled every 10 seconds, and each trim call recorded the process RSS before and after it. The baseline ran for about 24 hours, the trimmed build for about 40, and one run was made of each. The trimmed run is the one shown in the candle charts in the Trigger selection section.

Baseline. RSS grew from 14.0 GiB to 32.9 GiB over 24 hours and was still rising when the run ended.
Trimmed. RSS climbed to 16 GiB within the first hour, to 17 GiB by the sixth, and then stayed within an 18–20 GiB band for the rest of the 40 hours, drifting upward at about 30 MB per hour.
At 24 hours of process life, RSS was 18.3 GiB in the trimmed build against 32.9 GiB in the baseline, 44% lower. The trimmed value is read from the trim log's per-call RSS record, so the two processes are compared at the same age.
Over the 40 hours, the trimmed build made 1,389 calls, 1,173 of them from the sweep. 91% of them reduced RSS, by a median of 75 MB and a maximum of 1.38 GB, and the observed RSS decrease across all calls summed to about 147 GB, against a net RSS growth of about 14 GB.
RSS minus Referenced in the trimmed steady state was about 0.2 GiB on a 19 GiB process, about 1%, so almost all resident memory had been accessed since the window opened. ### Performance network

The performance lab runs a closed five-validator network under a fixed transaction load. Comparison tests with and without malloc_trim each ran for five hours. Validator nodes used 2.58 GB less RSS with trimming enabled, a reduction of 14.8%. Client-handler nodes used 3.33 GB less, a reduction of 18.2%. The CPU and consensus-timing measurements from the same runs are in the Overheads section.

Mainnet validators

Two of Ripple's validators connected to Mainnet, one on the default UNL (dUNL) and one not, ran 3.2.0 without heap trimming from 16 June to 31 July 2026 and 3.3.0 with it from 7 August. The trim call was removed from 3.2.0 the day before its release, so the 3.2.0 builds in this comparison do not contain it, although the release notes initially listed the change. The week between the windows saw the 3.2.1 hotfix release and an unrelated network incident, and was excluded. The 3.2.0 windows close before the incident, and leaving out their final three days changes the plateau means by less than 0.05 GiB. Process RSS is recorded by Prometheus from the process exporter, and both windows on both nodes open on a fresh start of the process, so the two versions are compared at the same age. The figures below are the plateau each node settled at after its first five days.

Node	3.2.0, no trim	3.3.0, trim	Change
Non-dUNL validator (`online_delete` 4,096)	20.1 GiB	17.4 GiB	−2.7 GiB, −13%
dUNL validator (`online_delete` 50,000)	19.4 GiB	15.0 GiB	−4.4 GiB, −23%

Resident memory of two of Ripple's Mainnet validators, 3.2.0 without heap trimming and 3.3.0 with a trim after every sweep, each window opening on a fresh start and plotted by days since start. Dotted lines are the plateau means after day five. The dUNL node's two-day rhythm is online delete: it rotates its history every 50,000 ledgers, about every two days, and under 3.3.0 each rotation's release leaves RSS instead of staying with the allocator. The non-dUNL node rotates every 4,096 ledgers, every few hours, so its releases are small and its line is smooth.

This is a comparison between versions, and 3.3.0 differs from 3.2.0 in more than the trim call. The reduction is nonetheless attributable to trimming: nothing else in the release targets memory retention, the production graphs show the sawtooth of release and regrowth that trimming produces, and the reduction is in line with the lab and performance-network tests, where malloc_trim was the only difference between the test runs compared.

The Mainnet validators test has the most weight of the three, since these are production nodes. The windows are 44 and 38 days long, the savings appear on both nodes, and they are well above the ordinary drift either node shows over a window of that length.

Overheads

Trim latency

The MallocTrim helper was instrumented to measure the cost of each call. With debug logging enabled, the helper records, for every invocation, the process RSS before and after execution and the wall-clock duration.

On the lab validator connected to Mainnet, a sweep trim took a median of 141 ms and a 95th percentile of 340 ms, at roughly one call per minute, which is a fraction of a per cent of the sweep interval, spent on a background thread. The four 12-hour padding runs, with 620 to 706 sweep trims each, gave the same picture, with mean durations of 163 to 179 ms. The duration of a call grows with the size of the heap and with the amount of memory freed since the previous sweep, so nodes with a different node_size or online_delete configuration may see longer calls.

The trim operation executes as part of the background sweep job, once all cache sweeps have completed and outside any cache lock, running on the sweep thread. In glibc, malloc_trim acquires each arena's lock sequentially during trimming. Threads attempting allocation or free operations on a locked arena are blocked until the lock is released. The following subsection presents consensus-timing measurements to assess the practical impact of these waits.

CPU and consensus timing

In the performance lab, a five-validator network was used to execute comparison tests with and without malloc_trim, each for five hours under a fixed transaction load. CPU usage on validator nodes increased by 0.7% with trimming enabled, while client-handler nodes showed a 3.4% decrease. Both changes are small. Consensus timing was evaluated using Cliff's delta with Benjamini–Hochberg correction across all metrics. No changes were detected in ledger close interval, its 80th percentile, voting duration, or transaction-set acquisition time. Per-transaction apply, flush, and build times showed statistically significant differences, as expected given the number of ledgers processed, but all effect sizes were below the 0.147 threshold considered negligible by this method. The test was repeated several times with consistent results, and the figures above are from one round.

Stability

Two checks were made on whether trimming affects a node's participation in the network.

The first comparison was made during the 3.2.0 release-candidate period, when a Mainnet node ran candidate builds with the trim call for 22 effective days and a candidate build without it for 4.65 days. Departures from the full state occurred at rates of 2.41 and 2.15 times per day, with median durations of 3.16 and 3.13 seconds, respectively. Consensus step-outs were observed at 2.23 and 2.15 times per day. None of these differences, nor the difference in daily counts of SHAMapMissingNode events, was statistically significant. The comparison is limited by the short duration of the window without the call, so only large effects can be excluded.

The second check was conducted as part of an investigation into slow initial synchronisation on Devnet during the 3.3.0 release candidates, where new nodes required 36 to 59 minutes to reach full synchronisation. Trimming was evaluated as a potential cause. In the slowest of three instrumented runs, trimming accounted for 305 calls totalling 72 seconds during a 59-minute sync, representing about 2% of wall time, with approximately 19 seconds occurring within the synchronisation window. Sync time did not vary with sweep interval settings of 60, 600, or default values, and trim pauses did not align with the acquisition timeouts that defined the slow runs. The investigation found no evidence that trimming contributed to slow sync times. The causes of the slowdown lay elsewhere and are outside the scope of this post.

Outstanding questions and next steps

Introducing explicit heap trimming after the cache sweep consistently reduced resident memory usage on xrpld nodes across all measured environments: by 13% to 23% on Ripple's Mainnet validators, by 15% to 18% in the performance network, and by 44% on the lab machine under the heavier load of late 2025. The call cost a fraction of a per cent of the sweep interval on a background thread, and had no measurable effect on consensus timing or node stability. This change was released in version 3.3.0, implemented at a single call site without additional configuration parameters.

The observed reduction in resident memory varied widely: 23% and 13% on the two Mainnet validators, 15% to 18% on the performance network, and 44% on the lab machine. This variation exceeds what can be explained by measurement differences alone and requires further investigation. One hypothesis fits the evidence so far: the reclaimed memory is memory the allocator retained after short-lived allocations were freed, so its volume follows the rate of allocation churn, which in turn follows transaction load. The largest reduction was recorded in November 2025, during a period of network-wide memory growth, while the Mainnet measurements were made in the summer of 2026. If transaction volume or composition during that period drove the growth of retained memory, the saving would shrink as load normalised. The stabilisation of untrimmed validators at a plateau is consistent with this hypothesis. Verification will require historical correlation of network load with memory metrics and controlled lab experiments with varied transaction mixes.

Several areas for further optimisation have been identified:

Trim frequency. Performing trimming on every sweep is straightforward but may not be the best schedule. Adjusting the schedule to trim every few sweeps, or only when a sweep releases memory above a defined threshold, could reduce the number of trimming operations while retaining most of the memory savings.
Padding. The initial four single runs excluded a large effect from pad and did not allow for ranking of different settings. Conducting repeated, interleaved runs is necessary to determine whether maintaining a small memory reserve provides measurable benefit.
Residual growth. The gradual increase in memory usage observed with trimming enabled reflects the live working set, which cannot be reduced through allocator configuration. Addressing this requires revisiting the cache experiments from part one and re-running them in the updated lab environment with trimming enabled.
Allocation churn at the source. Trimming returns memory after the application has freed it. It does nothing to reduce how much the application allocates and frees in the first place, and that churn is what causes the fragmentation the trim works around. The complementary work is to find which code paths allocate the most, using the same profiling as in part one, and to reduce it where possible: allocating fewer short-lived objects, or reusing them instead of recreating them.

Memory in an XRPL Node: Hunting a Leak That Wasn't There

Valentin Balaschenko — Thu, 05 Nov 2026 15:13:41 +0000

For operators

If you measure your node's memory, read the xrpld process, its RSS from /proc/<pid>/status or a process exporter, not the systemctl status figure, which includes the kernel's page cache.
If you run xrpld 3.3.0 or later, your node already trims the heap after every cache sweep. There is nothing to configure.
If you build or run xrpld with an allocator other than glibc's, the trim does not apply.
If your memory graph has become a fine sawtooth since upgrading, that is the trim working, not a fault.

Memory ballooning

Memory growth in XRPL nodes was observed in late summer 2025. Operators reported increasing memory usage on their validators that did not decrease, including on nodes configured identically to others that were unaffected, and across different hardware platforms. The cause turned out not to be a memory leak but the memory allocator retaining memory the application had already freed, and a fix shipped in xrpld 3.3.0. This post describes how that was established.

A memory regression identified in one release during that period was addressed within weeks, but this had no observable impact for operators. The observed memory behaviour existed before the regression and persisted after the fix. The underlying issue remained unresolved at the time. Operators monitoring node caches asked why memory did not come back after a cache sweep, the routine pass that clears expired entries.

Early in the community discussion, it became clear that consistent measurement was essential. Some reports referenced memory usage from systemctl status, which reflects the memory of the service's cgroup. A cgroup is the kernel's resource-accounting container that systemd wraps around every service. In contrast, per-process readings from /proc/<pid>/status or smaps report only the process memory. These two metrics often differed by several gigabytes due to the kernel's page cache. When xrpld (formerly rippled), the XRPL node software, reads from or writes to its database files, the kernel caches these file pages in memory. This cached memory is not included in the process's Resident Set Size (RSS), the kernel's count of pages held in RAM by the process, because xrpld's database library, NuDB, uses standard file I/O rather than memory mapping. The kernel charges this cached memory to the service's cgroup because xrpld initiated the I/O. On busy nodes, the page cache can reach several gigabytes, increasing as the node accesses its database and decreasing when the kernel reclaims memory or when online delete removes old database files. These fluctuations can occur within minutes and are not controlled by the application. As a result, the same machine can report very different memory usage figures depending on the measurement method, leading to differing interpretations by operators.

On a dedicated xrpld machine, it may appear that the distinction between process memory and total memory usage is irrelevant, since it is all the node's memory in the end. However, the two types of memory are managed differently. The kernel controls the page cache, which grows to use available RAM and causes the system-wide memory figure to reflect total machine capacity rather than the process state. In contrast, the process's own memory cannot be dropped by the kernel, only swapped, and it is the only part that code changes can affect. Per-process memory is therefore the right measure of the application's own behaviour, and all subsequent analysis uses it. RSS is the coarsest of these, since it counts every resident page whether or not the process still touches it. The next section adds a finer one, the working set, which counts only the memory the process has actually accessed over a period.

A memory leak was the first suspicion. Agreeing on how to measure memory made it possible to test that suspicion against the alternatives: normal cache operation, a feature retaining more state than expected, or an issue below the application layer. Without the ability to answer this reliably, for any xrpld version and whatever mix and volume of traffic the node handles, using trusted metrics, any conclusions would be speculative.

A toolset was assembled to collect and analyse the relevant measurements. The following analysis describes the investigation, the findings, the attempted fixes, and the eventual realisation that the root cause was located in a different layer of the stack.

The toolset

The investigation focused on three key questions, with the requirement that no conclusion would rest on a single tool.

What are the primary sources of the increased memory usage?
What mechanisms are responsible for the observed memory growth on some nodes but not others? For instance, which operational flows, features, or traffic patterns trigger it?
How reliable are the measurements under real-world conditions? This had to be answered before the other two meant anything. The previous section established that the focus is on process memory, not system-wide memory. The remaining concern is the reliability of each measurement tool. Each instrument has limitations: RSS can be affected by glibc allocator behaviour, flame graphs may attribute bytes to generic frames if stack traces are incomplete, and the kernel's Referenced bit provides only an approximation. Using multiple tools and cross-verifying results is necessary to ensure accuracy.

The approach uses three measurement methods, with visualisation applied to the collected data.

A procfs-based working-set tool, specifically Brendan Gregg's wss.pl, uses /proc/<pid>/clear_refs and /proc/<pid>/smaps to estimate Working Set Size (WSS) over a defined window. This tool is integrated with a custom sampling loop that timestamps each reading and records gross totals. It is a low-overhead, always-on, coarse-grained measurement.
eBPF/BCC memleak traces allocator calls using uprobes, which are dynamic tracepoints on user-space functions, with aggregation performed in the kernel. This method is lightweight and provides call stacks for outstanding allocations.
A custom glibc shim, implemented as an LD_PRELOAD library, is inserted by the dynamic loader ahead of libc to intercept every allocator call. It interposes on malloc, free, calloc, realloc, posix_memalign, and related functions to capture each allocation with a backtrace. Unlike the sampling tools, it records every allocation and every free, so it can answer questions about individual allocations that aggregated data cannot. The cost is a several-fold slowdown and a dedicated run per question, so it serves as the ground truth rather than as a monitoring tool.
Flame graphs are generated using Brendan Gregg's FlameGraph toolkit, with input from the eBPF data, to visualise and analyse call-stack data. These tools represent a trade-off between overhead and detail. The investigation begins with coarse, low-overhead measurements, and more detailed analysis is performed only when initial results indicate where to focus.

Layer 1: procfs-based WSS

The primary objective is to determine whether the additional resident memory is memory the process keeps accessing, or memory it holds but has not touched for some time, such as cache entries that were filled once and never read again. RSS alone cannot make this distinction, as it counts all pages in RAM regardless of recent access.

The Linux kernel maintains a per-page Referenced bit, which is set when a page is accessed and can be cleared on demand. wss.pl uses this mechanism by clearing all bits, waiting for a specified interval, and then measuring how much of the process's memory has been accessed during that period, representing the working set for the window. In cumulative mode (-C), the tool clears the bits once and observes Referenced increasing toward RSS as pages are accessed. Only the clearing operation is costly, so sampling frequency can be adjusted as needed.

Eleven and a half hours of the ballooning on our System76 lab machine, September 2025 — wss.pl in cumulative mode, 60 s samples. RSS climbs at ~0.74 GiB/h — spikes partially recover, but the floor only rises. Referenced converges toward RSS — the growth is being accessed, not sitting cold — and the band between them narrows from ~9 GiB to ~3.8 GiB.

The key metric is the difference between RSS and Referenced, representing resident memory that has not been accessed since the start of the measurement. This includes caches, fragmented allocator heaps, and retained but idle structures. The gap narrows over the run as Referenced approaches RSS, indicating that most held memory has been accessed since the start of the measurement. However, the Referenced bit does not distinguish between untouched pages containing live but inactive data and other types of memory. Accurately identifying the nature of this gap is central to the investigation.

Leaks associated with growing live structures, such as caches, queues, or in-flight buffers, result in Referenced increasing in parallel with RSS, since the leaked memory continues to be accessed. Cold leaks, where memory is allocated but the pointer is lost and never accessed again, appear only as an expanding gap and cannot be distinguished from other untouched memory using this method. This limitation is one reason WSS is not used in isolation.

Two caveats:

The clear_refs mechanism provides only an approximate measurement. The kernel traverses page tables to set and read the Referenced bits, introducing a small amount of latency during sampling, around 5% according to Gregg's documentation. Excessive sampling can interfere with the kernel's page-aging information and affect memory reclamation decisions. To mitigate these effects, long sampling intervals are used, and measurements are conducted primarily in a controlled environment.
Tracking memory utilisation using this method is considered experimental and is documented as such in sources including LWN and Brendan Gregg's WSS post. This approach provides a baseline measurement rather than a definitive conclusion.

Layer 2: eBPF + BCC memleak

WSS indicates that memory usage is increasing but does not identify the specific code locations responsible. To obtain this information, BCC's memleak tool is used.

memleak attaches uprobes to the allocator entry points in libc for a specific process. Each allocation is recorded with its size and stack trace, and each free operation removes the corresponding record. After a period of execution, memleak reports the call stacks associated with the largest outstanding allocations. By sampling these reports at regular intervals, it is possible to track how much memory each subsystem has allocated and not yet freed over time.

This approach is feasible for production workloads because BCC aggregates data within the kernel, transmitting only summary information to user space. Streaming every malloc event instead would produce more data than could be collected without slowing a process that allocates as frequently as an XRPL node. Testing showed no measurable impact on node throughput. For accurate stack traces, builds must include debug symbols and frame pointers. Without them, call stacks may be incomplete.

For analysis beyond the largest allocation sites, the byte-attributed data is processed with flamegraph.pl. The width of each box represents the number of bytes attributed to that stack, with the largest boxes indicating the heaviest allocation sites. This visualisation enables rapid identification of the heaviest allocation sites.

Layer 3: the glibc shim

eBPF aggregates allocations by call stack, which identifies the subsystem responsible but not the individual allocation, its size, or the order in which allocations and frees occurred. The glibc shim records exactly that, every allocation and free with a backtrace, at the cost of slowing allocation-intensive code several-fold. It is therefore reserved for targeted runs that answer a specific question about individual allocations. The remainder of this analysis did not need that level of detail, so only the first two layers are used.

Following the bytes: TaggedCache

The SHAMap, which maintains the ledger state, does not manage the lifetimes of its nodes directly. This responsibility is handled by the TaggedCache, a partitioned, expiry-based cache that supports the TreeNodeCache (for tree nodes), the FullBelowCache (tracking locally complete subtrees), and several additional structures. Profiling a live node with memleak identified the TaggedCache subsystem as the primary consumer of outstanding memory. Key components include canonicalize, which consolidates node copies into a single shared instance, the refresh loop (freshenCaches) that copies cached entries to the new database file during online delete, and sweep threads responsible for evicting expired entries. Most of the remaining memory usage was attributed to the creation of SHAMap nodes and items managed by the cache, including fetch-and-deserialise operations and inbound ledger-data handlers. The node was configured with node_size set to medium, which determines cache sizes in xrpld, and online delete was enabled.

Peak outstanding allocations were measured on a test-lab validator connected to Mainnet in September 2025. Here, 'outstanding' refers to memory allocated after memleak was attached and not yet freed, representing allocations since tracer attachment rather than the process's total memory usage. The graph displays the total outstanding memory as its full width, with each category's share indicated by horizontal width: dark for TaggedCache machinery, light for cache-managed objects, and grey for other allocations. Stack traces were recoloured by category for clarity. The original flame graph is available for reference.

The data was collected with node_size set to medium, indicating that the observed memory usage was not due to unusually large cache configurations. This pattern persisted across varying traffic levels. In all peak snapshots of outstanding memory, 84–99.7% was attributed to cache operations, primarily under SHAMapStoreImp::freshenCaches, which re-populates the new database file from cache keys during online delete. Transient memory spikes were also associated with cache maintenance, proportional to the cache's key population.

A flame graph attributes memory usage to the code responsible for allocation, not to the structure that ultimately holds the memory. In this codebase, allocation occurs during fetch and deserialisation, and nodes are subsequently managed by the cache through canonicalize. Therefore, the data identifies a likely source of memory usage, but confirming this would require further experimental validation.

Cache experiments

At this stage a likely cause had been identified. Over several months, a series of experiments was conducted to make the cache smaller, cheaper, or better behaved. Each experiment tested a specific theory regarding memory usage.

Theory: some caches may be unnecessary. The initial approach was to disable the TaggedCache and observe the impact (#5439, with initial commits titled "TEST: Disable tagged cache to measure performance"). Disabling the TaggedCache revealed its role as both a cache and the canonicalization point, as well as its function in warming the new database file during online-delete rotation. Complete removal was not feasible. The cache could only be made smaller or shorter-lived, and because rotation warms the new database file from the cache's keys, shrinking it trades memory for post-rotation latency. The same pull request then focused on the NodeObject cache in DatabaseNodeImp, which stored recently read database records in memory despite the underlying database engine and kernel page cache already retaining these records. Removing this cache reduced memory usage by 3–5% with a minor increase in CPU use.

Theory: the cache structure leads to memory waste on large hardware. The TaggedCache is partitioned per hardware thread, causing fixed overhead to scale with the number of cores rather than the workload. Reducing the partition count by half (#5442) resulted in a 5–6% memory reduction.

Theory: cache entries persist longer than necessary. The sweep evicts entries that are unreferenced or expired. To test this, a probe build (#5910) set all entries to expire immediately, converting each sweep into a full eviction. This approach reduced memory usage by 8–10%.

Theory: some structures are never cleaned. The LedgerHistory::ledgersByIndex_ map, which associates ledger sequence numbers with hashes, was not subject to sweeping and accumulated entries indefinitely. This resulted in a small but unbounded growth, a logical leak: entries were added for every validated ledger and never removed (#5461). It was the only such leak the investigation found. The fix in #5874 bounds the map on nodes with online delete enabled. Full-history nodes, which keep every ledger, still accumulate about 1.4 MB per day. The growth was measured in kilobytes per hour while the overall issue was measured in gigabytes, so it did not account for the observed memory growth.

Individually, each change showed progress. However, the combined effect was less than expected. While the sum of individual improvements suggested a 20% reduction, the cumulative measurement in a single build was only 5–7%. This discrepancy indicated either unreliable measurements or a lower bound imposed by factors outside the application. Both explanations turned out to be true, and the second was the larger problem. The experiments remain relevant, each demonstrating measurable effects with tools that have since been improved. Further testing in a rebuilt lab environment is planned, but the focus has shifted to addressing the underlying constraint.

Of the four changes, the NodeObject cache removal (#5439) was merged and later reverted as a precaution, without any problem having been attributed to it, the partition-count reduction (#5442) remains open, the sweep-everything build (#5910) was a probe never intended to ship, and the ledgersByIndex_ fix (#5874) was merged in October 2026.

Released memory is not reclaimed

In October 2025, the investigation moved from periodic snapshots to continuous monitoring. The memleak-based sampler measured TaggedCache's outstanding bytes, everything its stacks had allocated and not yet freed, every few tens of seconds during hour-long runs. In parallel, WSS tracked RSS and Referenced.

The results were clear. The outstanding allocations from the cache subsystem consistently returned to near zero on each sweep cycle, which occurred approximately every two minutes. The sweep-everything probe from #5910, which evicts the entire cache at each sweep, amplified this effect. Allocation and deallocation rates matched in real time on the unmodified build. Despite this, RSS and Referenced memory continued to increase.

Seventy-two minutes on the unmodified develop build, connected to Mainnet, 21 October 2025. Top: bytes allocated by TaggedCache-machinery stacks and not yet freed — the memleak sampler, ~25 s snapshots. memleak reports the ten largest outstanding stacks per snapshot, so TaggedCache absent from the top ten plots as zero. Peak 64 MiB, median 2 MiB, every excursion collapsing within minutes (sweep interval ≈ 121 s). Bottom: the same process on the same clock — RSS climbs 13.7 → 15.5 GiB (+1.7 GiB, ~1.5 GiB/h) while Referenced converges toward it (wss.pl, cumulative mode). That rate is faster than September's 0.74 GiB/h — it varied with the run and the node's age — but the direction never did. One of four captures that day across both builds — the cache's outstanding bytes stayed in the MiB range in every one of them. This run is shown because it has a full RSS/Referenced companion on the same clock.

This combination of results rules out an application-level cause for the growth, including the cache. The cache's outstanding allocations are measured in MiB, while the observed memory growth is in GiB. Across all stacks reported by memleak, the median outstanding memory was 0.02 GiB, increasing by approximately 0.1 GiB during the run, compared to 1.7 GiB of RSS growth. Anything accumulating toward that figure would have appeared in the top ten stacks. Memory that the application has freed but the kernel still counts as resident, and that is still being touched, is not a leak and does not indicate cache malfunction. This behaviour is the allocator's.

When xrpld calls free, glibc's malloc typically retains the pages on its internal free lists, anticipating future reuse by the application. The kernel cannot request memory back from the allocator. Under memory pressure it can drop page cache or swap pages out, but it cannot instruct malloc to release its holdings. glibc only returns free space at the end of each heap when the unused tail exceeds a threshold. Freed memory blocks that are interleaved with live allocations, as is common with caches containing objects of mixed lifetimes, remain stranded within the heap. This is fragmentation. The allocation pattern of an XRPL node, which involves bursts of millions of small, short-lived tree nodes at sweep and rotation boundaries, and the use of per-thread arenas (each busy thread receives its own heap region), leads to fragmentation in every arena. This accumulates to gigabytes of memory that are free from the application's perspective but remain resident according to system metrics.

The key distinction is between memory released by the application and memory reclaimed by the allocator. The application can release memory, but only the allocator can reclaim it. There was no mechanism in the node's code path to connect these actions.

The investigation focused on identifying a memory leak. No leak of any consequence was found. The allocator was retaining memory as part of a general-purpose strategy that is poorly suited to this workload. The application can precisely identify when memory is no longer needed, but lacks a mechanism to inform the allocator.

There is a solution available. glibc provides malloc_trim, which can be invoked by xrpld at an appropriate point in its operation. The implementation details, performance impact, and effects on memory usage in Mainnet validators will be discussed in the next post.

The diagnostic toolset fulfilled its purpose by identifying the root cause: the allocator handled xrpld's allocation pattern poorly, and neither side had a way to tell the other what it knew.

Acknowledgements

This investigation began with reports from operators running the software on Mainnet, who measured the growth on their own nodes, compared notes on how to measure it, and kept asking the questions that turned out to matter. Thanks are due to them, to those who reviewed the change in the open-source repository, and to those who reviewed these two posts.