Mustafa ERBAY

Posted on Sep 17 Originally published at mustafaerbay.com.tr

Whose Ten Percent Is dirty_background_ratio

#linux #kernel #pagecache #writeback

My server says vm.dirty_background_ratio = 10 and vm.dirty_ratio = 20, the distribution defaults. On a machine with 94 GiB of RAM, /proc/vmstat turns those two lines into numbers: nr_dirty_background_threshold 2028661 pages (7.7 GiB), nr_dirty_threshold 4062282 pages (15.5 GiB). The disk writes at 250 MB/s with oflag=direct. So if 15.5 GiB of dirty pages pile up, draining all of it takes more than a minute, and during that minute everyone who calls fsync joins the queue. I was going to open this piece with "on a big-RAM machine, write dirty_bytes instead of a percentage"; the measurements pushed that into second place.

I wrote a 4 GiB file. Background writeback did not wait for 7.7 GiB; it started at the flusher's first wakeup. The ten percent was right; what was wrong was my assumption that it applied to the system. The kernel multiplies that ten percent by a share it assigns to the writing process's cgroup, and a cgroup that has never written has a share of zero. Below is where that share comes from, how to measure it, and what the dirty_* knobs really decide on a container host. The kernel measured is 6.8.0-139 (Ubuntu 24.04); the places that changed in later versions are noted separately at the end.

Ten percent of what

The kernel documentation defines both parameters with the same sentence: "a percentage of total available memory that contains free pages and reclaimable pages", and warns right below that "the total available memory is not equal to total system memory". In the source this is global_dirtyable_memory(): free pages minus the kernel's reserve, plus active and inactive file pages. With my server's numbers: 18,630,070 free + 1,066,432 inactive file + 785,659 active file = 20,482,161 pages; since twenty percent of the base is 4,062,704 pages, the base is 20,313,520 pages and the 168,641-page difference (659 MiB) is the reserve. Anonymous memory is not part of the base; it is not exactly MemAvailable either.

There is a third line between the two thresholds that the documentation does not mention. balance_dirty_pages() does not start slowing a process at dirty_ratio; in the code it is called dirty_freerun_ceiling, and its value is (thresh + bg_thresh) / 2. With the defaults that is 15 percent, 11.6 GiB on my server. The brake's balance point (setpoint) sits above that, halfway between the free-run ceiling and the limit; once twenty percent is reached, the process is forced into sleeps of up to 200 ms (MAX_PAUSE, HZ/5). There is also the question of how often the check runs: balance_dirty_pages is not called on every write but when the pages a process has dirtied reach its own nr_dirtied_pause quota or the per-CPU counter ratelimit_pages. The latter is dirty_thresh / (CPUs × 32), 7,052 pages or 27.5 MiB on this 18-vCPU machine; the comment in the source chose the number so that "when all CPUs are dirtying in parallel, we cannot go more than 3% over the dirty memory thresholds".

Two more parameters bound time: dirty_writeback_centisecs=500, the flusher waking up every 5 seconds; dirty_expire_centisecs=3000, an inode counting as old and being written 30 seconds after it was dirtied. There is another 30 in my server's /etc/fstab too: discard,commit=30,errors=remount-ro. That is the default the Ubuntu cloud image has written since 2023 (livecd-rootfs 2.804, lunar); they moved a patch from the linux-kvm kernel into fstab. That 30 is the age of the ext4 journal transaction; it has nothing to do with the page cache, it only reads that way when the two sit side by side.

4 GiB, first wakeup

The test rig is the server itself: Ubuntu 24.04, kernel 6.8.0-139, ext4 root, a QEMU virtual disk, 18 vCPUs, sixty-odd containers on top. One script records Dirty/Writeback from /proc/meminfo and the MiB written to sda from /proc/diskstats once a second; a separate process writes to its own 4 KiB file every 250 ms and times the fsync. The write job is dd if=/dev/zero bs=1M count=4096 in a fresh cgroup opened with systemd-run --scope, no sync. The writeback:* tracepoints are on, and the ftrace clock is set to boot (I will explain why shortly).

t+1.2   Dirty=493 MiB    Writeback=0     sda_w=0
t+3.3   Dirty=1109 MiB   Writeback=0     sda_w=52 MiB
t+6.5   Dirty=2056 MiB   Writeback=0     sda_w=0
t+7.5   Dirty=2363 MiB   Writeback=0     sda_w=0
t+8.6   Dirty=1747 MiB   Writeback=14    sda_w=745 MiB
t+11.8  Dirty=744 MiB    Writeback=55    sda_w=520 MiB
t+15.0  Dirty=29 MiB     Writeback=0     sda_w=453 MiB     (dd finished: 14.9 s, 289 MB/s)

Dirty peaked at 2.3 GiB and melted from t+8 on at 520-745 MiB/s; in this run the global counter never got within even a third of 7.7 GiB (the virtual disk that gives 250 MB/s for oflag=direct writes that wait one at a time shows this rate under the kernel's queued drain). The small 52 MiB write at t+3 is no coincidence either; the tracepoints explain both together. The lines below belong only to this cgroup's wb (filtered by cgroup_ino, nr_pages differences in parentheses):

561370.456  H_START (dd started)
561372.999  writeback_start   reason=periodic   kupdate=1
561372.999  writeback_queue_io age=30000 enqueue=0  reason=periodic      ← no 30 s old inode
561373.002  writeback_start   reason=background
561373.002  writeback_queue_io age=0     enqueue=3  reason=background
561373.045  writeback_written reason=background                          (13,314 pages = 52 MiB)
561378.119  writeback_start   reason=periodic   kupdate=1
561378.119  writeback_queue_io age=30000 enqueue=0  reason=periodic      ← still no 30 s old inode
561378.165  writeback_written reason=periodic                            (13,312 pages = 52 MiB)
…          130-odd more chunks in the same shape, 2836 MiB in total, all reason=periodic

The flusher's first wakeup is at t+2.5 (my sampling script started writing in the same cgroup three seconds before dd, so the 5-second timer was armed by it). The wakeup does the kupdate job first: it looks for inodes older than 30 seconds and finds none (enqueue=0). Then it asks the background question and the answer is yes: it writes one 52 MiB chunk, on the next round the answer turns to no, the loop breaks. Five seconds later, the second wakeup: kupdate again, again no old inode, but this time it writes the whole inode, 2.8 GiB, with reason=periodic. The age filter in queue_io() is applied only to the b_dirty list (move_expired_inodes); an inode returning from an unfinished write waits on b_more_io, and that list is spliced onto b_io unconditionally at every wakeup. In other words, once the kernel has started writing an inode, it never puts it through the 30-second rule again. The remaining half of the question is why the background answer was yes at the first wakeup.

Meanwhile, the ftrace clock. In the first runs I saw reason=sync among the writeback_start lines; somebody was calling sync. I spent ten minutes looking for who (cron jobs, timers, watchdog scripts, neighbouring containers); when I caught it with sched_process_exec, the parent was the bash of my own ssh session. The sync I had run before dd showed up in ftrace's local clock 7 seconds after dd. I measured it: the local clock was 9.31 seconds ahead of /proc/uptime (559746.55 versus 559737.24); after echo boot > trace_clock the two agreed to within 0.006 seconds. The documentation says of local that it "may not be in sync across CPUs"; if you are going to lay ftrace timestamps next to date or /proc/uptime, switch to the boot clock first, or you will find ghost processes like I did.

Who sets the share

Whether background writeback starts is decided by wb_over_bg_thresh(), and the first half of the function is my old reading: if the global dirty page count exceeds bg_thresh, yes. If it does not, the second half kicks in:

thresh = wb_calc_thresh(gdtc->wb, gdtc->bg_thresh);
if (thresh < 2 * wb_stat_error())
    reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
else
    reclaimable = wb_stat(wb, WB_RECLAIMABLE);
if (reclaimable > thresh)
    return true;

wb is a bdi_writeback; with cgroup writeback enabled (it is on ext2, ext4, xfs, btrfs and f2fs; on Ubuntu 24.04 with cgroup v2 that means one per container and per systemd unit) every cgroup has its own wb for every disk. And wb_calc_thresh computes that wb's share: fprop_fraction_percpu(&dom->completions, wb->completions, ...), the ratio of the writes this wb has completed to all completed writes in the domain, times bg_thresh. The ratio is not a count over a fixed window but an exponentially aging average; VM_COMPLETIONS_PERIOD_LEN is 3 seconds and the counters are halved every period. The /sys/class/bdi documentation describes the same mechanism for devices, "each device is given a part of the total write-back cache that relates to its current average writeout speed in relation to the other devices"; the min_ratio/max_ratio there are per device too, not per cgroup. A cgroup that has never written has a share of zero; because a zero threshold is below 2 * wb_stat_error(), the kernel reads the exact sum instead of the approximate counter and says "over the threshold" at the first dirty page.

I wanted to read this rather than infer it. A short bpftrace script that, on entry to wb_over_bg_thresh, records wb->memcg_css->cgroup->kn->id, the return value of wb_calc_thresh, and the final decision:

bpftrace -e '
kprobe:wb_over_bg_thresh { $wb=(struct bdi_writeback*)arg0; @cg[tid]=$wb->memcg_css->cgroup->kn->id; }
kprobe:wb_calc_thresh  /@cg[tid]/ { @dom[tid]=arg1; }
kretprobe:wb_calc_thresh /@cg[tid]/ { @t[tid]=retval; }
kretprobe:wb_over_bg_thresh /@cg[tid]/ {
  printf("cg=%d domain_bg=%d MiB wb_thresh=%d MiB over=%d\n",
         @cg[tid], @dom[tid]*4/1024, @t[tid]*4/1024, retval);
  delete(@cg[tid]); delete(@t[tid]); delete(@dom[tid]); }'

The decisions from the run above:

09:26:54 cg=12265201 domain_bg=7835 MiB wb_thresh=0 MiB    over=1
09:26:54 cg=12265201 domain_bg=7834 MiB wb_thresh=0 MiB    over=1
09:26:54 cg=12265201 domain_bg=7832 MiB wb_thresh=6058 MiB over=0
09:27:04 cg=12265201 domain_bg=7833 MiB wb_thresh=7697 MiB over=0
09:27:10 cg=12265201 domain_bg=7815 MiB wb_thresh=7484 MiB over=0

The first line is the whole story: the domain's threshold is 7,835 MiB, my cgroup's threshold is 0 MiB, decision "over". After a single 52 MiB chunk the share had jumped to 6,058 MiB, because in that three-second period mine was almost the only wb writing to the disk; the decision turned to no, and the periodic wakeup five seconds later wrote the rest. In the previous run I had extended the same experiment with a second dd: another 4 GiB in the same cgroup 20 seconds after the first one finished. Dirty climbed to 4,102 MiB and waited, and never saw 30 seconds; by second sixteen the neighbouring containers' writes had eroded my share to 2,700 MiB and over=1 arrived. Same cgroup, same 4 GiB, seven seconds the first time, sixteen the second. The highest thresholds the other cgroups saw in that run are worth noting too: 0 MiB in twenty-six cgroups, 161, 409 and 1,756 MiB in three; 7,880 MiB only for my dd.

The conclusion sits in one sentence on the cgroup-v2 documentation page, and I had skipped over it for years: the vm.dirty_* ratios "apply the same to cgroup writeback with the amount of available memory capped by limits imposed by the memory controller and system-wide clean memory". The ratios are the same; the base and the share change per cgroup.

The bill for the early drain

Background writeback starting early is hardly bad news; dirty pages spending less time in RAM means less loss in a power cut. The neighbour pays the bill. The data from the fsync probe that ran through the whole experiment, each sample assigned to the second it landed in, splitting seconds with less than 300 MiB/s written to sda from seconds with more:

quiet (<300 MiB/s):  n=4696  p50=5.6 ms   p90=21.6 ms   p99=76.8 ms
draining (≥300):     n=429   p50=42.4 ms  p90=155.0 ms  p99=292.8 ms

A process fsyncing its own 4 KiB goes from a median of 5.6 ms to 42 ms, its p99 leaning on three hundred milliseconds; that the quiet p50 is 5 ms is a reminder that sixty containers share the disk. It is not dd writing the file that does this; it is the 2-4 GiB queue dd leaves behind draining to disk. Calling sync means waiting for that queue too: with 3.7 GiB of dirty pages, time sync took 7.04 seconds. The sync in reboot's last step waits the same seconds in the same place.

A memory limit changes both the share and the threshold

When you give a container memory.max, the dirty_* arithmetic is done in a second domain as well: the memcg domain. Its base is built by mdtc_calc_avail(), the cgroup's file pages plus the headroom left under the limit (capped by the clean memory in the system). The same 4 GiB dd with MemoryMax=1G:

dd: 11.50 s, 374 MB/s
memory.stat file_dirty: oscillating between 60 and 138 MiB, file=983 MiB constant
memory.events max: 12449
balance_dirty_pages tracepoint: 659 checks, sleep time zero in 611 of them

Dirty pages never exceeded 140 MiB; over a 983 MiB base, ten percent is 98 MiB and fifteen percent is 147 MiB. dd's slowdown is not really the brake; the max counter in memory.events says it hit the limit 12,449 times, and every hit means reclaiming from its own page cache. To see the brake itself I put io.max next to the limit, IOWriteBandwidthMax="/dev/sda 50M", 1 GiB of dd:

dd: 17.90 s, 60.0 MB/s
balance_dirty_pages: 1150 checks, sleeps in 1080 of them; 13.5 s in total (three quarters of 17.9 s), 6-100 ms each

The disk is idle, the host's Dirty is 80 MiB, but because cgroup writeback charges the drain of dd's pages to that cgroup's io account, the dirty pages melt at 50 MB/s and balance_dirty_pages locks the process to that rate. An IO limit slows buffered writes too; with a delay, and if there is a memory limit.

What happens when you write dirty_bytes

I tried my original intention too: vm.dirty_background_bytes=268435456, vm.dirty_bytes=1073741824 (256 MiB / 1 GiB; writing them zeroes the ratios, the documentation says "only one of them may be specified at a time"). In a fresh cgroup the 4 GiB dd took 11.86 seconds at 362 MB/s; at the end Dirty was 192 MB, sync 0.22 seconds, balance_dirty_pages zero events. The disk drains at 530 MiB/s, dd dirties at 362 MB/s, dirty pages never touch the 640 MiB free-run ceiling. No brake, no queue, a relaxed fsync neighbour. The cost is that buffered writes now run at disk speed rather than RAM speed.

Two things have to be seen. First, the share mechanism stays in place with dirty_bytes too; the cgroup's share of a fixed 256 MiB threshold also starts at zero. Second is sneakier: a byte threshold is not carried into the memcg domain as bytes. The comment in domain_dirty_limits() is explicit, "the byte settings can't be applied directly to memcg domains; convert them to ratios by scaling against globally available memory". 256 MiB divided by the 77.5 GiB base comes to 0.3 percent, and a memory-limited container applies that percentage to its own base. MemoryMax=1G plus the same dirty_bytes settings, 1 GiB of dd: file_dirty at most 7 MiB, balance_dirty_pages 831 checks, 165 sleeps; the dirty count the tracepoint saw ranged from 5.8 to 13.9 MiB. What you thought was a 1 GiB ceiling on the host is a 13 MiB ceiling inside the container.

The application can manage the queue too

PostgreSQL has solved this on its own side since 9.6 (2016), and the release note politely scolds the kernel: "Many operating systems are not smart about managing this and allow large amounts of dirty data to accumulate before deciding to flush it all at once, causing long delays for new I/O requests until the flushing finishes." The fix is checkpoint_flush_after, default 256 kB on Linux, which "will limit the amount of dirty data in the kernel's page cache, reducing the likelihood of stalls when an fsync is issued at the end of the checkpoint, or when the OS writes data back in larger batches in the background". The call underneath is sync_file_range. I applied the same thing to the 4 GiB write in Python: every 8 MiB, SYNC_FILE_RANGE_WRITE on the new range and WAIT_BEFORE|WRITE|WAIT_AFTER on the previous one (Python 3.12's os module has no sync_file_range, so I called it from libc via ctypes):

plain: write 4096 MiB in 6.89 s (594 MiB/s), max Dirty 4068 MiB, final fsync 7.10 s, total 14.00 s
sfr:   write 4096 MiB in 9.85 s (416 MiB/s), max Dirty 3 MiB,    final fsync 0.01 s, total 9.86 s

Total time dropped, dirty pages stayed at 3 MiB, the closing fsync disappeared. The 594→416 difference that looks like "writing got slower" is fake; the first number is the speed of writing into RAM, and its bill is paid in the seven-second fsync. sync_file_range gives no durability, it touches neither metadata nor the journal; the closing fsync is still needed, it just takes 0.01 seconds now. The same recipe applies to everything that produces large files without being a database (backups, exports, log rotation).

What I would do

On a container host I have stopped reading dirty_background_ratio as the answer to "when does the flush start"; the answer depends on that cgroup's share of writes, and for most cgroups that is zero. If you fear dirty pages waiting in RAM for a long time, this mechanism is already working on your side. The thing to fear is what the queue does to the neighbour while it drains, and without measuring fsync latency that noise stays invisible, even while the Dirty graph looks calm.

On the tuning side my order is this. With this much RAM, dirty_bytes/dirty_background_bytes make the global threshold machine-independent; I found 256 MiB / 1 GiB lossless on this disk, and would write smaller on a slower one. But on a machine running memory-limited containers, that setting cuts the containers' dirty budget down to a few MiB; there, keeping the ratio and giving containers memory.max deliberately is the more honest trade. For your own processes that produce large files, paced writing with sync_file_range does the same job better without touching dirty_*. Know that in a container with io.max, buffered writes drop to that rate too.

Three places are enough for measuring: Dirty/Writeback in /proc/meminfo; DirtyThresh/BackgroundThresh in /sys/kernel/debug/bdi/8:0/stats (the BdiDirtyThresh: 0 kB line there is the root wb's share, and it being zero is no longer surprising); the writeback:* tracepoints, with the ftrace clock on boot. To see a cgroup's current share you need the bpftrace script above on 6.8; on later versions it is easier.

The ten percent is still where it was in the documentation. It is just no longer ten percent of the system, but of whoever is writing to the disk at that moment.

Versions: measurements on 6.8.0-139 (Ubuntu 24.04). In 6.10, wb_over_bg_thresh computes the share directly with __wb_calc_thresh (the kprobe:wb_calc_thresh above stays silent there, attach to __wb_calc_thresh), and /sys/kernel/debug/bdi/<device>/wb_stats shows every cgroup wb's thresholds without bpftrace. In 6.14, __wb_calc_thresh gained a floor: the share does not drop below (thresh − dirty) / 8, and the background decision (wb_bg_dirty_limits) uses this floored value; a fresh cgroup's background share is no longer 0 but roughly 0.9 GiB in a 7.7 GiB domain. The "chunk write at the first wakeup" behaviour in this article may not repeat in the same form on those versions; the share division itself is still there.

DEV Community