Mustafa ERBAY

Posted on Sep 21 Originally published at mustafaerbay.com.tr

Slot Zero Is Right: Which Copy Does md repair Believe?

#linux #mdadm #raid #kernel

There is a sentence in the "Scrubbing and mismatches" section of md(4): for RAID1 and RAID10, repair means "all but one block are overwritten with the content of that one block." Which block? The manual doesn't say. I have read past that sentence for years with a vague assumption in my head, something like "presumably a majority vote or a checksum." Today I got the answer from the source code and from a lab: the disk in the lowest slot number. No vote, no checksum. Slot zero is right; if it is wrong, it copies its wrongness to everyone else.

This post follows that sentence. On VPS3 I built a RAID1, a RAID5 and a RAID6 out of loop files, corrupted eight bytes of a single 4 KiB block in each, and watched what check, repair and the distribution's monthly scrub timer did with it. Some results were what I expected; two were not. The first surprise was RAID1 serving the corrupted copy to readers under load. The second was the mismatch_cnt counter remembering only the last segment on machines where the monthly scrub runs in pieces. In the dm-integrity and dm-verity posts I asked "who finds the bad block" of the checksumming layers; this post asks the same question of the layer that has no checksum, md itself.

The timers wake up when an array appears

The setup is boring: 256 MiB files from truncate, losetup, mdadm --create. What is interesting is what happened in the second I created the first array. Right after mdadm --create /dev/md/lab1 --level=1 --raid-devices=2 /dev/loop6 /dev/loop7, three lines appeared in systemctl list-timers that had not been there before:

Tue 2026-09-22 06:53:09 +03   mdcheck_continue.timer   mdcheck_continue.service
Tue 2026-09-22 22:57:59 +03   mdmonitor-oneshot.timer  mdmonitor-oneshot.service
Sun 2026-10-04 03:43:33 +03   mdcheck_start.timer      mdcheck_start.service

These ship with the package, systemctl is-enabled says "enabled", and until that moment they had never run. The reason is in the [Install] section: WantedBy=mdmonitor.service. The timers hang off a service, not a target, and mdmonitor.service is started by the 63-md-raid-arrays.rules udev rule when an array with MD_LEVEL matching raid[1-9]* shows up. No array, no monitor; no monitor, no monthly scrub. Worth knowing, because the answer to "why has the mdcheck timer never run on this box" is sometimes "because the array is RAID0."

The timer itself is the systemd counterpart of Ubuntu's old /etc/cron.d/mdadm: OnCalendar=Sun *-*-1..7 1:00:00, the first Sunday of the month at 01:00, plus up to 24 hours of random delay. The service runs /usr/share/mdadm/mdcheck --duration "6 hours"; when the six hours are up it stops the scrub with idle, writes where it got to into /var/lib/mdcheck/MD_UUID_..., and mdcheck_continue picks up from there the next day. Ubuntu made this switch in 2022 (kinetic, 4.2-3ubuntu1), Debian in 2023; Ubuntu's changelog note explains that "previous daily / monthly checks are performed, but are randomized, such that performance is not as impacted across a cluster of machines." The old checkarray script is still in the package, but nothing calls it any more.

That "six hours, continue tomorrow" design will come back later in the post.

RAID1: eight bytes, four hundred reads

Once the array had synced I wrote a recognisable 4 KiB block at the 100 MiB offset; it contains lines like MDLAB-BLOCK-0000-ORIGINAL-DATA-. Then I bypassed md, went straight to the second leg, /dev/loop7, and changed bytes 20–28 of that block to CORRUPT!. Data starts at a 1 MiB offset in this array (Data Offset : 2048 sectors), so the address on the leg is 101 MiB. The two copies now differ by eight bytes and md has no idea; RAID1 has no checksum and no way to test one copy against the other.

First experiment: ten dd iflag=direct reads through md. All ten came back clean. That reassured me for a moment, until I opened read_balance and saw why. In 6.8, for non-rotational disks the read balancer picks the leg with the fewest pending requests; on a tie the first one found wins, which is slot 0. On an idle array every read goes to slot 0. The corrupted leg sat in slot 1 and never got a turn.

Second experiment: eight parallel loops, fifty reads each. Of four hundred reads, 367 returned ORIGINAL and 33 returned CORRUPT!. Once slot 0's queue fills up under load, the balancer turns to slot 1 and the bad copy goes to the reader without any warning. The filesystem doesn't notice; ext4 has no data checksums. The application doesn't notice; if those bytes sit in the middle of a customer's name, the customer's name changed. Eight percent, for eight bytes. Make the disk bigger, run it longer, the ratio doesn't change; only who gets hit does.

Then check:

echo check > /sys/block/md127/md/sync_action
# 1.56 seconds later
cat /sys/block/md127/md/mismatch_cnt      # 128
cat /sys/block/md127/md/last_sync_action  # check
journalctl -k | grep md127
# md: data-check of RAID array md127
# md: md127: data-check done.

One hundred and twenty-eight, for an eight-byte difference. The manual is honest about this: md works in 64 KiB units, and when it finds a difference it does not investigate how many sectors actually differ; it adds the whole unit (128 sectors) to the counter. In raid1.c's process_checks that is a single line: atomic64_add(r1_bio->sectors, &mddev->resync_mismatches). And look at the kernel log: "data-check done." No location. For RAID1, md writes the address of the difference nowhere. A counter, no coordinates; on RAID5 we will see the opposite.

repair: whatever slot zero believes

I wanted to see what the source says in the lab, because "the first slot wins" could have been wrong in two different ways. Maybe it sorts by /dev/ name; maybe there is a rule that favours the leg written most recently. Neither.

I set up two blocks. At 100 MiB the bad copy is in slot 1 (loop7); at a new block at 150 MiB the bad copy is in slot 0 (loop6). Then echo repair > sync_action. Before and after:

                    loop6 (slot 0)   loop7 (slot 1)
100 MiB  before     ORIGINAL         CORRUPT!
150 MiB  before     CORRUPT!         ORIGINAL
100 MiB  after      ORIGINAL         ORIGINAL      <- fixed
150 MiB  after      CORRUPT!         CORRUPT!      <- corruption spread
mismatch_cnt after  256

process_checks picks its primary variable like this: start from slot 0, take the first leg whose read returned without error. Everything after that is bio_copy_data(sbio, pbio); the primary's pages are copied onto the other leg. In a three- or four-way mirror it is the same loop: find the primary, copy it over every remaining leg; two healthy legs get no vote against one bad one. On RAID10 the rule works as "the first copy in that stripe"; which disk holds the first copy changes from stripe to stripe with the layout, but it is still order, not a vote. At the 150 MiB block slot 0 was bad and slot 1 was good; repair replaced the good copy with the bad one, and now both copies are consistently wrong. The next check returns zero.

I am not writing this up as a bug; md has no information that would let it do better. Two copies, no checksum, no third party to say which one is right. That is what stands behind the manual's "all but one block." The right trade-off, in my view, is to never run repair without having looked at a check result first; if the counter is non-zero, go to the disks' SMART/NVMe logs and read errors, work out yourself which leg is suspect, and if necessary rebuild that leg with --fail + --remove + --add. A rebuild also copies the remaining leg, but this time you chose the remaining leg.

One more detail: mismatch_cnt is not cleared when a check finishes; it stays until the next check/repair/resync starts. But a rebuild (recovery) does not clear it. In the lab, a counter showing 16384 still showed 16384 after I pulled a leg, re-added it and let it fully rebuild; the check I ran right afterwards said zero. When you read the counter, read last_sync_action too; if it says "recovery", the number you are looking at is from a previous life.

RAID5 and RAID6: parity is a witness, not a judge

I repeated the experiment on a three-disk RAID5. Instead of computing where the block lands on which leg, I searched the files for the signature with grep -ob; it turned up at the 52 MiB offset in two files, one being data and one parity (the other data chunk in the stripe is zero, so parity equals the data). I corrupted eight bytes on the data leg.

This time all ten reads returned CORRUPT!. RAID5 does not look at parity when reading; parity is a spare calculation that only comes into play when a disk is missing. On a healthy array, whatever is in the data chunk goes to the reader. Then check:

mismatch_cnt = 8
kernel: md126: mismatch sector in range 102400-102408

Two differences. The number is 8 rather than 128, because RAID5 works in 4 KiB stripe slices (RAID5_STRIPE_SECTORS). And this time the kernel writes an address: sector 102400. That is a component-disk coordinate, not an array one; sh->sector is the stripe-row address that is the same on every disk. My block at 100 MiB (4 KiB block 25600, array sector 204800) lands on row 100 of a two-data-disk array, i.e. at 50 MiB on each component, sector 102400; grep found the signature at 52 MiB because of the 2 MiB Data Offset. The coordinate RAID1 withholds, RAID5 gives you; that is the pr_warn_ratelimited line inside handle_parity_checks5. The line is rate-limited: with a thousand corrupted stripes you will see a handful of lines in the log, not a thousand.

What did repair do? Reading the code there is exactly one target: sh->ops.target = sh->pd_idx. The parity disk. Data is assumed correct, parity is recomputed. The lab shows exactly that: after the repair the XOR of the three legs is zero, a second check reports zero mismatches, and the block read through md is still CORRUPT!. The manual says "on a truly clean RAID5 or RAID6 array, any mismatches should indicate a hardware problem at some level," and that is true; but repair erases the evidence of that problem.

I expected RAID6 to be where it got interesting. Two parities (P and Q) are mathematically enough to identify a single bad disk; that is a well-known property of RAID6's Reed–Solomon side. I built a four-disk RAID6, corrupted the data leg, check again said 8 and logged the same "mismatch sector in range" line. On repair, what handle_parity_checks6 does is this: if zero_sum_result has the P flag, recompute P; if it has the Q flag, recompute Q. There is no branch that suspects a data disk. The outcome is the same as RAID5: both parities rewritten to match the corrupted data, second check zero.

The information is there, but the kernel doesn't use it. The tool that does lives in mdadm's own repository: raid6check. It is in neither Ubuntu noble's nor Debian trixie's mdadm package; the file lists show no binaries beyond /usr/sbin/mdadm and /usr/sbin/mdmon. I built it with make raid6check from the mdadm 4.6 source in an ubuntu:24.04 container and copied it to the server. Its output on the corrupted stripe:

Error detected at stripe 100, page 0: possible failed disk slot 0: 1 --> /dev/loop12
Auto-repairing slot 1 (/dev/loop12)

It found the right disk and, with autorepair, recomputed the data from P and Q; the block was ORIGINAL again and the kernel's check came back zero. Had I run the kernel's repair on the same array first, that chance would have been gone, because P and Q would have been brought into agreement with the corrupted data and there would have been no contradiction left for raid6check to look at. Order matters: on RAID6, when the check counter is above zero, raid6check comes before repair. Two limits: according to the manual, check mode works on an online array, but since autorepair writes to the components bypassing md, I would only run it on an array with no write traffic; and on a degraded array it does not run at all, the manual says it refuses.

The counter's memory: six hours

Up to here the counter was at least a reliable total. Now back to the six-hour budget from the first section.

The mdcheck script writes echo idle > sync_action when the time runs out and saves the sector it reached; the next day, with --continue, it does echo $start > sync_min; echo check > sync_action. So what does each new check do to the counter? In md.c, when md_do_sync starts with the MD_RECOVERY_SYNC bit set: atomic64_set(&mddev->resync_mismatches, 0). It is treated as a scrub from scratch, not a continuation.

In the lab: one corrupted block at 20 MiB and one at 200 MiB, sync_max at a sector near the middle of the array (262144 of 522240).

segment 1: check, sync_max=262144   -> sync_completed 262144/522240, mismatch_cnt 128
           echo idle                -> sync_min set itself to 262144
segment 2: sync_max=max, check      -> mismatch_cnt 128

Two corrupted blocks, counter 128. The first half's result is gone. On any array that does not finish in six hours, the mismatch_cnt you read at the end of the monthly scrub describes only the last day's piece. The duration depends on component size: a mirror built from 10 TB disks needs fourteen hours at 200 MB/s, and when the scrub is throttled down to speed_limit_min (default 1000 KB/s) under load, the six-hour budget covers far less ground and the number of pieces grows.

There is a way to add up the pieces, but it lives outside the script. mdadm --monitor (on Ubuntu, mdmonitor.service, running /sbin/mdadm --monitor --scan) looks at /proc/mdstat every 60 seconds and emits a RebuildFinished event when a scrub finishes or is interrupted; if mismatch_cnt is above zero at that moment, it appends it to the message. When I interrupted a throttled scrub halfway, this landed in the journal:

mdadm: RebuildFinished event detected on md device /dev/md127, component device  mismatches found: 16384 (on raid level 1)

So the per-segment numbers are in the journal; adding them up is on you. Two traps. First, RebuildFinished is not one of the events that send mail; mdadm.8 spells it out, only Fail, FailSpare, DegradedArray, SparesMissing and TestMessage produce email. Having written MAILADDR root does not bring you the mismatch count; you need a PROGRAM line or something that reads the journal. Second, the 60-second loop: the monitor reads /proc/mdstat when a udev event or the 60-second timeout arrives, and it only emits RebuildFinished if it previously saw a progress percentage (the st->percent >= 0 condition). That is why it never saw my 1.5-second lab check, neither its start nor its end. On real arrays a scrub takes hours so this is not a problem, but if you ever wonder "why didn't the monitor log it", that can be the reason. A hint for anyone who wants to catch it: RebuildFinished goes to syslog at Warning priority and the kernel's mismatch sector in range line is a pr_warn; a priority ≤4 filter drops both into the same bucket.

And then there is the script itself. The mdcheck in Ubuntu 24.04 never reads the mismatch_cnt file; grep mismatch returns nothing. Upstream rewrote the script in November 2025 (--start and --restart modes, Checked_<UUID> marker files; the 4.5 in Ubuntu 26.04 carries that version, where mdcheck_start now only clears the markers with --restart and the daily job runs --start instead of --continue). The new script doesn't contain the word mismatch either. On the Prometheus side, node_exporter's mdadm collector parses /proc/mdstat (node_md_state, node_md_disks, node_md_blocks_synced); since v1.9.0 it also reads node_md_raid_disks and node_md_degraded from sysfs, but it never touches the mismatch_cnt file. The kernel counts, the distribution schedules, the monitor logs; there is no link in the chain that hands the number out as an alert.

The counter is above zero; is it real?

There are also cases where the counter lies; the manual lists two, and I tried both.

The first is --assume-clean. If you skip the initial sync when creating the array, md assumes the two legs are identical; if they are not, the first check counts the difference. I wrote 8 MiB of random data to the first leg and recreated the array with --assume-clean: mismatch_cnt 16384, exactly 8 MiB. A number in never-written space that means nothing to the data system. This is why a large number on the first scrub after mdadm --create is normal; what is scary is that the number stays there on the following scrubs too.

The second is swap. The scenario the manual describes: while a page is being written to swap, the memory manager marks it "clean", and the page may change while the write is in flight; on RAID1 the same page goes to two disks separately, so the two copies can come out different. Harmless for the data, because that page is already considered discarded; but check counts it as a mismatch. I tried to produce it: mkswap on the array, swapon -p 5, a Python process in a cgroup with a 64 MiB memory ceiling randomly modifying a 180 MiB buffer for forty seconds. Swap showed 3.5 MiB in use, the process paged in and out continuously; the check afterwards said zero. The race window exists, but in forty seconds, on loop devices sitting behind the page cache, I could not hit it. I am writing this not as "it doesn't happen on swap" but as "remember the manual's warning when you read a non-zero counter on a swap array."

Those two cases are the reason behind the manual's sentence that mismatch_cnt "can not be interpreted very reliably on RAID1 or RAID10." RAID5/6 have no such escape; there, any number above zero says something went wrong somewhere, but not what.

What I would do

On my own arrays, from now on, this is the order. The monthly scrub should be a check, never a repair; Ubuntu's and Debian's script already does that. When the scrub finishes, or each time a segment is interrupted, something should collect the RebuildFinished lines in the journal; a ten-line script called through PROGRAM, or an alert rule watching the journal, whichever you have. On RAID5/6, put the mismatch sector in range lines in the same place; they carry coordinates.

If the counter is above zero, the first reflex should not be repair. On RAID1/10, first look at the disks' own error counters; if the suspect leg is clear, fail it and rebuild it, because a rebuild hands the "which copy do I believe" decision to you, while repair hands it to the slot number. On RAID6, build raid6check and run it before repair; reverse the order and you have erased the contradiction the detection needs. On RAID5 there is no referee between parity and data; repair only silences the counter.

And when you read mismatch_cnt, read two more files next to it: last_sync_action (if it says "recovery", the number is stale) and which range sync_completed covered (in a segmented scrub, only the last segment). The counter is not a diagnosis; it is a light saying somebody needs to look, and sometimes it is on in the wrong room.

Conclusion

I now read that sentence in md(4) differently: not "all but one block" but "all but slot zero." There is a design decision behind it and it is reasonable: md has no information to referee between copies, so it doesn't referee; it applies a deterministic rule. The problem is not the rule; it is that the rule's name is absent from the manual, and that the word repair reads like "fixes." It doesn't fix; it makes consistent. Consistently wrong is quieter than inconsistently right.

Versions: VPS3 Ubuntu 24.04, kernel 6.8.0-139-generic, mdadm v4.3 (Ubuntu 4.3-1ubuntu2.1), util-linux 2.39.3. Comparison: vps5 Ubuntu 26.04, kernel 7.0.0-31, mdadm 4.5-5ubuntu1. raid6check built from the mdadm-4.6 release (GitHub release 24 March 2026) in an ubuntu:24.04 container. Source reading against v6.8 drivers/md/raid1.c, raid5.c, md.c. The lab arrays, loop devices, swap entry and /var/lib/mdcheck were removed when done; mdmonitor.service and the mdcheck timers were returned to their pre-array state (inactive).

DEV Community