DEV Community

Cover image for Benchmarking 10GbE NAS-to-NAS Transfers: iperf, NFS, cp, and rsync
Haruki Niwa
Haruki Niwa

Posted on Edited on AI-assisted

Benchmarking 10GbE NAS-to-NAS Transfers: iperf, NFS, cp, and rsync

One of our Synology NAS systems was nearly full, so I planned to move data to another NAS over our existing 10GbE network. Before migrating anything, I wanted to establish realistic file-copy performance without modifying the NAS configuration.

iperf2 reached 9.42 Gbps, but the copy commands told a different story: cp completed a large-file copy at 479–494 MB/s, while a single rsync process reached about 181 MB/s. I compared the tools already installed on the NAS systems and examined their CPU and storage activity to understand the difference.

This is a performance investigation, not a report of a completed production migration. In particular, a faster first copy is not necessarily a better migration workflow.

Test setup

The tests were conducted on November 9–10, 2026. Hostnames, addresses, and research-data paths are anonymized.

Component Source NAS Destination NAS
CPU Intel Atom C3538, 4 cores, 2.10 GHz Intel Atom C3538, 4 cores, 2.10 GHz
Linux kernel Synology 4.4.302+ Synology 4.4.302+
RAM Approximately 20 GiB Approximately 4 GiB
Storage Btrfs on six-disk RAID5 Btrfs on six-disk RAID5
Network 10GbE 10GbE
Volume usage Approximately 98% Approximately 40–41%
Role NFSv3 server Read-only NFS client; local file writes

The source NAS exports data over NFSv3 and sends it directly over 10GbE to the destination NAS, which writes to its local Btrfs RAID5 volume.

The destination mounted the source share over NFSv3/TCP, with an effective rsize of 128 KiB. An administrative PC was used to run commands, but it was not in the file-data path.

I tested an existing 16,285,199,445-byte file (15.17 GiB). Some tests copied the entire file; others copied its first 8 GiB. Throughput is reported in decimal MB/s. Copied data was checked against the corresponding source bytes using SHA-256. These are short, repeated large-file tests; cache state was not controlled.

1. The network baseline

I measured TCP throughput independently of disk and filesystem operations using iperf2:

# Destination NAS
iperf -s -B <DEST_IP> -p 5201 -i 1

# Source NAS
iperf -c <DEST_IP> -B <SOURCE_IP> -p 5201 -t 10 -P 4
Enter fullscreen mode Exit fullscreen mode
TCP streams Throughput
One 9.10 Gbps
Four 9.42 Gbps

The network could carry traffic close to 10GbE line rate. But iperf does not include NFS processing or destination filesystem writes, so it cannot predict file-copy throughput.

2. NFS reads and full-file copies

Reading the first 8 GiB of the source file to /dev/null took 13.43 seconds (640 MB/s):

# Run on the destination NAS; SRC is on the NFS mount
dd if="$SRC" of=/dev/null bs=4M count=2048
Enter fullscreen mode Exit fullscreen mode

Copying the same range to the destination's local volume using dd with conv=fsync consistently reached 394–397 MB/s. The read-only result was probably affected by caching: during the observed intervals of a later transfer, the source RAID showed almost no physical reads. It would therefore be incorrect to interpret 640 MB/s as sustained source-HDD throughput.

I then compared standard copy commands:

Command Copied data Elapsed time Throughput
dd, bs=1M, conv=fsync 8 GiB 19.92 s 431 MB/s
dd, bs=4M, conv=fsync 8 GiB 21.98 s 391 MB/s
dd, bs=16M, conv=fsync 8 GiB 22.47 s 382 MB/s
cp 15.17 GiB 33 s 494 MB/s
rsync -W --stats 15.17 GiB 90 s 181 MB/s

A bar chart comparing observed single-process throughput: cp at 493.5 MB/s and rsync -W at 180.9 MB/s for a complete 16.29 GB file.

A separate cp test reached 479 MB/s. All copies matched the relevant source data by SHA-256.

These are observed command-completion rates, not identical durability-controlled workloads. The dd runs used 8 GiB and an end-of-run fsync; cp and rsync copied the whole file without an equivalent explicit final sync. Cache state and test order may also have affected results. The dd numbers do not establish an optimal block size.

The destination ran rsync 3.1.2. Its -W option disables the delta-transfer algorithm, but whole-file mode is already normally the default when both paths are local paths—even when one points to an NFS mount. Rsync's transfer checksums and sender/receiver processing introduce work that cp does not perform in the same way. The measurements alone do not identify which operation accounts for the slowdown.

3. What changed with parallelism?

For dd, I split the same 8 GiB range across independent output files, keeping the total amount of copied data constant:

Concurrent dd processes Elapsed time Aggregate throughput
1 21.66 s 397 MB/s
2 18.07 s 475 MB/s
4 18.34 s 468 MB/s

Two processes improved aggregate throughput by about 20%, while four provided no additional benefit. The concatenated output parts matched the original 8 GiB SHA-256.

In a separate rsync experiment, each process copied the entire file to its own destination. Consequently, the two-process test transferred twice as much data:

Concurrent rsync jobs Total output Elapsed time Aggregate throughput
1 16.29 GB 90 s 181 MB/s
2 32.57 GB 116 s 281 MB/s

Aggregate throughput rose by about 55%, and both outputs passed SHA-256 verification. However, each individual copy took longer. This supports concurrency as a potential option for multiple independent files, not as a method that accelerates one file by itself.

4. CPU and RAID activity

I collected one-second iostat -cdx 1 samples on the destination and focused on its six-disk RAID5 device (md2). Approximate averages over the corresponding measurement intervals were:

Destination metric cp 1 × rsync 2 × rsync
CPU user time 0.5% 27% 43%
CPU system time 67% 26% 38%
CPU idle time 23% 45% 17%
md2 write throughput 488 MB/s 182 MB/s 284 MB/s

CPU percentages refer to the four-core system as a whole; the two-rsync experiment was conducted the following day.

With cp, the destination generated substantially more RAID write traffic and spent more time in kernel code. With one rsync copy, user CPU time was much higher—approximately one core's worth—while the RAID wrote more slowly. Running two rsync jobs increased both CPU usage and aggregate throughput.

This pattern is consistent with user-space processing contributing to the slower single-process rsync result. It does not prove that checksumming is the sole cause, nor that RAID5 or the CPU has a fixed performance ceiling. The measurements were not a function-level CPU profile.

5. Why fio produced much lower figures

I also tested destination-local writes, without NFS, using fio 3.33. Both jobs used 8 GiB, 1 MiB blocks, direct I/O, and an end-of-job fsync:

I/O engine Queue depth Throughput
sync 1 72.9 MB/s
libaio 16 288 MB/s

The depth-1 job completed one I/O at a time; the asynchronous job maintained depth 16 for 99.8% of operations. Their different results illustrate how queue depth affects this workload. Neither value is a universal write-speed limit, nor directly comparable to buffered cp and rsync measurements.

6. Faster copying versus a reliable migration

To explore a possible workflow, I copied the full file with cp, preserved its modification time, then checked whether rsync would retransmit it:

# SRC and DST refer to the source file and a newly created copy
cp "$SRC" "$DST"
touch -r "$SRC" "$DST"
rsync -rtW --dry-run --itemize-changes --stats "$SRC" "$DST"
sha256sum "$SRC" "$DST"
Enter fullscreen mode Exit fullscreen mode

This initial copy reached 479 MB/s. The dry run found zero files to transfer, and both SHA-256 hashes matched. This confirms only the behavior of one unchanged regular file. Rsync's standard quick check compares size and modification time; it does not independently hash skipped files.

For an actual directory migration, rsync remains attractive because it supports repeatable incremental synchronization and can be configured for interrupted transfers and file metadata. cp alone offers a simpler first-copy path, but the cp-then-rsync combination has not been validated for complete directory trees, including Synology ACLs, owners, symlinks, changing files, and safe source cleanup.

Conclusion

A healthy 9.42 Gbps TCP link did not translate into near-line-rate file copying. On this pair of NAS systems, cp completed full-file copies at 479–494 MB/s, while one rsync process reached 181 MB/s. Parallel rsync improved aggregate throughput, and CPU monitoring suggested additional processing costs were relevant.

The main lesson is not that cp is a better migration tool than rsync. Raw first-copy throughput and safe, repeatable synchronization are different requirements. Rather than change working NAS configurations to chase a benchmark, I stopped at these results and kept the next step separate: verifying a resumable directory-migration procedure and its permissions, consistency, and final cutover before deleting anything from the source.

References


AI disclosure: This article documents my own technical setup and results. An LLM was used to help draft, organize, and edit the English text. I reviewed the technical descriptions and measurements against my original logs.

Top comments (0)