DEV Community: Yoshiki Fujiwara(藤原善基)@AWS Community Builder

Why Amazon FSx for NetApp ONTAP block performance differed by about 2x on the same specified values, isolated by measurement

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 30 Sep 2026 05:43:42 +0000

Why Amazon FSx for NetApp ONTAP block performance differed by about 2x on the same specified values, isolated by measurement

While measuring, I hit a case where the templates and the specified values matched, yet the number
moved every time I measured. Rather than let it go, I wanted to isolate whether it was a
measurement mistake or something about the environment, so I eliminated the causes one at a time.

The same thing shows up when a post-migration performance test does not agree, and when you want to
settle on one representative value while sizing an initial build. It is a question that eats a lot of
time, so this article is the record of that isolation.

What this article does and does not cover: it covers investigating the cause of the performance
swing, and how to isolate the bottleneck using four CloudWatch metrics. It does not cover mount
procedures, how to confirm multipathing, or figures usable for sizing (details under "What this
article does not cover" below). All figures are from a Single-AZ configuration; Multi-AZ is
unverified.

dev.to series: FSx for ONTAP Block Protocols, Measured.

A 1 MiB sequential read on Amazon FSx for NetApp ONTAP (hereafter FSx for ONTAP), measured with the
same template, the same specified values and the same workload, varied by 2.64x across
environments. Each one was stable within its own run. This is not measurement scatter.

This is the record of eliminating the causes one at a time. It was not the path, not how the
connection was made, not the queue count, not the distribution, and not the iopolicy (Linux's
multipath setting for distributing I/O across paths to an NVMe device). What was
acting on it was the layout on disk, which breaks after 300 seconds of 4 KiB random writes and
comes back when the data is rewritten sequentially.

Three times along the way I nearly settled on the wrong explanation. First: "if it exceeds the
specified value it must be served from memory." Second: "the layout must also be setting the
amplification factor for 4 KiB random reads." Third: "the amplification factor must be decided by
whether 4 KiB random writes ever ran on that device." All three were disproved once I aimed a
measurement that could decide them. All three are in here. The third had five observations
lined up cleanly behind it and survived until a control experiment killed it.

What was measured

Assuming no prior knowledge of FSx for ONTAP, here's what varies between the deployments.

Deployment 1: AWS Cloud, single VPC, single AZ. One EC2 client -> (NVMe/TCP, 1 MiB sequential read) -> Amazon FSx for NetApp ONTAP (same template, same specified values)
Deployment 2: same AWS Cloud, single VPC, single AZ. One EC2 client -> (same connection, same workload) -> Amazon FSx for NetApp ONTAP (same template, same specified values)
The invisible difference between Deployment 1 and 2: layout on disk. It never appears in the template or the list of specified values

(Deployment 1, Deployment 2), each measured from one EC2 client with the same connection and the
same workload. Shows what can be held identical (path, connection setup, queue count,
distribution, iopolicy) against what cannot (layout on disk)" width="800" height="726">

Figure: **matching the CloudFormation template and the specified values doesn't make the layout
on disk match across a redeploy.* Path, how the connection is made, queue count, distribution,
and iopolicy can all be held to the same value across deployments. Layout is the one condition
that changes every deployment and never appears in the list of specified values.*

Item	Role in this article
Deployment 1 / Deployment 2	Separate FSx for ONTAP file systems, built from the same template and the same specified values
Conditions that can be matched	Path, how the connection is made, queue count, distribution, iopolicy
Condition that can't be matched	Layout on disk. Breaks after 300 seconds of 4 KiB random writes, comes back when rewritten sequentially

What this article does not cover. Mount procedure is out of scope
(repository record, Japanese).
Confirming whether multipathing is usable is also out of scope (separate article).
The figures in this article cannot be used for sizing either. Why they cannot is the subject of
the article. All figures are from a Single-AZ configuration; Multi-AZ is unverified.

One of three, but there is no reading order. The block-protocol measurements are split by
reader.

Article Reader

A Sessions and queues People about to mount

B Confirming whether multipathing is usable People who followed the procedure and got stuck

C What moves the numbers (this article) People who measure the numbers, and people who cite them

None of the three presupposes the other two. Read only the part you need.

The same property on the file-protocol side (NFS / SMB / S3 API) is covered in
S3 Burst Part 2, with the primary record of those figures in
throughput, IOPS and concurrency (Japanese).
Those figures are also iorate=max values from this environment, so the way of measuring
described here applies to them unchanged.

	Article	Reader
A	Sessions and queues	People about to mount
B	Confirming whether multipathing is usable	People who followed the procedure and got stuck
C	What moves the numbers (this article)	People who measure the numbers, and people who cite them

The environment measured

Item	Value
File system	FSx for ONTAP second generation SINGLE_AZ_2, 6,144 MBps x 1 HA pair, SSD 4,096 GiB, provisioned 200,000 IOPS
Region	ap-northeast-1, single AZ
ONTAP	Not one version. It spans 9.18.1P5 (the early deployments), 9.18.1P6 (the per-path measurements) and 9.18.1 (the last three deployments). This is exactly the unresolved candidate left at the end of this article (the cross-article version mapping is in the ONTAP version matrix)
NVMe read cache	Disabled (confirmed on both nodes)
LUN / namespace	600 GiB, on a 900 GiB volume
Before measuring	All 600 GiB was written once. Unwritten blocks on a thin-provisioned volume return zeros, so reading without writing first measures nothing
Client	c5n.9xlarge; 50 Gbps network is the guaranteed figure
Instrument	VDBENCH 5.04.07 directly, `iorate=max` (unlimited), 512 threads, 60 s warm-up + 300 s measurement
Test data / efficiency settings	Not recorded whether the VDBENCH payload was compressible, or whether volume inline efficiency (`StorageEfficiencyEnabled`) was enabled. This article's subject is on-disk layout, so whether efficiency settings affected results is an unconfirmed variable

The window is 300 seconds, so the figures in this article include burst. They cannot be cited as a
baseline.

Every figure in this article is an iorate=max saturation point, not an operating point.
No target IOPS was given; it is what the path would take. This distinction bears on the subject
of the article itself. A saturation point answers "where does it jam", and does not answer "at
what response time can it be used".

The file-protocol side of the same repository measured that gap. With a target of 4,400 IOPS the
result was 4,406 MB/s at 8.86 ms; unlimited, it was 4,204 MB/s at 121.79 ms.
Unlimited is 5% lower in throughput and 14x worse in response time.
Recording one iorate=max point as "the ceiling" leaves behind a figure that is lower than what
is reachable and an order of magnitude worse in response time.

Response time was recorded as measured in only the two series under "not concurrency" below.
Elsewhere the values VDBENCH printed were not kept, and the environments are torn down. But with
512 threads fixed and no rate limit, response time is the identity 512 / IOPS (derived 216.6 ms
against measured 216.588 ms, confirmed to agree). The figures below can be read through that
identity: 1,288.87 MB/s is about 397 ms, 3,407.60 MB/s about 150 ms, 546.02 MB/s about
937 ms. All three are derived, not measured.

The response-time-versus-load curve was not measured. The file-protocol side produced a curve
across target IOPS values and a cutoff (the IOPS at which a given latency is reached) with
auto_vdbench, but the same thing could not be
produced on the block side. auto_vdbench creates test files on a mount path and cannot drive a
raw device or an NVMe namespace. Open work.

The same configuration varied widely

This is the part of the article I would most like read carefully.

As F-3, the same shape was measured three times within one connection.

Run	1 MiB sequential read
1st	3,401.49 MB/s
2nd	3,407.60 MB/s
3rd	3,407.59 MB/s

Within one connection it reproduces to 0.18%. The second and third differ by 0.01 MB/s.

But three environments built with the same procedure, the same template and the same negotiated
result (4 queues) produced different values.

Environment	1 MiB sequential read	Stability within the window
First	1,288.87 MB/s	stable
Second	2,366.49 MB/s	stable
Third	3,401-3,408 MB/s	stable (0.18%)

The spread is 2.64x. Each was stable within its run, so this is not measurement scatter.

So you cannot size from a single figure of the form "NVMe/TCP throughput on this configuration is
N MB/s". At the point where I had measured the second environment, I was about to write 2,366.49
as the representative value. I had not followed my own rule of repeating three times, so I noticed
one environment later than I should have.

I had put forward one candidate at the time: Asymmetric Namespace Access (the mechanism that
tells the host which of several paths to the same namespace is the shortest one, referred to as
multipathing throughout this article) is unusable, so
which of the two controllers the device under measurement sits behind can change per deployment.
Measurement said otherwise.

The path candidate is gone

I brought in a kernel with multipathing enabled (Rocky Linux 9.7) and measured within one deployment, on
one namespace, with the paths narrowed to one at a time.

Path	1 MiB sequential read
optimized only (the node holding the namespace)	1,278.11 MB/s
non-optimized only	1,279.07 MB/s

0.08% apart. Either controller gives the same figure, so the path cannot explain that spread.

How the connection is made was eliminated too. Connecting from optimized first, from non-optimized
first, and using connect-all as the AWS procedure has it, gave 1,276.60 / 1,277.38 / 1,277.28
MB/s. A spread of 0.06%. The queue count was the same in every configuration
(queue_count=5, 10 established sockets to :4420).

What was acting on it was when the data being read was last written

Within one deployment, one device, one path and one policy, rewriting the 600 GiB brings it back.

State	1 MiB sequential read
Immediately before the rewrite (control)	1,239.15 MB/s
Immediately after the rewrite	2,298.54 MB/s
Once more, right after that	2,296.12 MB/s

1.85x. And the second read does not fall back. This is not a property that is consumed by one
read.

This is the only place in this article where a ratio is written. The numerator and denominator
come from the same file system, the same device, the same path and the same policy: the reason a
ratio can be written here is the same reason none is written in the other sections.

CloudWatch answers what the cold side is running into.

State	`DiskIopsUtilization`	Disk throughput utilization	Network utilization
Cold read	avg 57-60 / max 95.0-95.4	30 / 42	8.7 / 11.6
Warm read	avg 13.6-14.4	40 / 41	19.2 / 19.5

The ceiling on the cold side is SSD IOPS. It is not the 6,144 MBps of throughput capacity that
was specified (utilization 42% or below). It is not the network either (11% or below).
And at 2,298 MB/s on the warm side, none of the utilization figures FSx for ONTAP publishes is at
its ceiling. What is acting on it there is not known.
(What is being read here is DiskIopsUtilization / disk throughput utilization / network
utilization from the
second-generation file system metrics.
The utilization metrics emit one point per HA pair and per aggregate.)

Not reads, and not elapsed time

Since it moves with "when it was written", I aimed separate measurements at what brings it down.
On top of one fill, reads, elapsed time and writes were applied in turn.

Immediately after	1 MiB sequential read
The fill	2,263.8 MB/s
A second read straight after	2,267.5
A third read straight after	2,268.4
30 minutes idle	2,258.5
300 seconds of 4 KiB random writes	1,272.8
Refilling	2,235.4

Reads do not bring it down. Three full passes over 600 GiB, a spread of 0.2%
Time does not bring it down. 30 minutes with nothing running, within 0.4%
4 KiB random writes bring it down. 2,258.5 to 1,272.8, a fall of 1.77x

Do not generalize this to "writes bring it down". What was aimed was one point: 300 seconds of
4 KiB random writes. The result of separating shape from duration and aiming again is in
the section below, and the generalization
did not hold.

What falls is not cache but the layout on disk

I first explained this as "cache retention". That was wrong. Read CloudWatch interval by
interval and the mechanism comes out plainly.

State	Client	One read on the disk side	Disk IOPS	Against the specified 200,000	`DiskRead / DataRead`
Right after the fill (5 intervals)	2,219-2,263 MB/s	87.3-89.6 KiB	25,743-27,404	13%	97-105%
After 4 KiB random writes (6 intervals)	1,151-1,267 MB/s	11.7-11.8 KiB	177,527-189,629	89-95%	169-188%

DiskRead / DataRead is about 100%, so it is reading from disk right after the fill too.
It is not being served from memory. What changed is the size of one read on the disk side, by a
factor of 7.5. Disk IOPS rises 7.3x as a result, and saturates at 89-95% of the specified value.

So: 4 KiB random writes break the layout on disk, and the same 1 MiB read becomes many small
reads. Reads do not change the layout, so they have no effect, and neither does time. Rewriting
sequentially restores a contiguous layout. This is also why only 4 KiB random reads are
unaffected: a random 4 KiB is one read on the disk side whatever the layout is.

DiskIopsUtilization alone does not get you here. It tells you whether something is
saturated; it does not tell you why it saturated (because each read is small).

And it goes back and forth within a single run. I think this is the strongest evidence.

What was running	One write	One disk read
Fill, then 3 reads	1,024 KiB	89.1 / 89.6 / 88.3 / 87.4 KiB
The read after 30 minutes idle	—	87.9 KiB
4 KiB random writes, then read	4.0 KiB	58.7 to 11.8 to 11.6 KiB
Sequential refill, then read	1,024 KiB	88.2 / 87.7 KiB

30 minutes idle does not move it, a 4 KiB write brings it down, a sequential write brings it back.
This is neither a difference between deployments nor measurement scatter.

Note: this d_io depends on the workload too. In 1 MiB sequential read intervals it is 88 or
11.7 KiB; in 4 KiB random read intervals it is 4-5 KiB. When comparing, match the client-side
size of one operation. Without that, the layout difference and the workload difference mix.

As a benchmarking practice, I think this is the most portable thing here.
If you publish a sequential read figure, record whether small random writes were interposed
between writing and reading. Without them, the value does not move however many times you read or
however long you wait. With them, it changes by nearly 1.8x. As the next section shows, 300 seconds
of sequential writes does not move it.

What breaks it is not writes but small random writes

This section was measured later. The wording above reads as "writes break it", and that was
too broad. On the same device, I separated the shape of the write from its duration.

Step	1 MiB sequential read	Against the reference
Reference (not broken)	2,364.78 MB/s	—
4 KiB random writes for 30 seconds	2,343.63 MB/s	-0.9% (not broken)
4 KiB random writes for 300 seconds	546.02 MB/s	-77% (broken)
After a sequential refill	2,364.18 MB/s	-0.03% (fully restored)
Sequential writes for 300 seconds	2,365.59 MB/s	+0.03% (not broken)

Sequential writes do not break it. 300 seconds of writing, 0.03%. What breaks it is small random writes only, not "writes" in general
30 seconds does not break it, 300 seconds does. The threshold is somewhere in between, and where is not known
The unbroken side does not depend on the specified IOPS. 2,364.78 at 100,000, 2,366.23 at 200,000
For the broken side, proportionality cannot be written. The 546.02 MB/s here (specified 100,000) and 1,272.8 MB/s on another deployment (specified 200,000) give a ratio of 2.33 — it moves in the direction of the specified IOPS but not by 2.00, and the two are not from the same deployment. No run varied the specified IOPS while in the broken state, so this is not a control experiment

In practice, record whether you "read after small random writes", not whether you "read after
writes". A workload that writes sequentially and reads sequentially is not subject to this.

The conclusion that you cannot size from a single point is unchanged. What changed is the
reason. Not "an unexplained 2.64x", but "even on the same configuration it moves 1.8x
depending on whether small random writes were interposed after the data being read was last
written".
Why the 3,401-3,408 MB/s environment came out higher still remains unexplained.

The warm-side ceiling sits on the client side

In the warm state, none of the
published utilization metrics
was at its ceiling. I used two EC2 clients to decide which side it stops on.

What was measured	1 MiB sequential read
Rocky 9.7 (concurrent)	2,040.69 MB/s
RHEL 9.7 (concurrent)	2,163.86 MB/s
Total	4,204.55 MB/s
One EC2 client alone (same state)	2,265.7 MB/s

The total is 1.86x one client. The ceiling sits on the client side. The storage side still had
headroom; it was stopping at what one client could produce.

What determines the per-client ceiling was not measured. It is about 16-17 Gbps per client, short
of the 50 Gbps of a c5n.9xlarge.

Before you write "I measured the storage ceiling", try adding an EC2 client.
If the total grows, what you were measuring was the EC2 client's ceiling.

The same shape appeared on a path that is not block. On 2026-09-19, on the NFS path (first
generation 2,048 MBps, SSD IOPS 40,000, two c5n.9xlarge reading 300 GiB that does not overlap),
1,195.27 alone gave a total of 1,901.48 MB/s, 1.59x. Alone, disk throughput utilization was
63% and not saturated; with two clients it reaches 102-103%.

There, the single-client side was followed up the next day, and it turned out to be two settings
rather than a resource. Transfer size (rsize 64 KiB to 1 MiB, +21%) and concurrency (8 to 16
threads, +14%) took a single client to 1,646.17 MB/s, at which point the file system side pins at
102.6%
(measurement, Japanese).

It was followed further (2026-09-20), and "the client side" split into parts. On second
generation 6,144 MBps / 200,000 IOPS with 4 KiB random reads, varying only how the working set is
divided, five ways:

Division	Against the reference	Disk side / specified 200,000
1 EC2 client / 1 SVM / 1 volume (reference)	—	60.7%
Threads doubled only	+8.9%	67.6%
2 volumes (same SVM)	+1.0%	66.8%
2 SVMs	+26.3%	83.1%
2 EC2 clients	+52.7%	96.6%

What acts on it is neither thread count nor volume count, but the number of IP addresses the client
talks to and the number of clients. With two clients the disk side reaches 96.6% of the specified
value, and only there does the file system side become the limit
(measurement, Japanese).

The same separation was not done for the 16-17 Gbps per client in this section. There is no
block-side measurement varying transfer size, concurrency and the number of peer IP addresses.
Had I followed it that far instead of stopping at "the client side", the block side might also
have turned out to be settings and division.

"Measuring with one client does not determine the kind of ceiling" holds across protocols, so the
procedure in this section is portable.

At least it is not concurrency

The candidate "there must not be enough outstanding requests" fell. In the unbroken state, I
varied thread count alone by 4x.

Threads	1 MiB sequential read	Response time (measured)
512	2,363.91 MB/s	216.588 ms
1,024	2,364.28 MB/s	433.283 ms
2,048	2,365.58 MB/s	860.258 ms

Throughput moves by 0.07% and only response time scales precisely. Applying 4x the threads does
not raise the per-client ceiling; only the time requests spend waiting goes up 4x.

This table is also where the hazard of iorate=max is most visible in this article.
"2,364 MB/s" is the same on all three rows, but the response time behind it spans 216 ms to
860 ms. Write throughput alone and that difference disappears.

The file-protocol side was the same. NFS 4 KiB random reads at 512 / 1,024 / 2,048 threads gave
94,949.6 / 97,915.3 / 97,014.5 IOPS (a spread of 2.2%), with response times of
5.391 / 10.456 / 21.108 ms (measured). That ceiling is not concurrency either.
Note that 94,949.6 is from a deployment with half the SSD capacity (4,096 against 8,192 GiB), and is
0.7% from the original measurement of 95,630. Between those two points, capacity was not
acting on it.

Do not cite those two figures as a same-conditions reproduction. Re-measured on 2026-09-20 with
a 600 GiB working set, the reference was 81,313.9, 14% below the figures above.
The original measurement did not record its working set, so the conditions cannot be matched.

Match the time of day when comparing configurations

My first comparison measured four configurations in sequence, across time. And as the section
above shows, a sequential read figure moves 1.85x with elapsed time since the write. I thought I
was changing the policy; I was changing elapsed time as well.

The same shape is not confined to policies. I once wrote a 23% difference for multipathing on versus off,
and that too measured the two at different times. When comparing configuration A against
configuration B, match the elapsed time from write to read, and if you cannot match it, it is better
not to compare.

On choosing between protocols

There is no protocol difference between iSCSI and NVMe/TCP at one connection. That conclusion
is easy to miss if you compare figures measured on different file systems, so the comparison is kept
below to show why.

Comparison	iSCSI	NVMe/TCP
Different file systems side by side (misleading)	1,135.18 MB/s	591.64 MB/s
As the documented procedure has it	1,135.18 MB/s	1,288.87-3,407.60 MB/s (3 deployments)
4 KiB random reads under that procedure	44,593 IOPS	126,964 / 128,154 IOPS (2 deployments)

Read on its own, the top row looks like "the ordering inverts at one connection." But those two
figures were measured on different file systems, and side by side the difference reads as a protocol
difference that isn't there.

I re-measured on one file system. A 600 GiB LUN and a 600 GiB namespace were placed together on
a single 1,800 GiB volume, leaving lun= as the only line differing between the parameter files.

One connection, same file system	1 MiB sequential read
iSCSI, 1 session	1,135.19 MB/s
NVMe/TCP, 1 I/O queue	1,135.88 MB/s
Difference	0.06%

The ordering does not invert. There was no protocol difference at one connection.
591.64 belongs to the deployment tier: one-connection figures appear at only two positions,
4.73 Gbps and 9.08 Gbps, and within one deployment iSCSI and NVMe/TCP appear at the same
position. The connect-all side gave 2,366.32 against the published 2,366.49, agreeing to 0.007%
(measurement, Japanese).

The question left over is what 591.64 was. In the single mode as published, all four workloads
landed between 591.5 and 591.7, the shape of hitting the ceiling of a single path. The
re-measured run does not pin there. The cause has not been confirmed.

No ratio is written. The as the documented procedure has it row still has its numerator and
denominator from different file systems, and the NVMe/TCP side scatters across deployments.
Computed from one pair it comes to "2.08x", but recomputed over the measured spread it is
1.14-3.00x. Pick one figure out of that spread and write "N times", and the picking is itself the
conclusion.

The iSCSI side cannot be treated as a fixed value either. For sequential reads, one session and
both portals agreed across two deployments (1,135.18 and 1,135.19, 0.01 MB/s apart), but two
points are not a distribution. And 16 sessions is still one deployment only. There is no
basis for "iSCSI is stable". The evidence is simply not there.

One thing can be kept as a comparison. 4 KiB random reads agreed to within 1% on the same two
deployments where sequential reads scattered (126,964 and 128,154). It is the sequential read
that scatters, and nothing else.

This was later corroborated. On the same deployment where a rewrite moved sequential reads by
1.85x, 4 KiB random reads gave 131,506 (warm) against 131,550 / 131,749 / 131,736 (cold), a spread
of 0.2%. Whatever moves sequential reads by 1.85x does not act on 4 KiB random reads.

4 KiB random reads do, however, scatter across deployments for a different reason (three tiers
observed: 131k / 203k / 242k). That cause could not be confirmed. Two things were established.

One is that 4 KiB random reads scale precisely with the specified SSD IOPS.
Only the specified value was changed, within one deployment.

Provisioned SSD IOPS	4 KiB random read	Ratio to the specified value
200,000 (before lowering)	131,413.5 IOPS	0.657
100,000 (after lowering)	66,016.9 IOPS	0.660
100,000 (created that way on another deployment)	65,070-65,947 IOPS	0.651-0.659
200,000 (after raising it on that deployment)	131,529.7 IOPS	0.658

The ratio going down is 1.990 and going up is 2.02. The measured value lands at about 66% of the
specified value, the same ratio at all four points. The 1 MiB sequential read in the same
states was 2,265.7 / 2,366.23 MB/s and does not change: sequential reads do not depend on the
specified IOPS.

The reverse control was taken differently, in the following run. The first attempt lowered from
200,000, and the request to restore it was refused with "cannot start until at least 6 hours have
passed since the last storage capacity update".
A provisioned IOPS change is treated as a storage capacity update, with a six-hour cooldown.
Billing continues while you wait, so the next run was created at 100,000 and raised to 200,000
partway through: in the increasing direction it takes one update and does not hit the cooldown.
An update stays in UPDATED_OPTIMIZING for about 18 minutes (about 17 minutes when lowering).
If you measure by lowering, plan to throw the environment away; if you need a round trip, design
it in the increasing direction.

The other is that what sets the spread between deployments is the disk read amplification factor.

The specified value was 200,000 on every deployment. What differed was how many disk-side reads one
client read becomes. CloudWatch retains 5-minute values for 63 days, so the intervals whose
average read I/O is 4 KiB can be pulled out and checked without rebuilding the environment.

Deployment	Client IOPS	Disk IOPS	Against the specified 200,000	Amplification (byte ratio)
High side	203,961	162,562	81%	96.5%
High side (another deployment)	206,582	172,158	86%	83.3%
Low side	131,617	195,905	98%	193.7%
Low side (another deployment)	130,080	195,941	98%	190.2%

The IOPS the client sees is the specified IOPS divided by that environment's amplification factor.
131,617 x 1.49 = 196,109; 203,961 x 0.797 = 162,557. Both check out.

At an amplification of about 1.5, the disk saturates at 98% of the specified value and the client stops at 131k
At an amplification of about 0.8, the disk has headroom at 81% and the client reaches 204k

The "about 66% of the specified value" from the proportionality above is the reciprocal of that
1.5. The same relation was being seen two different ways.

Here I nearly settled on a wrong explanation once. I thought "if it exceeds the specified value
it must be served from memory", but the way to decide that is the ratio of DiskReadBytes to
DataReadBytes (a few percent or less means it is served from cache). Applied for real it was
104-181% across all five deployments: none of them was being served from cache. The cause
was looking only at DiskIopsUtilization. It tells you whether the disk is saturated; it tells
you nothing about the relation to what the client sees.

If you measure, take four: DataReadBytes / DataReadOperations / DiskReadBytes /
DiskReadOperations. Taken in 5-minute intervals, the average I/O size lets you cut out workload
intervals after the fact. You do not have to rely on the timestamps in your own notes.

So what is the reason the amplification factor splits between 0.8 and 1.5? This is what remained to
the end.

Here another of my hypotheses was disproved. I thought "the layout above must also be setting the
amplification factor", but even right after a sequential refill had restored the layout to 88 KiB,
random read amplification stayed at 190-208%. The layout for sequential reads and the
amplification factor for random reads are separate variables.

There was one candidate. Line up the five deployments by "did 4 KiB random writes ever run on that
device" and they split cleanly.

Deployment and interval	4 KiB random write history	Amplification	Client IOPS
Never ran (a run that measured reads only)	none	96.5%	203,961
An interval before it had run	none	83.3%	206,582
After it ran	yes	170.1%	140,423
After it ran	yes	193.7%	131,617
After it ran (a sequential refill does not restore it)	yes	190-208%	97,769-130,080

All five followed that ordering. Even so, this is an ordering of observations, not a control
experiment. I wrote it down as something that would be settled by applying the steps in turn on a
single device.

And that candidate was disproved too

I applied it. It was disproved. On a new deployment I began measuring right after a fill, with
4 KiB random writes never having run.

State	4 KiB random read	Against the specified 100,000
Right after a fill on a new file system (no random write history)	65,947.6 IOPS	0.659
After 300 seconds of 4 KiB random writes	65,430.8 IOPS	0.654
After a sequential refill	65,070.0 IOPS	0.651

It is already 0.659 with no history. Where the "none" rows in the table above were 0.83-0.97,
the same "none" condition produced a value on the 1.5 side. The spread across the three states is
1.3%, and the amplification factor moves with neither layout nor write history.

Five observations lining up cleanly was a coincidence. That is the third disproved hypothesis.

An ordering of observations is not a substitute for a control experiment. All five followed it
without exception, the mechanism was plausible, and it was wrong. When you write "all n cases
follow" as a basis, check at the same time how many unvaried variables were lining up in the same
direction. In these five, the patch version and the client were lined up the same way too.

What remains is those two. The difference between the 0.83-0.97 of F-6 / F-7 and the 1.5 side
later on comes down to the ONTAP patch version (9.18.1P6 against 9.18.1) and the client used
for the measurement. Neither was varied within a single run, so they are named as candidates and
nothing more.

One of those two cannot be varied by a reader either. CreateFileSystem's
FileSystemTypeVersion
is documented as being for FSx for Lustre, and FSx for ONTAP has no parameter for specifying a
version. The list of updatable properties in
Updating file systems
does not include the ONTAP version either. So the patch version can be neither chosen nor
pinned, and a control experiment varying that variable cannot be assembled on the user side.
The client is the only one that can be varied. If you want to press on here, what you can do is
show that the amplification factor does not move when the client is changed, narrowing the
candidate to the patch version. It does not establish causation.

The policy, incidentally, had nothing to do with it. Switching queue-depth to numa to
round-robin to queue-depth during a 4 KiB random read produced no step, and against the 132,044.8
IOPS average of the run containing all three policies, queue-depth alone gave 131,413.5 IOPS —
a difference of 0.5%.

Five things I hit on the instrument and the teardown

It took nine failures to reach a measurement. All of them gates, none of them numbers.
Four of those happened before reaching a mount, so they are in a
separate article
(A: sessions and queues); what belongs here are the five on the instrument and the teardown.
The detail is in the
block measurement runbook
as a table of symptom, cause and how to check.

vdbench resolves include= against the current directory, not against the path given to -f. The fill "completed successfully without writing anything". Had I gone on without noticing, I would have been measuring reads against unwritten blocks (which return zeros).

6. key=value on a vdbench command line is not an override of the workload definition.
It is an assignment to a placeholder inside the parameter file. Passing xfersize=1024k failed with
Unused parameter substitution.

I reported an SSM wait limit as a failure. A 600 GiB write did not fit inside a 7.5-minute
wait, I saw status: InProgress, judged it a failure and tore the environment down.
A wait limit is not a failure.
A sentinel string matched my own explanatory text. Against an implementation that "prints
MISSING if the module is absent", I was deciding with grep -q 'MISSING', and it matched the
explanatory line "A MISSING line means…" in the same output, so F-2 was skipped although NVMe/TCP
was available. A sentinel that your own explanatory text satisfies is not a sentinel.
The teardown trap died partway through the teardown. Under set -euo pipefail, reading
describe-stacks after a delete always fails once the stack has finished disappearing, and that
failure kills the trap itself. The delete ran but the retry did not, and a DELETE_FAILED stack was
left behind. Put the delete call first in the trap, and set +e immediately after it.

Three more hit during the rewrite

The nine above are from the first measurement. The run that recounted things per path added three
more. All of them the same shape as the nine: looking successful while measuring nothing, or
not being able to tell that nothing was measured.

A counter table returned empty. nvmf_lif had zero rows even after 600 GiB had been written,
and reading zero rows as zero bytes gets you the right conclusion for the wrong reason. Which
table to look at, and the detail of that trap, are in the separate article
(B).
A socket aggregation matched my own metric. To sum bytes per destination I treated any field
ending in :4420 as a destination, and bytes_acked:4420 (a byte count) happened to end in
4420, growing one spurious path named bytes_acked. The value was 0 so the sum was not broken, but
it is the same mistake as the eighth.

12. "It passed locally so it will pass in CI" did not hold. This is the repository side rather
than measurement, but the structure is the same. A check scans tracked files, and a new file is
outside the scan while it is untracked. Green locally, red in CI. "Check after the last edit" is
not enough; it is also "after the last git add".

Summary

The same configuration varied by 2.64x across environments. Do not size from a single measured value
What is acting on it is the layout on disk. One disk-side read goes from 87-89 KiB to 11.7 KiB (a factor of 7.5), disk IOPS rises 7.3x and saturates at 89-95% of the specified value. DiskRead / DataRead is about 100%, so the faster side is reading from disk too. It was not being served from memory
What breaks it is small random writes only. Three reads in a row, a spread of 0.2%; 30 minutes idle, within 0.4%; 300 seconds of sequential writes, 0.03%. 4 KiB random writes cost 0.9% at 30 seconds and 77% at 300 seconds, and a sequential refill restores it. It goes back and forth within a single run. Where the threshold sits between 30 and 300 seconds is not known
The warm-side ceiling sits on the client side, and it is not concurrency. Two clients concurrently total 4,204.55 MB/s, 1.86x one client. Meanwhile 512 to 2,048 threads moves throughput by only 0.07% and only scales response time from 216.6 to 860.3 ms. NFS 4 KiB random reads are the same, a spread of 2.2% (5.391 to 21.108 ms). Before you write "I measured the storage ceiling", try adding an EC2 client. Adding threads will not tell you
4 KiB random reads scale precisely with the specified SSD IOPS (about 66% of the specified value, the same ratio at four points). It holds in both the decreasing and the increasing direction. Sequential reads do not depend on the specified IOPS
What splits IOPS between 131k and 204k across deployments is the disk read amplification factor. Client IOPS = specified IOPS / amplification factor. DiskIopsUtilization alone will not tell you. Take DataReadBytes / DataReadOperations / DiskReadBytes / DiskReadOperations. What sets that amplification factor is not known. That it is neither layout nor write history is settled; the remaining candidates are the ONTAP patch version and the client. The patch version cannot be specified on FSx for ONTAP, so a control experiment varying that variable cannot be assembled on the user side
Observation order is not a substitute for a control experiment. I had a candidate that five cases followed without exception; I aimed a measurement at it and it was disproved. Unvaried variables were simply lined up in the same direction
When comparing configuration A against B, match the elapsed time from write to read. I compared without matching it, wrote "1.73x from the policy", and withdrew it later
Which protocol comes out ahead changes with the procedure and the I/O shape. Publish which procedure measured what, not which one is faster
Do not write throughput alone. The figures in this article are iorate=max saturation points, and the same 2,364 MB/s sits at a 216 ms point and at an 860 ms point. Without response time alongside it, that difference disappears

Notes on the AWS documentation

This article contains no remarks about the documentation, and there is nothing to submit.

What it contains are errors in my own way of measuring and measurements that the published
specification cannot account for. Neither is a claim that the documentation is wrong, so I am
setting them out separately.

Content	Kind
2.64x across the same configuration / 1.85x from layout	Measurement. The published specification does not promise this granularity
The cause of amplification 0.8 against 1.5 is unknown	Left unresolved. The candidates are the ONTAP patch version and the client, and the former cannot be chosen on FSx for ONTAP, so no control experiment can be assembled
The per-client ceiling (block side)	Not measured. On the NFS side it was followed through on 2026-09-20 (the table above), so the same way of varying things should apply to the block side
Comparing policies across time, writing 1.73x, and withdrawing it	My error
`nvmf_lif` returning no rows	ONTAP-side behaviour. A question for the vendor, not a matter of AWS documentation

The remarks about the documentation are in the other two: A and
B, each with its own submission status.

Submission status across the three block-protocol articles and S3 Burst Part 2 is collected in
AWS documentation notes and inquiries across the block-protocol articles.

Primary record of the figures

The primary record of the figures and their conditions is in
per-protocol measurement results (Japanese),
and the reproduction procedure is in the
block measurement runbook (Japanese).
Whatever is unconfirmed is written as unconfirmed, so please do not drop that when citing.

Pitfalls with the VDBENCH measurement tool, and items to always record, are collected in
Considerations when measuring FSx for ONTAP's block protocols.

Moving on-premises block storage to AWS? The "divide by 625 MBps" sizing approach comes out wrong

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 30 Sep 2026 05:43:26 +0000

Moving on-premises block storage to AWS? The "divide by 625 MBps" sizing approach comes out wrong — measured session and queue counts

What this article does and does not cover: it covers the measured failure of the "divide by
625 MBps" sizing approach, and four configuration failures hit before mounting. It does not
cover performance conclusions, a comparison against file protocols, or distributing block storage
to multiple sites (details under "What this article does not cover" below).
All figures are from a Single-AZ configuration; Multi-AZ is unverified.

When using FSx for ONTAP over iSCSI or NVMe/TCP, how many sessions should you open? AWS's
procedures surface one number: "Amazon EC2 single client maximum of 5 Gbps (~625 MBps)",
which appears on both the
iSCSI and
NVMe/TCP pages as
the lead-in to the procedure for adding sessions.

One thing first: AWS does not instruct you to divide. The source text says "if you want
throughput beyond this value, follow the EC2 network bandwidth page's procedure to add
sessions," and it's marked (Optional). No formula is given. "required bandwidth ÷ 625 =
session count" is a sizing approach I built after reading that figure. This article tests
my sizing approach, not a formula AWS published.

Measuring it, that sizing approach missed in both directions. 1.82x with one session, 0.20x
with sixteen. And going from one session to two moved sequential throughput by zero bytes.

So this figure can't be used as a divisor to determine session count. What's documented as a
single flow's ceiling in EC2 network bandwidth
turned out to be a separate thing from what a block protocol session measures per session.

NVMe/TCP has nothing equivalent to nr_sessions; the analogous quantity is queue count. That
doesn't honor a request either — asking for 36 produced 4.

What was measured

Assuming no prior knowledge of FSx for ONTAP, here's where this article's measurement happens.

NetApp ONTAP (second-generation SINGLE_AZ_2) file system over either iSCSI (multiple sessions) or
NVMe/TCP (multiple queues)" width="800" height="614">

Figure: what this article measures is the connection count between one EC2 client and one FSx for
ONTAP file system, not multiple clients pooled together.

The figure states the same thing as the table below. What this article measures is the
connection count between one EC2 client and one FSx for ONTAP file system, not multiple
clients pooled together.

Item	Role in this article
VPC / AZ	Single AZ, no network detour, so what's measured is the protocol itself
EC2 client	Varies session/queue count and measures 1 MiB sequential reads
FSx for ONTAP	Provides the iSCSI LUN or the NVMe/TCP namespace
Connection count	For iSCSI, TCP sessions counted with `ss`; for NVMe/TCP, requested vs. effective queue count

What this article does not cover. No performance conclusion is drawn; the numbers here only
serve as evidence that the division misses. The same configuration swings 2.64x across
environments, so don't use these figures for sizing. Why, and what actually drives the numbers,
is in a separate article.
There's no comparison against file protocols (NFS/SMB) either — there's a reason they don't belong
in the same table. And there's no discussion of distributing block storage to multiple sites — this
architecture's distribution layer (FlexCache) distributes volumes, not LUNs or namespaces.

This is one of a set of three, but there's no order to read them in. The block protocol
measurements are split by reader.

Article Reader

A Session and queue counts (this article) Anyone about to mount

B Checking whether ANA is available Anyone who followed the procedure and got stuck

C What moves the numbers Anyone measuring or citing figures

None assumes the other two. Read only what you need.

	Article	Reader
A	Session and queue counts (this article)	Anyone about to mount
B	Checking whether ANA is available	Anyone who followed the procedure and got stuck
C	What moves the numbers	Anyone measuring or citing figures

Measurement environment

Item	Value
File system	FSx for ONTAP second-generation SINGLE_AZ_2, 6,144 MBps x1 HA pair, SSD 4,096 GiB, provisioned 200,000 IOPS
Region	ap-northeast-1, single AZ
ONTAP	9.18.1P5 (the three divisor-table data points). The queue negotiation section used a separate deployment, and its version was not recorded — it wasn't read at measurement time and can't be recovered after the fact. I can't write "all figures in this article are the same version." (Multiple versions coexisting across this configuration is covered in the separate article (C). The cross-article version mapping is in the ONTAP version matrix.)
NVMe read cache	Disabled (confirmed on both nodes)
LUN / namespace	600 GiB, on a 900 GiB volume
Before measuring	Wrote the full 600 GiB once. Thin-provisioned unwritten blocks return zero, so reading without writing first measures nothing
Client	c5n.9xlarge; 50 Gbps network is the guaranteed value
Measurement tool	VDBENCH 5.04.07 direct, `iorate=max` (unlimited), 512 threads, 60s warmup + 300s measurement
Test data / efficiency settings	Not recorded whether the VDBENCH payload was compressible, or whether volume inline efficiency (`StorageEfficiencyEnabled`) was enabled. On the file-protocol side, efficiency settings measurably affected figures; whether the same applies to block is unconfirmed

Because the window is 300 seconds, this article's figures include bursts. They can't be cited as
a baseline.

This article's figures are the saturation point of iorate=max, not an operating point in
production. No target IOPS was given — this is however much could be pushed through. On the
file-protocol side of the same repository, giving a target of 4,400 IOPS produced 4,406 MB/s at
8.86 ms, while unlimited produced 4,204 MB/s at 121.79 ms. Unlimited is 5% lower
throughput and 14x the response time.

With threads fixed at 512 and unlimited, response time is an identity of 512 / IOPS (confirmed
against a separate measurement: 216.6 ms derived versus 216.588 ms measured). The values below
can be re-read through that identity — 1,135.18 MB/s is about 451 ms, 1,970.43 MB/s is
about 260 ms. Both are derived values, not recorded measurements.

The divisor division doesn't hold

Over iSCSI, I varied the counted number of TCP connections and measured 1 MiB sequential
reads. These are actual counts from ss, not the requested value.

TCP counted	Measured	Predicted from 625 MBps	Measured / predicted
1	1,135.18 MB/s	625 MB/s	1.82x
2	1,135.18 MB/s	1,250 MB/s	0.91x
16	1,970.43 MB/s	10,000 MB/s	0.20x

With one session it's faster than predicted; with sixteen it's a fifth of the prediction.

And going from one session to two moves sequential throughput by zero bytes — 1,135.18 appears
twice in a row. So the ceiling hit with one session is not a single flow's capacity.

1,135 MB/s is 9.08 Gbps. In the same environment, the file protocols landed near the documented
5 Gbps on a single connection — NFS 591.62 MB/s (4.73 Gbps), SMB 574.24 MB/s (4.59 Gbps).
A single iSCSI session carries roughly twice that. Why is unconfirmed. (This comparison is
against single-connection figures from other protocols in the same environment, not a direct A/B
test against iSCSI.)

Here's the split between what's documented and what's an unverified candidate. The single-flow
5 Gbps ceiling itself is documented. AWS defines a single flow as "a TCP or UDP flow identified
by a unique 5-tuple (source IP, destination IP, source port, destination port, protocol)" and
states that outside a cluster placement group, a single flow is limited to 5 Gbps
(Amazon EC2 instance network bandwidth).
The same page documents the workarounds too — up to 10 Gbps inside a cluster placement group,
up to 25 Gbps within the same AZ with ENA Express, or spreading traffic over multiple paths with
Multipath TCP (MPTCP)
(ENA Express).

From here on, these are unverified candidates. Whether this measurement environment was
inside a cluster placement group was not recorded and is unconfirmed. 9.08 Gbps would fit inside
a 10 Gbps allowance, but whether this configuration was inside that allowance is unknown.
Whether iSCSI, NVMe/TCP, NFS, and SMB all count as "one" the same way against the 5-tuple
definition is also unconfirmed. ENA elastic network interfaces are documented as having up to 8
RSS queues
(Optimize network performance on EC2 Windows instances —
that page covers Windows; I haven't checked the equivalent for Linux), and if protocols
distribute across CPU cores differently, the apparent ceiling could differ as a result. This
article does not test these candidates. A separate measurement would be needed.

The requested queue count doesn't hold

NVMe/TCP has nothing equivalent to nr_sessions. It opens a TCP connection per queue, so the
quantity that corresponds to session count is queue count.

Requesting via --nr-io-queues produced this. NCQA (Number of Completion Queues Allocated) and
NSQA (Number of Submission Queues Allocated) are the actual completion and submission queue
counts the NVMe Get Feature command reports back to the host.

Mode	Requested queue count	Effective NCQA / NSQA	Controller count
Single flow	1	1	1
AWS's procedure as written (one `connect-all`)	unspecified	4	2
Explicit queue count	36 (vCPU count)	4 (unchanged)	2

Requesting 36 produced 4 regardless.

This isn't speculation — it's documented on the ONTAP side. vserver nvme subsystem show's
description of -default-io-queue-count states "The actual value used when a connection is
established may vary depending on the host and transport protocol used," and
vserver nvme show-host-priority
states that I/O queue count is determined per combination of node, transport, and host
priority. NetApp's KB has an example where setting 15 on a subsystem produced 2 on the host
side.

Don't use the example's value as your own environment's value. In the example above,
nvme-tcp / regular is 2, but in this environment it was 4.

As a byproduct, one thing left as "unconfirmed" in planning got resolved. queue_count includes
the admin queue and matches NCQA + 1 (1 -> 2, 4 -> 5). With one queue, there was also exactly
one established TCP connection carrying I/O.

Four failures hit before mounting

Nine failures were hit before reaching a measurement. All were gates, not numbers. Each one
looks like it succeeded while measuring nothing. Four of those happened before mounting, so
they're here; the rest concern the measurement tool and teardown and are in the
separate article (C:
what moves the numbers). Details, as a table of symptom / cause / how to check, are in the
block measurement runbook.

JunctionPath turned out to be required. I had assumed it wasn't needed since a LUN isn't read through a junction, but the API rejects that. Worse, the property reference's entry for JunctionPath says This parameter is required. in the body while the same property's Required: column says No. That's a literal self-contradiction (still present as of 2026-09-17). Trust the API. It fails 25 minutes after file system creation starts.

The same page has an example of writing this without contradiction. StorageEfficiencyEnabled
uses Required: Conditional and explains the condition in the body (required when the volume is
RW). Conditional is a value this reference already uses, so JunctionPath could take the
same form.

I can't state exactly which condition makes it required. What I measured is one case,
OntapVolumeType: RW, where it was required. I did not test whether it's optional for
DP. Determining the condition belongs to whoever holds the API's implementation;
what I can report stops at "the body and the Required: column disagree."

2. AL2023 does not create /etc/nvme/hostnqn. AWS's procedure assumes this file exists and
reads it with cat (RHEL creates it). Generating it with nvme gen-hostnqn was necessary.

3. iSCSI login is asynchronous. iscsiadm --login returns before iscsid actually connects. In
this environment it took about 100 seconds. Counting with ss right after returned 0, so
every alias and fill attempt right after also came up empty. "The command returned" is not
"the connection is up."

4. find_multipaths yes does not create a map for a single-path device. When intentionally
measuring a single flow, the path count is 1 by design, and with the default setting
/dev/mapper/<alias> never appears. Explicit registration via multipath -a <wwid> was
necessary. That wwid is also more reliably read from the device with scsi_id than assembled as
3600a0980+serial (assembling it is useful as a cross-check, not as the primary source).

Summary

Sizing session count by dividing required bandwidth by 625 MBps missed in both directions. 1.82x with one session, 0.20x with sixteen.
Going from one session to two doesn't move sequential throughput. The ceiling hit with one session is not a single flow's capacity
NVMe/TCP queue count is the result of negotiation. Requesting 36 produced 4. Read the effective value with nvme get-feature --feature-id 7. queue_count includes the admin queue, so it equals NCQA + 1
All four configuration failures looked like success while measuring nothing. JunctionPath is required, AL2023 doesn't create hostnqn, iSCSI login is asynchronous at about 100 seconds, and find_multipaths yes doesn't map a single-path device.
Don't use this article's figures for sizing. The same configuration swings 2.64x across environments
What drives that 2.64x swing is out of scope for this article. It's covered in a separate article

AWS documentation notes and support inquiries

This article touches two points in the documentation. One has been filed; the other is not
being filed.

Finding	Source	Status
`JunctionPath`'s body and `Required:` column disagree. Traced to the API reference, which propagates into CloudFormation and the CDK	same	Filed (2026-09-17). AWS's reply (2026-09-22): explained that when OntapVolumeType is RW, JunctionPath is required; when DP, it cannot be specified at all — so "Required: No" has a rationale when read across both volume types combined. The proposed change to `Required: Conditional` was described as feedback to be shared with the internal team, with no commitment on adoption. The question of whether the API reference is upstream of CFN/CDK went unanswered ("internal AWS information, cannot disclose"); instead AWS said it would file feedback against the API reference, CFN, and CDK pages individually
Sizing by dividing 5 Gbps (~625 MBps) doesn't match measurement	iSCSI / NVMe/TCP procedures	Not filed. AWS never states a formula, so this isn't a documentation error

Why the second one isn't filed. This is a record of my own sizing approach missing, not a
finding about the documentation. The figure itself is correctly documented as a single flow's
ceiling; I was the one who used it as a divisor for session count, and that's where it broke
down.

The first was filed with AWS Support on 2026-09-17. The submission included the observed
requirement (omitting it fails stack creation 25 minutes in) and the fact that the CDK inherits
it as an optional type, requesting a fix on the API reference side.

AWS's 2026-09-22 reply settles only the rationale behind "Required: No" (it holds when RW and
DP are read as one combined attribute). It doesn't change the underlying fact that JunctionPath is
required when creating an RW volume, and whether Required: Conditional will actually be adopted
remains unconfirmed.

Primary record of the figures

The primary record of the figures and conditions is in the
per-protocol measurement results
(Japanese); the reproduction steps are in the
block measurement runbook
(Japanese). Anything marked unconfirmed there is marked unconfirmed for a reason — don't drop
that qualifier when citing it.

Considerations before sizing session and queue counts, and pitfalls hit before mounting, are
collected in
Considerations when measuring FSx for ONTAP's block protocols.

Submission status across the three block-protocol articles and S3 Burst Part 2 is collected in
AWS documentation notes and inquiries across the block-protocol articles.

Amazon S3 Files vs. Amazon FSx for NetApp ONTAP: two designs for using S3 as a file (S3 Burst Part 2)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 30 Sep 2026 05:43:06 +0000

Amazon S3 Files vs. Amazon FSx for NetApp ONTAP: two designs for using S3 as a file, measured on the same host (S3 Burst Part 2)

Keeping the primary copy of your data in one place while reaching it both as a file (NFS/SMB)
and over the S3 API: have you had to weigh which design to pick before committing to one?

There is more than one way on AWS to combine the two. Worth pausing on this, though: every
design has a point where "fast enough" turns into a ceiling, and that point sits in a different
place for each one. It helps to know where each one bottlenecks before choosing. This article
measures three of them from the same host.

The same configuration as the image above, reproduced so it reads in environments where images
don't render.

flowchart LR
    subgraph AWS["AWS Cloud"]
        subgraph VPC["VPC, single AZ"]
            EC2["EC2 client"]
            subgraph A["A: This architecture"]
                AP["S3 Access Point"]
                FSXN[("Amazon FSx for<br/>NetApp ONTAP")]
                AP --> FSXN
            end
            S3["Amazon S3<br/>(bucket)"]
            S3FILES["Amazon S3 Files<br/>(via efs-proxy)"]
            EC2 -->|"write over S3 API"| AP
            FSXN -->|"read over NFS / SMB"| EC2
            EC2 <-->|"read/write over S3 API"| S3
            EC2 <-->|"read/write over NFS"| S3FILES
        end
    end

The figure states the same thing as the table below. Only A uses a different protocol for
writes versus reads; B and C have the same client reading and writing over the same protocol.
The figure's three panels map directly to the three below.

	Path	Write	Read
A	FSx for ONTAP's S3 Access Point (this architecture, below)	S3 API	NFS / SMB
B	Amazon S3	S3 API	S3 API
C	Amazon S3 Files (an S3 bucket mounted over NFS)	NFS	NFS

Only A uses a different protocol and client for writes versus reads. That's the substance of
this architecture: something written over the S3 API is read over NFS/SMB with no copy job in
between. That's why the figure shows two arrow types for A, while B and C are a single
bidirectional arrow — the same client reads and writes over the same protocol in B and C.

Measure throughput and lay it out in a table, and the three numbers sit side by side. Sitting side
by side reads as "this one is faster," but each of the three was bottlenecked for a different
reason. Under each panel of the figure is where that path was actually bottlenecked. A throughput
value provisioned on the file system, client-side network bandwidth, the CPU of a process running
on the client. What the bottleneck is matters more to design than what the ceiling number is.

The measuring side has its own prerequisite. There are five settings that, left at default,
measure something other than what you intended. That's covered in the
previous article (Part 5, "five defaults that can invalidate a measurement").
This article's figures were measured after changing all five from default.

Scope of this article

What it covers

The three ceilings that bind on the FSx for ONTAP side, and which one actually applied
What bottlenecks each of the three paths
What FlexCache does for reads
What becomes the ceiling as client count scales to 8
Whether a 15-minute sustained write decays
The bottleneck shift with small objects
What each tool can and can't observe
What hasn't been measured yet

What it does not cover

Which configuration is faster. The three figures in this article mix a provisioned ceiling with an unprovisioned elastic ceiling. They aren't laid out to be read side by side for a winner, so wherever a number appears, the type of ceiling it's hitting is stated alongside it.
How to choose a configuration itself. Where to place the read side, which side holds the primary copy, when you want writes visible on the bucket side — these are decided by other factors and throughput numbers alone don't decide them.
The semantics of Amazon S3 Files propagation. That file-to-bucket propagation converges within roughly a 60-second window, and the reverse direction is on the order of seconds, is a different subject and belongs in a separate article.
IAM policy and the authorization model. The AWS side of an S3 Access Point (IAM identity policy, access point policy, VPC endpoint policy, SCP) and ONTAP's two-layer authorization are a different axis from this article's subject (throughput). Minimum-privilege policy examples are in the S3 AP design guide's authorization design section.

Measurement environment

Figures can't be reproduced without the environment they came from, so it's placed first.

Item	Value
Measured on	2026-09-01 and 2026-09-02
Region	ap-northeast-1
ONTAP	9.18.1P3D1 (the second-generation NFS 2-host measurement used 9.18.1P6. The cross-article version mapping is in the ONTAP version matrix)
File system	SINGLE_AZ_1 (first-generation Single-AZ). Measured at two throughput capacities, 128 MBps and 2048 MBps
SSD	1024 GiB, `DiskIopsConfiguration` at two points, 3,072 and 40,000
Client	c5n.9xlarge (single-client measurements), c5n.2xlarge x8 (client-count scaling test — adding one client at a time and observing how the combined throughput grows)
Volume	UNIX security style, NFSv3, `actimeo=0`
Test data	Uncompressible (from `/dev/urandom`). Zero-filled data was not used. These are figures from a world with efficiency features off
AZ configuration	All figures are from a Single-AZ configuration. Multi-AZ is unverified

Both NFS and SMB were measured, but the scope differs. The per-protocol comparison table and
per-path bottlenecks are measured over NFS. What was measured over SMB is the growth pattern with
increasing client count, the 15-minute sustained write, and Multichannel channel count. The
seven scenarios under cache control were not measured over SMB. This architecture can be read
over either NFS or SMB, but this article's figures are all NFS. If you plan to read over SMB,
the figures below don't directly apply. What carries over is the measurement approach and the
procedure for isolating which type of ceiling is binding.

Reproducibility under identical conditions was 0.02-0.14%. With that baseline established, a
"moved 30%" statement later in this article can be said not to be run-to-run noise.

Unless otherwise noted, figures in the body are single measurements. Where something was
measured twice, both values are shown as 415.1 / 415.7 rather than averaged or taking a median.
Totals in the client-count tables are the sum of each host's value.

And every figure in this article is a saturation point — however much could be pushed through at
a fixed concurrency. No point gives a target throughput or target IOPS. This distinction
matters as much as stating which type of ceiling applies — a saturation point answers "where does
it choke," not "what response time does it deliver at that load." Measuring both on a separate file
system (second-generation, 6,144 MBps), a target of 4,400 IOPS produced 4,406 MB/s at 8.86 ms,
while unlimited produced 4,204 MB/s at 121.79 ms. Unlimited is 5% lower throughput and 14x
the response time. Source:
the series with a varied target value
(Japanese).

Response time exists for the S3 API path and not for the file path. The S3 API side's body
notes p50 alongside throughput (363.7 ms at concurrency 1, 1054.9 ms at concurrency 64, and so on).
The file path's load generator didn't retain the values it output, and the environment has since
been torn down. Because concurrency was fixed, it can be re-derived through the identity
concurrency / IOPS, but that's a derived value, not a measurement.

A separate article pushes further on the same property on the block protocol side
(separate article). It
covers a case where quadrupling thread count moves throughput by only 0.07% while response time
alone quadruples, and the same 2,364 MB/s appears at both a 216 ms point and an 860 ms point.
That same caution applies when citing this article's figures.

The block protocol side is a set of three.
Session and queue counts
(for anyone about to mount),
checking whether ANA is available
(for anyone who followed the procedure and got stuck), and
what moves the numbers
(for anyone measuring or citing figures). There's no order to read them in.

The measurement host was c5n.9xlarge. The reason is isolation. With an "Up to"-labeled instance,
once a measured value plateaus, you can't tell whether you hit the storage-side ceiling or the
client-side bandwidth ceiling. c5n.9xlarge's 50 Gbps is a guaranteed value. Every figure below
sits well under 50 Gbps, so the possibility that the client-side bandwidth was the cause of a
plateau can be ruled out.

Only the client-count scaling test used eight c5n.2xlarge instances instead, since that test wants
more clients rather than a bigger single client.

The client-count scaling test and the sustained write used a separate file system

The table above is first-generation. Only the client-count scaling test and the 15-minute
sustained write were measured on a separate, second-generation file system. The first-generation
one is a shared environment, and raising or lowering its provisioned throughput capacity would
affect other measurements.

Item	Value
Measured on	2026-09-05
File system	SINGLE_AZ_2 (second-generation Single-AZ), 6,144 MBps x1 HA pair
SSD	4,096 GiB, provisioned 200,000 IOPS
Client	c5n.2xlarge at 1 / 2 / 4 / 6 / 8 hosts. 64 threads and `nconnect=16` fixed per host

Because the environment differs, these figures can't be placed in the same column as the table
above. Load per host was fixed and only the host count was varied, because fixing total thread
count and dividing by host count would change two things at once per data point, making it
impossible to separate which one caused the effect.

Six settings to change from default before measuring throughput (Part 5)

Each one, left at default, measures something other than what you intended. And the number
that comes back doesn't look unnatural, so measuring alone won't reveal it. The table and how each
was noticed are in Part 5; only the names are listed here.

Linux's NFS mount opens only one TCP connection by default (plateaus around 590 MB/s)
Data created with dd if=/dev/zero (all-zero blocks never reach disk)
Volume inline efficiency (compression/deduplication) enabled
DiskIopsConfiguration at AUTOMATIC
Reads slightly exceeding the read cache's capacity

More on item 5 — the read cache's two-tier structure

Item 5 is covered in detail because it bears directly on this article's figures.

FSx for ONTAP's file server has two tiers of read cache: in-memory and NVMe
(performance specs). For
ap-northeast-1's first-generation Single-AZ, what's published differs by tier.

Tier	Capacity at 2048 MBps	Source
In-memory	256 GB	Performance specs, the "other regions" table
NVMe read cache	Not stated	No column exists in that same table

For NVMe, only a condition is stated in a separate section: "A Single-AZ 1 created on or after
2022-11-28 with a throughput capacity of 2 GBps or more gets a read cache." This file system meets
that condition. Checking with the ONTAP CLI directly confirmed it was enabled.

128 MBps configuration : system node external-cache -> 0 records
2048 MBps configuration: is_enabled: true (both nodes)

And how SSD IOPS gets used is stated explicitly in the same documentation. SSD IOPS is used only
when reading data that's in neither the in-memory cache nor the NVMe cache.

With that in mind, the measurement can be read this way. A 280 GiB read exceeds the in-memory
cache's 256 GB (= 238 GiB) by 18%, but there's an NVMe read cache behind it. This
configuration's NVMe capacity isn't published (in the four regions with a column, it's 1,900 GB at
the same 2048 MBps). The measured 98.5-99.9% of read bytes not going to disk is consistent with
coming back from these two tiers (breakdown under "Per-tool visibility" below).

That's why the observation "raising SSD IOPS from 3,072 to 40,000 made reads 7.14x faster"
didn't square with the documentation. If it's coming back from cache, SSD IOPS shouldn't matter.

That mismatch went away on re-measurement (2026-09-17). Setting the working set to 512 GiB —
2.15x the in-memory cache's 238 GiB — and disabling the NVMe cache, re-measuring showed
96-126% of read bytes coming from disk, with the low end saturating at DiskIopsUtilization
112%. "SSD IOPS determines the disk throughput level" is consistent with measurement under
this condition. So the mismatch's cause wasn't the documentation — it was my measurement:
a 280 GiB working set only exceeded the cache by 18%.

The ratio itself changed, though. 256.47 -> 1,246.67 MB/s, 4.86x. 7.14x doesn't
reproduce. And what's capping the high side, around 1,250 MB/s, is none of the published
utilization metrics (disk throughput 67% / IOPS 43% / network 43%). It isn't concurrency
either — going from 8 to 32 to 64 threads drops it from 1,071 to 999 to 958, with response time
alone scaling proportionally (6.9 -> 31.0 -> 59.3 ms). This is left unresolved.

To check in your own environment, the order is two steps. Check for an NVMe cache with
system node external-cache, and then read a volume that reliably exceeds the sum of both
tiers in one pass. A first-generation Single-AZ at 2 GBps or more gets an NVMe cache by
default — there's no way to prevent it from being created, only to create it and then disable
it. I forgot this once and measured four points with it left enabled.

Item 6 — SMB Multichannel disabled by default

ONTAP's SMB Multichannel is disabled by default. This is a separate setting from SMB 3.x
itself, so a channel can stay at one even while negotiated at dialect=3.1.1. That state held in
this measurement environment: while one host was reading, only one established TCP connection to
the SVM existed.

Enabling it doesn't reach connections already established. Enabling only affects new
connections. Removing and re-mapping the drive kept the same local port, and it only became four
channels after restarting the SMB client service.

max_connections_per_session is a separate value. With Multichannel disabled, setting it to 32
still yields one channel, and you can misread "raised the limit but it didn't increase" as a
result.

The mechanism, sources, and where to read channel count are in
SMB Multichannel is disabled by default, and enabling it doesn't reach already-established connections
(Japanese).

Three ceilings that limit writes on the FSx for ONTAP side

Before getting into the three-path comparison, here's what limits this architecture's own side.
There's a ceiling that doesn't move even when throughput capacity is raised, and not knowing it
leads to a sizing miss.

One term first. Throughput capacity is a value — 128 / 256 / 512 / 1024 / 2048 MBps and so on —
that you set on the file system. It isn't metered usage; you're billed by time against the
value you set. This article measures at two points, 128 MBps and 2048 MBps.

For ap-northeast-1's first-generation Single-AZ, three ceilings bind on writes, and the one that
actually applies is whichever is smallest.

Ceiling	This environment's value	How it's determined
Provisioned throughput capacity	128 / 2048 MBps	Set by you
Disk throughput out of SSD	768 MBps (768 MBps/TiB x 1 TiB, when SSD IOPS is `AUTOMATIC`)	Driven by the provisioned SSD IOPS
Per-HA-pair ceiling	Write 750 MBps (read is 2,048 MBps)	Determined by the file system's generation and configuration

Source: performance specs.

The second row isn't a fixed value determined by SSD capacity alone. 768 MBps/TiB is what you
get when SSD IOPS is set to AUTOMATIC (3 IOPS/GiB). Raising SSD IOPS with USER_PROVISIONED
raises that 768 MBps/TiB figure right along with it. Conversely, raising only throughput
capacity to 2048 MBps while leaving SSD IOPS at AUTOMATIC leaves disk throughput capped at 768
MBps/TiB. There are two levers, and raising only one doesn't move the ceiling.

Laying out which configuration hits which ceiling:

Throughput capacity	Ceiling that applied	Measured (uncompressible, 8 MiB, concurrency 64)	Against that ceiling
128 MBps	The provisioned 128 MBps	133.4 MB/s	104%
2048 MBps	The per-HA-pair write ceiling, 750 MBps	415 MB/s (measured twice: 415.1 / 415.7)	55%

Landing at 104% of the 128 MBps configuration — slightly over the provisioned value — is a
burst effect. This configuration's disk throughput can sustain 128 MBps continuously, or 600
MBps briefly. Bursting was available but the measurement stopped at 133.4, so the provisioned
throughput capacity was the binding ceiling in this configuration.

On second-generation, the meaning of the provisioned value itself is different. In a separate
measurement, a second-generation Single-AZ's provisioned value points to a disk throughput
baseline, and the network-side baseline is roughly twice that. On a configuration provisioned
at 1,536 MBps, 1 MiB sequential reads produced 2,882 MB/s for 27 minutes, then 1,439 MB/s.
Carrying this table's first-generation reading over to second-generation misses a read sizing
estimate by 2x
(measurement record)
(Japanese).

At the 2048 MBps configuration, the provisioned value stopped being the ceiling. The denominator
for percentage changes here, so both are given. 415 MB/s is 20% against the provisioned 2048
MBps. Against the per-HA-pair write ceiling of 750 MBps that actually applied, it's 55%. The
surprise of "provisioned 2048 MBps, only got 415" is explained by the provisioned value no longer
being the applicable ceiling.

What's consuming the remaining 45% — the unused portion of the 750 MBps — hasn't been isolated.
Two candidates come from the published specs.

Writes use twice the network bandwidth of reads. This is documented explicitly, because a write replicates to the secondary file server, so a single write turns into twice the network throughput.
A fixed cost per request. Raising concurrency only stretched p50 (median latency) out to 1054.9 ms (at concurrency 64). If there's a fixed cost per single request, raising concurrency doesn't raise requests-per-second. If request rate doesn't rise, total throughput doesn't rise either.

Which one is doing how much hasn't been isolated. This isn't a case where "the reason it doesn't
reach 750 is this one thing" can be stated.

Bottlenecks sitting in different places per path

This is the main point. Three numbers that come out in the same unit, MB/s, were each plateauing
somewhere different. The figure's A / B / C map in that order.

Path	Where the bottleneck sits	What increases it
This architecture (FSx for ONTAP S3 Access Point)	Over the S3 API path, the provisioned throughput capacity on the file system, shared across all clients	Raise the provisioned value
Amazon S3	Not the service side — client-side network bandwidth	Add more clients
Amazon S3 Files	The CPU of the proxy process running on the client	Use a bigger client

Note: the row above is scoped to the S3 API path. It's based on measurement (1.01x with 4
clients). On the file path (NFS/SMB) of the same file system, reads exceeded the provisioned
value of 6,144 MBps by more than 2x. Reads served from cache don't go through disk, so stating
"the provisioned value is the ceiling" without naming the path would contradict that
measurement. Details in
per-protocol measurement results
(Japanese).

Here's the decision flow for telling which of the above a measurement is hitting, in your own
environment.

flowchart TD
    START["A measured value plateaus"] --> Q1{"Does the total grow<br/>when adding clients?"}
    Q1 -->|"No"| C1["Provisioned throughput capacity<br/>on the file system<br/>(this architecture's S3 API path)"]
    Q1 -->|"Yes"| Q2{"Is the per-client value<br/>near the client's network bandwidth?"}
    Q2 -->|"Yes"| C2["Client-side network bandwidth<br/>(Amazon S3)"]
    Q2 -->|"No"| Q3{"Is CPU high on a process<br/>running on the client?"}
    Q3 -->|"Yes"| C3["CPU of a proxy process<br/>on the client<br/>(Amazon S3 Files)"]
    Q3 -->|"No"| C4["A factor outside these three.<br/>Isolate it separately"]

The figure states the same thing as the table below. A single-host measurement alone can't
determine the type — telling them apart needs either adding clients, or holding transfer size
and concurrency fixed while varying something else at one host (details under
a single-client figure that was actually the client's own limit
above).

FSx for ONTAP — a shared ceiling that wasn't one thing

This is the left side of the figure's panel A, the S3 API write side. Adding clients doesn't
raise the total. With the same setting and varying client count, 4 clients produced 1.01x.
Since the total doesn't change, each client's share is roughly 1/n of it.

The reason is that the ceiling is a value provisioned on the file system. This is a
configuration where capacity planning is done per file system, not per client. If more clients
are expected, plan for it on the throughput capacity side.

Note that reads came out higher than the provisioned value. On a 128 MBps configuration, reads
reached 579.3 MB/s (8 MiB, concurrency 16), 4.5x the provisioned value. Using the write-side
ceiling to size reads will miss.

How the ceiling splits as client count grows to 8

The result above, up to 4 clients, is first-generation with the provisioned value as the ceiling.
Scaling to 8 clients / 128 connections on second-generation, with the same file system, same
setting, and same connection count, the total swung 5.5x. The only thing changed was "where each
host reads from."

With every host reading the same file:

Hosts	Connections	Client total	Port-measured	Per host
1	16	2,664.58 MB/s	2,698.7 MiB/s	2,664.58
4	64	8,562.94 MB/s	8,635.2 MiB/s	2,140.74
8	128	11,916.29 MB/s	12,173.0 MiB/s	1,489.54

With the working set held at a fixed 600 GiB, split into 75 GiB per host at 8 hosts, so no block
is shared:

Hosts	Connections	Client total	Port-measured	Per host
2	32	2,299.05 MB/s	2,323.4 MiB/s	1,149.53
8	128	2,173.37 MB/s	2,191.7 MiB/s	271.67

Quadrupling connections from 32 to 128 doesn't grow the total. It's already saturated at 32.

The difference comes down to how much is served from the file server's read cache. SSD IOPS is
used only when reading data in neither the in-memory nor the NVMe cache, per the
performance specs. Overlap
in read ranges pulls toward the former; no overlap pulls toward the latter. There isn't one
ceiling — there are two, and which one a workload hits depends on the workload.

A hypothesis that turned out wrong

Before this measurement, I had assumed the SVM's single data LIF was a roughly 4,400 MB/s
ceiling. That was wrong.

Since client-reported totals aren't evidence on their own, I cross-checked against ONTAP's
physical port cumulative counters (nic_common's transmit_bytes), differenced. The
port-measured value exceeded the client total by within 1.3% at every point, and all 8 points went
through the same single port. That means a single LIF was carrying 12,173 MiB/s — 102 Gbps.

One thing tripped up this cross-check. The instantaneous-value counter returned 0 even while
traffic was actually flowing. Only the differenced cumulative value was usable.

This changes how capacity planning should work. Rather than counting hosts, decide first whether
each host's read range overlaps. A non-overlapping design pulls toward the disk path's ceiling,
so adding hosts doesn't grow the total — that's what the 32-connection and 128-connection rows
above show.

A single-client figure that was actually the client's own limit

A second, similarly shaped mistake, this time by me. In a separate round, on a first-generation
2,048 MBps configuration, I got roughly 1,250 MB/s and treated it as "this configuration's read
ceiling." I checked with two clients (2026-09-19).

First-generation 2,048 MBps, SSD 1,024 GiB, SSD IOPS 40,000 (USER_PROVISIONED), NVMe read
cache disabled, volume 800 GiB, client c5n.9xlarge x2, each reading a non-overlapping 300
GiB, 1 MiB sequential, 8 threads, o_direct, ONTAP 9.18.1P6, ap-northeast-1.

Condition	Client total	Disk throughput utilization
Single host only	1,195.27 MB/s	63% (SSD IOPS also 40%)
Two hosts simultaneously	981.90 + 919.58 = 1,901.48 MB/s (1.59x)	102-103% (burst)

With a single host, the disk path was only at 63% — the file system side had headroom. Only
with two hosts did the disk side saturate.

From here I chased "so what's the per-host ceiling?" The answer was two of my own settings, not
a resource. On the same file system, I varied rsize to 1 MiB and thread count (the next day).

Condition (1 host)	Measured	Response	Disk throughput utilization
Transfer 64 KiB, 8 threads	1,195.27 MB/s	6.69 ms	63%
Transfer 1 MiB, 8 threads	1,446.66 MB/s	5.443 ms	77.2%
Transfer 1 MiB, 16 threads	1,646.17 MB/s	8.208 ms	102.6%
Transfer 1 MiB, 32 threads	1,647.42 MB/s	16.420 ms	102.6%
Transfer 1 MiB, 64 threads	1,648.71 MB/s	32.838 ms	102.6%

Transfer size added +21%, concurrency added another +14%, and then it stopped. Past 16 threads,
only response time keeps doubling while throughput doesn't move. It's queueing, not gaining.

And a single host reaches the same 102.6% that two hosts reached. IOPS sat at 50.9% and network
at 66.3%, neither saturated. I could not name a specific client-side device that was saturated.

This "can't name it" was traced to a different tier and resolved separately (2026-09-20). On
second-generation, 6,144 MBps / 200,000 IOPS, measuring five ways of splitting the working set
shows adding volumes moves it only +1.0%, while adding SVMs moves it +26.3% and adding
clients moves it +52.7%. So what matters is the number of distinct IPs the client talks to
and the client count, not thread count (+8.9% going from 512 to 1,024). With two clients, the
disk side reaches 96.6% of the provisioned value, and only then does the file system side become
the actual limiter. No single device can be named because the limiter isn't one counter — it's
how receive processing is distributed
(measurement)
(Japanese).

This reinforces this article's claim. The table at the top says to confirm the type of ceiling;
a single-host measurement can't determine that type. Using a value measured alone as a ceiling
for capacity planning under-buys. There are two ways to check — adding hosts and fixing transfer
size/concurrency — and both arrive at the same conclusion.

I also chased the discrepancy between them, without using any additional resources. Both file
systems had already been deleted, but CloudWatch retains data for 63 days after deletion.

This architecture doesn't publish a single burst-remaining metric (zero out of 77 metrics contain Balance). My earlier explanation, "because burst remainders weren't aligned," doesn't hold, because nothing to align exists.
The denominator for utilization was the provisioned value itself (2,048 MB/s). DiskReadBytes / (utilization / 100) converges to 2,048 across every minute. On first-generation, over 100% just means "more than the provisioned value came out," not evidence of spending a burst allowance. Second-generation's 1,536 config at 203.5% is against a baseline, so the same metric has a different denominator by generation.
And both sessions were hitting the same server-side ceiling of about 2,040 MB/s. The 15% client-reported difference wasn't a difference in what the file system delivered.

As one discrepancy closed, a second one — between two counter systems — opened, and it closed
the same day too. At saturation, the server reported about 2,043 MB/s while vdbench reported
1,646-1,649 (19% short). The cause was how the measurement was taken. Aligning the window's
start to a minute boundary and comparing only the minutes fully contained in the window, the
two systems agree within 0.2% (300s window: 271.04 vs 271.57; 900s window: 326.70 vs 326.07).
The 19% arose from taking a 60-second average of a minute where load was only partially applied
as the server-side value. Against a steady-state 271 MB/s, the start minute was 216.85 and the end
minute 58.47. There's nothing to restate about FSx for ONTAP here. Making the two agree
requires both aligning the window and discarding the partial minutes
(measurement)
(Japanese).

The shared ceiling split two ways over SMB too

The same shape showed up over SMB. Varying only client count across 8 clients didn't produce
one single answer.

Everything below the share is identical between the two. Same single CIFS share, same SMB 3.1.1
with 4 Multichannel channels, same physical port on the same node of the same SVM. Only what each
host reads differs.

Total at 8 hosts	Read range
10,721.19 MB/s (port-measured 11,194.7 MiB/s)	Every host reads the same range of the same file
4,225.31 MB/s	Each host reads its own 1/8 of the range

The former was still climbing at 8 hosts; the latter plateaued between 4 and 8. The former's
growth is the result of ONTAP's memory serving the overlap. Citing only one of them leads to the
opposite conclusion.

Client-side totals aren't evidence on their own here either. A single read can satisfy multiple
clients, so bytes never sent over the wire can get counted. Every point here is cross-checked
against ONTAP's physical port cumulative counters.

Per-point figures, and what this measurement doesn't answer (the 180-second window doesn't
separate burst from baseline), are in the
SMB host-count test
(Japanese).

Amazon S3 — a total proportional to client count

The figure's panel B is made of just this one bidirectional arrow.

Hosts	Write total	Per host	Read total	Per host
1	581.5 MB/s	581.5	496.3 MB/s	496.3
2	1158.0 MB/s	579.0	977.5 MB/s	488.8
4	2362.5 MB/s	590.6	1974.7 MB/s	493.7
6	3601.4 MB/s	600.2	3038.5 MB/s	506.4
8	4783.2 MB/s	597.9	3992.2 MB/s	499.0

Going from 1 to 8 clients, write totals grew 8.23x and reads 8.04x. The per-host value doesn't
drop. The total reached 38.3 Gbps at 8 hosts, and this range never showed a point where it
stopped being proportional to host count. Each host used a distinct prefix, so per-prefix request
ceilings weren't tested.

The roughly 500 MB/s visible with a single host was not Amazon S3's ceiling — it was the
client's. Since adding hosts grows the total, capacity planning here comes down to counting
client-side bandwidth.

Amazon S3 Files — the CPU of a proxy process on the client

This path's structure differs from the other two, so the structure is described first. The
figure's panel C shows the client and efs-proxy sharing the same box — that's it. Mounting S3
Files starts a transfer process called efs-proxy on the client, and the NFS client connects to
that process. It's visible in the mount info.

127.0.0.1:/ nfs4 rw,vers=4.2,rsize=1048576,wsize=1048576,proto=tcp,port=20812,addr=127.0.0.1

The mount target is 127.0.0.1 — itself. Data is then carried from efs-proxy to the service
side. FSx for ONTAP's mount target is the file system itself, so this is structurally different.

With that structure in mind, I measured what
not supporting nconnect
means in practice. nconnect is an option that increases the number of TCP connections an NFS
mount opens.

Stream count	Read	Write
1	67.9 MB/s	34.6 MB/s
4	264.3 MB/s	140.5 MB/s
8	451.3 MB/s	281.0 MB/s
16	404.8 / 450.0 MB/s	531.8 MB/s

Reads reach about 450 MB/s at 8 streams and don't grow further at 16 (writes are still growing
at 16, 531.8 MB/s). What follows covers the read plateau. 450 MB/s is 3.6 Gbps, under the
single-flow ceiling
(roughly 5 Gbps per network flow).
So the single-flow ceiling isn't what's binding.

The plateau sits at efs-proxy's CPU. While reading at 16 streams, CPU usage on the
efs-proxy process was 67.3% + 18.1% + 14.9% (an 8-vCPU host).

This is where the meaning of not supporting nconnect becomes clear. What nconnect increases
is the connection count of the NFS mount — between the NFS client and efs-proxy. Since the
plateau is efs-proxy's own CPU, even if it were supported, it wouldn't reach where the plateau
actually is.

So it isn't "slow because nconnect isn't supported." Since the bottleneck is a process's CPU on
the client, the fix is a bigger client, not a mount option. efs-proxy --tls being on this path
is the cost of a configuration where in-transit encryption is the default, not a defect.

One practical note. Specifying nconnect hung the mount (90-second timeout). Not a rejection,
not silent ignoring. The first measurement was entirely lost to this. When trying an unsupported
option, wrap it with timeout.

Absence of decay over a 15-minute sustained write

Writes had been measured in a 120-second window, so I re-measured at 900 seconds, on the
second-generation environment.

Item	Value
Steady-state window average	2,063.00 MB/s (900 seconds after a 60-second warmup)
10-second interval min / max	1,749.0 / 2,128.4 MB/s
Cumulative written	Roughly 1.77 TiB
Decay	Not observed. The first minute and the last minute sit in the same range

Absorption somewhere can be ruled out. In-memory cache is 256 GB and NVMe cache is disabled;
the amount written is 7x that. Volume efficiency was disabled, with 0 savings.

Stating the relationship to the published value directly: the documentation says for
second-generation, "reads get the full throughput capacity, writes get one-third of it," and
it separately names 6,144 MBps as an exception to that general rule, listing it in a separate
table with Single-AZ writes at 1,024 MBps. So the published value that should apply to a 6,144
configuration is 1,024 MBps. The measurement was 2.05x that.

This single point reproduced on a separate file system later. Same 6,144 MBps provisioned
value, 1,800 seconds (twice this measurement's window), 2,097 MB/s, a 0.3% difference between
the first 300 seconds and the last 300. Different day, different file system, different window
length: 2,063 and 2,097, a 1.6% difference. The no-decay conclusion is supported by two
observations.

Lowering the provisioned value and measuring again still exceeds it in the same direction.

Provisioned	Published value that should apply	Measured (sustained, median)	Against the published value
1,536	One-third of the general rule -> 512 MB/s	1,333 MB/s	2.60x
6,144	Exception table's 1,024 MB/s	2,097 MB/s	2.05x

Both points exceed the published write ceiling. Same direction, ratios of 2.60 and 2.05.

This used to be described a different way. It previously read "the general rule holds at
6,144 and misses at 1,536," but 6,144 / 3 = 2,048 being close to the measured 2,097 is a
coincidence. The documentation names 6,144 as an exception to the general rule, so applying
the general rule there doesn't hold in the first place. Corrected.

Which figure refers to what still isn't determined. The table header reads "Maximum
throughput from SSD storage," which may refer to a ceiling scoped to the SSD path only.
But this measurement closed off the likely absorption points — the cumulative 1.77 TiB is
7x the 256 GB in-memory cache, the NVMe cache was confirmed disabled on both nodes, efficiency
was disabled with 0 savings. Why it still comes out roughly 2x can't be explained from this
side.

The design-level effect is over-, not under-provisioning. Working backward from the general
rule to hit a write of 1,333 MB/s looks like it needs a 4,000 MBps-class provisioned value, but
1,536 was measured to be enough. The second-generation provisioned rate is $2.013/MBps-month
(ap-northeast-1, fetched 2026-09-04), so this misreading translates directly into overbuying.

Only two points were measured: Single-AZ, 1 HA pair. Multi-AZ, 2+ pairs, and provisioned values
other than 1,536 and 6,144 weren't measured
(measurement record)
(Japanese).

Capacity growth from continued writing

Trying to continue into a read-side measurement after this, it didn't complete. The cause is a
documented specification — a gap in my own design, not a documentation error.

Before starting, snapshot count was 0 and the snapshot policy was none. 30 minutes into the run, a
snapshot named starting with backup- appeared, holding 739 GiB, and the write rate dropped from
2,200 MB/s to 267 MB/s. Volume usage went from 88% to 100%, with 9.9 GiB free. Deleting files
with rm didn't return the free space.

The cause is the automatic daily backup.
Volume backups state that a
backup first takes a volume snapshot, and that snapshot sits inside the volume, consuming
capacity, and stays until the next backup. For an overwrite-heavy workload, the pre-overwrite
blocks are retained, so the amount written becomes capacity as-is.

Why it doesn't return on deletion is
also documented.
Data deleted from the active file system is not released as long as a snapshot still references
it.

Checking for snapshots before starting doesn't prevent this. Before the run it reads normally,
because no snapshot exists yet. There was a recovery mechanism, though.
It's documented
that volume autosize and Snapshot autodelete are designed to work together. Neither was active
by default in this setup. The actual situation was that the default was in effect because
neither was specified in the template. "Not specified" isn't "disabled."

15 minutes over SMB

No decay over SMB either. The 900-second steady-state window averaged 1,488.03 MB/s, with a
10-second-interval min of 1,452.70 and max of 1,514.40; the first-third average was 1,487.55
against a last-third of 1,486.57.

Changing the client from an 8-vCPU type to a 36-vCPU type only moved it 0.6% (1,478.95 versus
1,488.03). This value isn't determined on the client side.

The initial observation that a 300-second measurement (1,698.42 MB/s) was 12.4% higher than the
900-second one doesn't reproduce (re-measured 2026-09-20). Varying only window length at 6,144
MBps, the longer window actually came out 4.0% higher (1,375.4 versus 1,430.3 MB/s). Channel count
stayed constant regardless of measurement duration, and even writes moved only 1.9% across window
lengths. The original 12.4% gap looks like a one-off value, not a property of window length.
That said, the longer window's vdbench format was interrupted partway through, so it wasn't a
comparable pair
(re-measurement record)
(Japanese).

Against the same physical port on the same node of the same file system, NFS measured 2,063.00
MB/s and SMB 1,488.03 MB/s. Since the source of concurrency differs (16 connections specified by
the client versus 4 negotiated channels), this can't be attributed to a difference in the
protocols themselves.

Conditions and the 10-second-interval progression are in
the SMB sustained write
(Japanese).

Additional read capacity via FlexCache

One more thing was measured on the read side. This section covers the figure's panel A, where
Origin and Cache sit as separate file systems. FlexCache isn't a mechanism for sharing the
origin's (the replication source volume's) performance — it's a mechanism that adds separate
capacity on the read side.

Measurement point	Measured, uncompressible
FlexCache first read (cache not yet filled; read while filling)	306.8 MB/s
FlexCache resident (second read onward; cache already filled)	882.7 / 907.1 MB/s
Direct origin read (control — the comparison baseline; already read once and cached)	383.3 MB/s

Resident is 2.31x a direct origin read. The cache is a separate file system from the origin,
with its own throughput capacity, its own memory, and its own burst allowance. So while reading
from the cache, the origin's own ceiling doesn't apply.

The cost is the first read. Data that's never been read comes in at 306.8 the first time — 1/2.88
of resident. That's the figure to design around.

One qualification. This was measured with both origin and cache at 128 MBps. Raising the
origin to 2048 MBps puts a direct origin read into the 2000 MB/s class, so depending on
configuration, the 2.31x could reverse. Unmeasured.

Bottleneck shift with small objects

Raising throughput capacity 16x and SSD IOPS 13x didn't raise request rate. The six measured
points were 4 KiB reads and writes at concurrency 64 and 256, and 64 KiB reads and writes at
concurrency 64. None of the six grew, and writes dropped 25%. Against 40,000 IOPS, 279-605
req/s — 0.7-1.5% of provisioned IOPS.

The bottleneck sits on the S3 Access Point's request-handling side. For designs that handle a
large volume of small objects, raising throughput capacity and SSD IOPS doesn't help.

Per-tool visibility

What's available to check "what is this actually limited by" while measuring differs by path.
Not knowing this means missing the numbers needed to identify a cause.

What you want to see	Reaches it	Doesn't reach it
Per-volume throughput	ONTAP's volume counters	Client-side measurement (turns into a single-TCP-flow ceiling)
Node CPU	CloudWatch (`AWS/FSx`)	`fsxadmin`'s REST doesn't reach it
Fraction that went to disk	CloudWatch's `DiskReadBytes` / `DataReadBytes`	Can't be determined from the client side
S3 Files propagation lag	CloudWatch (`AWS/S3/Files`'s `ExportAge`)	—

Measuring from the client side turns into roughly 590 MB/s. Linux's NFS mount opens only one
TCP connection per server by default, and that's what limits it first. If you want to measure
the storage-side ceiling, read the storage-side counters.

CPU on both paths was around 22%. The NFS path delivers twice this architecture's throughput at
the same CPU level. So this architecture's cost isn't inside ONTAP — it's upstream, on the S3
translation side.

And on reads, DiskReadBytes / DataReadBytes was 1.4 / 0.9 / 1.5%. 98.5-99.9% of read bytes
didn't go to disk. That means it came back from cache, which links directly to Part 5's fifth
default.

AWS documentation notes and support inquiries

This article touches three points in the published specs. Two were filed on 2026-09-17. The
source for all three is
the performance specs.

Content	Type	Status
Second-generation writes exceed the published ceiling at both points measured (2.60x the general rule's 512 at a 1,536 provisioned value; 2.05x the exception table's 1,024 at a 6,144 provisioned value)	Gap between published value and measurement. Likely absorption points (two-tier cache, efficiency) closed off	Filed (2026-09-17). AWS's reply (2026-09-24): the "up to a third" figure is not a hard limit but a sizing indicator. A write uses bandwidth twice over — once to the active server, once to sync the standby server — but network burst capacity can push measured throughput above a third of the provisioned value, and AWS considers that expected behavior, not an anomaly. The table heading's "Maximum" is described as the ceiling to expect once burst is not a factor and the same file system is shared with other workloads. For sizing, AWS pointed back to the original rule of thumb: provision three times the required write throughput (e.g., a 1,333 MB/s requirement calls for a 6,144 MBps configuration)
ap-northeast-1's NVMe read cache capacity is missing from the table (the four regions with a column have it listed). The management page is also written scoped to "second-generation" only, so a first-generation user with a qualifying config can't tell they have a cache	Missing content. Presence was confirmable via ONTAP; capacity was not	Filed (2026-09-17). AWS's reply (2026-09-21): no documentation change. AWS's position is that a first-generation Single-AZ configuration at 2,048 MBps lacking a cache is not something the documentation promises against, and that enumerating every such corner case isn't practical, so the existing wording stands. Pointed instead to `system node external-cache show` for checking presence and capacity per file system. Both the capacity-publication ask and the management-page scope ask were effectively declined
The observation that raising SSD IOPS made reads 7.14x faster didn't square with "SSD IOPS is used only for reads not served from cache"	My own measurement gap. Hadn't read a volume that reliably exceeded the sum of both cache tiers	Not filed. Re-measuring 2026-09-17 with the working set at 2.15x the cache size squared with the documentation (ratio came out 4.86x; 7.14x doesn't reproduce). Not a documentation error

Why the third isn't filed. The mismatch's cause was on my side. Re-measuring settled it
(2026-09-17). With the working set at 512 GiB — 2.15x the in-memory cache — and the NVMe cache
disabled, 96-126% of read bytes came from disk, with the low end saturating at
DiskIopsUtilization 112%. This matches the documentation. Not a documentation error, so not
filed. Only the ratio changed — 4.86x, and 7.14x doesn't reproduce. And what's capping the
high side, roughly 1,250 MB/s, remains unresolved — that isn't a finding against the published
spec, it's a range I haven't measured.

The first and second were filed. Both were filed without me deciding "the correct value is
this" — as a gap or omission for whoever holds the values to judge. The first was framed as the
question "what does 'Maximum throughput from SSD storage (write)' actually bound?" and the second
as two points: publishing the capacity, and the management page's stated scope.

The first used to be described a different way in this article. It previously read "which of
the general rule and the exception table applies flips depending on the value," but the
documentation names 6,144 as an exception to the general rule, so applying the general rule
there doesn't hold in the first place. Corrected before filing.

Both first replies came in on 2026-09-21. The first stopped at confirming AWS's own measurement
clears a third of the provisioned value; the second landed on a decision not to change the
documentation. The underlying fact — a first-generation Single-AZ configuration at 2,048 MBps can
lack a cache — is something the documentation genuinely never promised against, and not
enumerating every corner case is a reasonable line to draw. What didn't go through, though, is
separate from that: asking for the capacity to be published, and asking the management page's
stated scope to cover first generation too, are both requests that a documented gap doesn't
answer by itself. Recording that they didn't land. The per-file-system way to check
(system node external-cache show) is documented, so checking it yourself is unaffected.

The first one got a follow-up on 2026-09-24. The "up to a third" figure turned out to be a
sizing indicator rather than a hard limit — a write spends bandwidth twice over (once to the
active server, once to sync the standby), and network burst capacity can push it above a third of
the provisioned value, which AWS considers expected rather than anomalous. The table heading's
"Maximum" is described as the ceiling to expect once burst isn't a factor and the file system is
shared with other workloads. For sizing, AWS pointed back to the original rule of thumb: provision
three times the required write throughput. This explanation doesn't dispute that the
measurement exceeded the published value — it only says how to read that published value.

The block protocol side's findings and filing status can be traced from the
separate article.
Submission status across the three block-protocol articles and this article is collected in
AWS documentation notes and inquiries across the block-protocol articles.

What hasn't been measured yet

Meaning not yet measured, not "can't be."

Reading more than 2x the cache size in one pass. At 512 GiB (2.15x the cache), 4.86x (256.47 -> 1,246.67 MB/s). The low side saturating on SSD IOPS is confirmed. What was capping the high side, around 1,250 MB/s, turned out to be transfer size and concurrency, not a resource (details under a single-client figure that was actually the client's own limit above)
Effect of measurement duration. Measurement duration itself is not the cause. What carries the gap is provisioned IOPS, payload compressibility, and unit-to-unit variance (measurement) (Japanese)
Measuring FlexCache uncompressed on a 2048 MBps configuration. The "2.31x" could reverse depending on configuration
What's capping the roughly 300 MB/s on the 128 MBps configuration. Reads on the same configuration converge to 297-317 MB/s for both warm and cold (values from the same measurement set as this article, not shown in the body table). This used to be described as "consistent with plateauing on a burst," but reading burst remainders ruled that out. The remainder plateaued at 99 -> 95% and returned to 99% after the load stopped — not exhausted. Against a disk burst ceiling of 600 MBps, the measured 254 MB/s is 42%; against a network burst ceiling of 1,250 MBps, roughly 285 MB/s is 23% — neither ceiling was reached. It isn't the roughly 590 MB/s single-flow ceiling either. Where it sits is unresolved (measurement) (Japanese). Note that DiskReadBytes / DataReadBytes in the same window was 87-97%, so reads on this configuration are coming from disk — a contrast with the 2048 MBps configuration's 1.4%, consistent with an in-memory cache of only 16 GB
The host count at which Amazon S3's total stops being proportional to host count. It's still proportional at 8 hosts / 38.3 Gbps
Adding S3 Files proxies. A configuration with separate mounts running more efs-proxy instances hasn't been tested
More than 8 hosts. The side sharing the same file dropped to a 1.12x increment at 8 hosts, so where it stops hasn't been observed
A point below 32 connections on non-overlapping ranges. It was already saturated at 32, so the connection count where saturation begins hasn't been bracketed
A sustained write longer than 900 seconds. No decay over 15 minutes, but nothing beyond that has been checked
Reads with the fraction served from cache varied. Capacity filled up and the run was aborted, so none of the 7 planned points were captured
Time-of-day effects. Two different days were checked, but not different times of day
SMB's 7 scenarios under cache control. Stopped by fill, not capacity. format=yes ignores the run block's xfersize and threads, writing at 128 KiB and queue depth 2, so overwriting the same file that otherwise reaches 1,488 MB/s only produced 165 MB/s (315 MB/s with formatxfersize=1m; queue depth stayed at 2). The parameter file points 8 SDs at a single file; splitting each range into its own file looks likely to solve both the speed and the design problem at once. This is a prediction, not a result. Background in two places stuck under cache control (Japanese)
Why SMB's 300-second and 900-second values differ. Doesn't reproduce. Re-measuring at 6,144 MBps removes the window-length difference; the longer window's vdbench run had also been interrupted, so it wasn't a comparable pair in the first place (measurement) (Japanese)
Conditions for raising Multichannel's channel count above 4. Setting max_connections_per_session to 32 still produced 4. The 4 comes from the client side's ConnectionCountPerRssNetworkInterface, which determines channel count per RSS-capable NIC. The server side's 32 is a per-session ceiling, not the limiter on the increasing side. Adding NICs or raising this value hasn't been tested
The discrepancy between server-side counters and the client-side measurement tool. The cause was measurement method — aligning the window's start to a minute boundary and comparing only the minutes fully contained in the window, the two systems agree within 0.2% (measurement) (Japanese)
What the 4 KiB random read ceiling actually is. It's on the client side. What matters is the number of distinct IPs the client talks to and the client count, not thread count. When citing this figure, write "per client" — it isn't the file system's ceiling (measurement) (Japanese)
Why 4 KiB random reads plateau at 47.5% of provisioned SSD IOPS. The ratio doesn't reproduce.
Smaller configurations are disk-IOPS-limited; larger ones follow a different formula.
So "47.5% of provisioned" is a value specific to that one configuration, not an
invariant with scale. When citing it, always state the 6,144 MBps / 200,000 IOPS condition
alongside it
(measurement)
(Japanese)
Metadata-operation performance under a high file count (creating, enumerating, and stat-ing a
large number of files). What this article measured is throughput, IOPS, and concurrency, not
metadata operations when the file count is large. NetApp's High File Count NAS Workloads
Technical Report suggests that inode capacity and a single directory's maximum size bear on
enumeration cost, but there's no figure measured on this architecture (S3 API collect -> origin
-> FlexCache distribution). Measuring it would need an origin volume holding a sufficient file
count and a measurement setup for metadata operations over it
The time taken to enumerate a directory holding a large number of files (ListObjectsV2, or an
NFS / SMB directory walk). The enumeration cost when files concentrate in a single directory
is something this project hasn't measured on this architecture
FlexCache's metadata-replication cost under a high file count. How the cost of replicating
metadata from origin to cache scales with file count is something this project hasn't measured
on this architecture. It's a different axis from this article's throughput measurements

Closing

Measuring three paths from the same host, the bottleneck sat in three different places for all
three. A throughput capacity provisioned on the file system, client-side network bandwidth, and
the CPU of a process on the client. This difference doesn't close as you add more clients.

And on the FSx for ONTAP side, there wasn't one ceiling. With the same file system and the same
connection count, the total swings 5.5x depending only on whether each EC2 client's read range overlaps.

Which one to choose comes down to where you place the read side, which side holds the primary
copy, and when you want a write visible on the bucket side. Throughput numbers are one input into
that decision, and they come later in the order of things to check.

What's worth taking away is a procedure: check the type of ceiling before comparing
throughput. Whether it's a provisioned value, client-side bandwidth, or a process's CPU on the
client. Line up numbers of different types in the same column, and you get a table shaped like a
comparison without being one.

And a value measured with a single EC2 client can't determine that type. On the same file system,
adding a second EC2 client grew the total 1.59x, and fixing transfer size and concurrency while
staying at one client grew it 1.38x. Either way arrives at the same ceiling. There are two ways
to check the type, and neither costs much extra.

A measurement that reproduces within 0.02-0.14% under identical conditions can still swing 30%
from a single misunderstanding about conditions. "Stable" isn't "correct." And this article's
figures are for one specific throughput capacity and one specific payload. They don't transfer
directly as your environment's numbers. What transfers is the measurement approach.

Hope this helps anyone running the same kind of measurement.

Defaults to check before measuring and items to always record are collected in
Performance testing considerations.

Until next time.

Amazon FSx for NetApp ONTAP NVMe/TCP assumes RHEL 9.3: why multipathing fails on Amazon Linux 2023

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 30 Sep 2026 05:42:42 +0000

Amazon FSx for NetApp ONTAP's NVMe/TCP procedure assumes RHEL 9.3: why multipathing is unavailable on Amazon Linux 2023, and how to check before you are billed

Followed the AWS procedure and run cat /sys/module/nvme_core/parameters/multipath only to find
the file isn't there?

Moving your target Linux distribution to Amazon Linux 2023 can stop a documented command from
working exactly as written. You are reading the procedure correctly, yet your environment alone
gives a different result — and tracking the cause down to a kernel build setting takes time.
This article is for anyone who got stuck at that point.

What this article does and does not cover: it covers how to confirm multipathing is unusable on
AL2023, and how to count whether it is actually using the second path. It does not cover
performance comparisons or a generalized conclusion for distributions other than AL2023 (details
under "What this article does not cover" below). All figures are from a Single-AZ configuration;
Multi-AZ is unverified.

dev.to series: FSx for ONTAP Block Protocols, Measured.

What was checked

Assuming no prior knowledge of FSx for ONTAP, here's what this article is confirming.

Location: a single AWS VPC, single AZ
Client: one EC2 instance (Amazon Linux 2023)
Target: one Amazon FSx for NetApp ONTAP file system (second-generation SINGLE_AZ_2). The HA pair's two controllers hold two paths to one namespace
- Path 1: to the optimized controller (shortest path)
- Path 2: to the non-optimized controller. Only when multipathing is available can the host tell that path 2 is a standby

the shortest route) and Path 2 (non-optimized, which looks like a normal route without ANA), both
reaching the same one namespace" width="800" height="682">

Figure: an FSx for ONTAP HA pair has two paths to one namespace, and **Asymmetric Namespace
Access is the mechanism that tells the host which one is the shortest path* (referred to as
multipathing throughout this article). Without this mechanism in the kernel, those two paths
don't appear as "one shortest path plus one standby": they appear as two separate devices
pointing at the same data.*

Item	Role in this article
VPC / AZ	Single AZ, so both paths to the HA pair's controllers sit on the same network
EC2 client	Checks for multipathing and the kernel configuration behind it
optimized path	The shortest path as determined by multipathing. Real data is expected to flow here
non-optimized path	The failover path. Without multipathing it appears as an ordinary path too

AWS's
Provisioning NVMe/TCP for Linux
includes a step that confirms multipath has come up: the fourth item under "To discover the target
NVMe nodes". It says that a returned Y means success (wording as of 2026-09-17; every
reference to that page below was checked on the same day).

cat /sys/module/nvme_core/parameters/multipath
# Y means success

On the Amazon Linux 2023 AMIs I used, that file did not exist. The kernel does not have
Asymmetric Namespace Access (a mechanism that tells the host which of several paths to the same
namespace is the shortest one) in it. The range I checked is the three kernel series returned by
al2023-ami-kernel-default-x86_64, so I cannot write "AL2023 always behaves this way". Checking
takes one command, and a later section gives it.

The procedure is written for RHEL 9.3. The same page's "Before you begin" says
Create an EC2 instance running Red Hat Enterprise Linux (RHEL) 9.3. Immediately after that there
is a statement about other distributions. In summary: on AMIs other than RHEL 9.3 some of the
utilities may already be present or the install command may differ, but apart from installing
packages, the commands in this section are valid on other EC2 Linux AMIs too (the original wording
is at the end of "Before you begin" on that page).

What this article reports is two counterexamples to that statement. Neither of them is
explained by package installation.

Command from the documentation	Result on AL2023	A packaging problem?
`cat /etc/nvme/hostnqn`	File does not exist	No. `nvme-cli` is installed; whether it creates this file differs by distribution
`cat /sys/module/nvme_core/parameters/multipath`	File does not exist	No. This is a kernel build option (`CONFIG_NVME_MULTIPATH`) and a package cannot change it

You can check this before any billing starts. That is the most practical part of this article,
I think.

This is not a claim that "the documentation is wrong". It is a report that these two do not
fall inside the stated range (apart from installing packages, valid on other AMIs too).
The state of my feedback to AWS is at the end of this article.

Then, after bringing in a kernel that does have multipathing, I found that the optimized /
non-optimized output of nvme list-subsys says "there are two paths", not "two paths are in
use". Whether they are in use has to be counted separately, and there is a trap in choosing
which table to count.

What this article does not cover. It does not compare performance. The figures here exist to
show that two clients agreed under the same conditions, and they cannot be used for sizing. What
moves the numbers is in a separate article. All figures are from a
Single-AZ configuration; Multi-AZ is unverified.

This is one of three, but there is no reading order. The block-protocol measurements are split
by reader.

Article Reader

A Session and queue counts Anyone about to mount

B Checking whether multipathing is available (this article) Anyone who followed the procedure and got stuck

C What moves the numbers Anyone measuring or citing figures

None of them assumes the other two. Read only the part you need.

	Article	Reader
A	Session and queue counts	Anyone about to mount
B	Checking whether multipathing is available (this article)	Anyone who followed the procedure and got stuck
C	What moves the numbers	Anyone measuring or citing figures

The environment I measured

Item	Value
File system	Amazon FSx for NetApp ONTAP, second generation SINGLE_AZ_2, 6,144 MBps × 1 HA pair, 4,096 GiB SSD, 200,000 provisioned IOPS
Region	ap-northeast-1, single AZ
ONTAP	9.18.1 (the reads and writes in the agreement table). For the 12 sequential-write points I failed to record the version. `FileSystemTypeVersion` returned `null` and I tore the environment down before re-reading it from the cluster API. The kernel investigation also spans deployments on other versions. The cross-article version mapping is in the ONTAP version matrix
NVMe read cache	Disabled (confirmed on both nodes)
LUN / namespace	600 GiB, on a 900 GiB volume
Before measuring	Wrote the full 600 GiB once. Unwritten blocks on a thin-provisioned volume return zeros, so reading without writing first measures nothing
Client	c5n.9xlarge; 50 Gbps network is a guaranteed figure
Instrument	VDBENCH 5.04.07 directly, `iorate=max` (unlimited), 512 threads, 60 s warm-up + 300 s measurement
Test data / efficiency settings	Not recorded whether the VDBENCH payload was compressible, or whether volume inline efficiency (`StorageEfficiencyEnabled`) was enabled. On the file-protocol side, efficiency settings measurably affected figures; whether the same applies to block is unconfirmed

The window is 300 seconds, so the figures in this article include burst. They cannot be cited as a
baseline.

The figures in this article are iorate=max saturation points, not operating points. No target
IOPS was given; they are what the system took when everything available was pushed at it. On the
file-protocol side of the same repository, giving a target of 4,400 IOPS produced 4,406 MB/s at
8.86 ms, while unlimited produced 4,204 MB/s at 121.79 ms. Unlimited was 5% lower in
throughput and 14 times higher in response time.

With 512 threads fixed and no rate limit, response time becomes the identity 512 ÷ IOPS
(derived 216.6 ms against measured 216.588 ms in a separate run). The figures below can be read
through that identity: 1,277.48 MB/s is about 401 ms, and 131,736.5 IOPS is about
3.9 ms. Both are derived values and were not recorded as measurements.

For this article's purpose, being a saturation point is not a disadvantage. The figures are
here to show that Rocky and RHEL agreed under the same conditions, and as long as both are
saturated the same way, the agreement test holds. Those values still cannot be used for
sizing.

Behaviour on this AMI

On this AMI the file named at the top does not exist. Reading the kernel configuration gives
# CONFIG_NVME_MULTIPATH is not set, and modinfo nvme_core has no multipath parameter either.
So multipathing is not in the kernel.

As a result the two controllers are not merged into one device; they appear as separate devices
pointing at the same namespace UUID (/dev/nvme2n1 and /dev/nvme3n1). nvme list-subsys shows
no optimized / non-optimized either.

The AWS procedure is written for RHEL 9.3, where it is present.

Please do not generalise this to "multipathing is not available on AL2023". What I observed were the
AMIs that the al2023-ami-kernel-default-x86_64 SSM parameter returned on 2026-09-12 and
09-13, and I did not record the kernel versions. The template said "record the kernel version
at measurement time" in place of pinning the AMI, and the block phase did not implement that.
That is a gap in my records.

Checking in your own environment is more reliable, and it is one command.
grep -i NVME_MULTIPATH /boot/config-$(uname -r)

I later widened the observation to three kernel series. All three returned by
al2023-ami-kernel-default-x86_64 (6.1.186 / 6.12.103 / 6.18.48) gave
# CONFIG_NVME_MULTIPATH is not set. You can check this before creating a file system, by
extracting the config from the kernel package in the repository. It is known before billing
starts, so checking in that order is cheaper.

To measure multipathing I then brought in another distribution. The AWS procedure is written for RHEL 9.3,
so I put both Rocky Linux 9.7, a rebuild, and RHEL 9.7 itself on the same namespace and compared
them.

Workload	RHEL 9.7	Rocky 9.7
1 MiB sequential read	1,277.48 MB/s	1,276.60–1,279.22 MB/s
4 KiB random read	131,736.5 IOPS	131,550.6 / 131,749.4 IOPS
4 KiB random write	195,439.8 IOPS	194,960.6 IOPS

Only the reads agreed in the same state (within 0.1%). Same minor version, same nvme-cli
(2.16-1.el9), same cold state. The kernels differ in z-stream (5.14.0-611.55.1 against 611.5.1).
For reads, there is now one piece of grounds for reading a rebuild's figure as the figure for the
product itself.

Both write workloads had the Rocky side measured warm, so the states are not aligned. Counting
the 0.3% difference on 4 KiB random write as "agreement" was my error, and the note in that same
table contradicted my own conclusion (corrected 2026-09-20). For that shape I can say neither
that it holds nor that it does not.

Sequential write was re-measured later, in the same state, for that shape alone. It does not
agree (2026-09-17). On a different file system (second generation, 1,536 MBps, 50,000 provisioned
IOPS), same namespace, same state, same parameters, six runs each.

Client Min Median Max Spread

Rocky 9.7 842.70 1,040.26 1,092.50 29.6%

RHEL 9.7 1,105.75 1,112.65 1,129.30 2.1%

Units MB/s (1 MiB sequential write, 8 threads, o_direct, 120-second window). These 12
points were taken under conditions that differ from the environment described at the top: a
different file system (second generation, 1,536 MBps, 50,000 provisioned IOPS), 8 threads rather
than 512, and 120 seconds rather than 300. The 512 ÷ IOPS reading given at the top cannot be
applied to these 12 points. RHEL's minimum is above Rocky's maximum: six against six, with no
overlap in range. RHEL's median is 7.0% above Rocky's, and the asymmetry in spread is larger than
the difference in medians.

So "a rebuild's figure can be read as the figure for the product itself" is limited to the
reads. When citing a sequential-write figure, say which of the two it was measured on.
4 KiB random write was not re-measured in the same state, so for that shape I can say neither
that it holds nor that it does not. I cannot state a cause. The only difference on record is
the kernel z-stream, and I did not vary it on its own. During the runs the file system side was at
72–81% disk throughput, 16–19% IOPS and 31–38% network, none of them saturated (so this is not a
case of the two meeting at a ceiling). For these 12 points I failed to record the ONTAP version.
FileSystemTypeVersion returned null and I tore the environment down before re-reading it from
the cluster API.

`iopolicy` on a kernel with multipathing, and the value the procedure states

This is the third counterexample. iopolicy (Linux's multipath setting that decides how I/O
is distributed across multiple paths to an NVMe device: round-robin cycles through them,
queue-depth sends to whichever has the shortest queue) is what step 5 of the same procedure
says to confirm as round-robin (and the example nvme list-subsys output at step 7 also shows
iopolicy=round-robin).

	Value
Value the documentation says to confirm	`round-robin`
Measured on Rocky Linux 9.7 / RHEL 9.7	`queue-depth`
What set it	the udev rule `71-nvmf-netapp.rules` shipped by `nvme-cli` (2.16-1.el9)

The kernel default is numa, neither round-robin nor queue-depth. It was queue-depth in
this environment because the NetApp-oriented udev rule bundled with the nvme-cli package set it.
Following the confirmation step returns a value other than the expected one.

I measured the effect on performance. It is negligible: switching policy mid-run produced no
step change, and the difference was 0.5%. The figures and the method are in
a separate article (C). So this is not a story about round-robin being
required for speed. The only part worth reporting is that the confirmation step returns a value
that differs from the documentation.

This can change with the nvme-cli version. What I observed here is one version, 2.16-1.el9,
and I did not confirm whether other versions ship the same udev rule.

How to check whether multipathing is using the second path

This section was added afterwards. It became necessary in order to do the analysis above, and it
is also the reason I once published a wrong conclusion.

nvme list-subsys shows the two controllers as optimized / non-optimized, and prints iopolicy
as well. That is a display of "there are two paths", not of "two paths are in use".

At first I varied iopolicy three ways, compared total throughput, saw that only queue-depth was
faster, and wrote that "only queue-depth uses the second path". That was wrong. I had not
counted bytes per path. A client-side total that does not increase is consistent with two different
facts: "the second path is not being used", and "it is being used but the ceiling is nearer than
the path".

Bytes per path can be read from two places

On the ONTAP side they can be read from a counter table. There is a trap here.

Table	Result in this configuration
`nvmf_lif`	Zero rows. Still empty after writing 600 GiB over NVMe/TCP
`lif`	Reports 0 bytes for the LIF in question. It does not count NVMe-oF
`nvmf_tcp_port`	Has the real data. `read_data` / `write_data` / `total_ops` per LIF

"The table exists but has no rows" and "this version does not have that table" add up to the same
appearance. Return 0 and move on, and you reach the correct conclusion, "the second path saw
0 bytes", for the wrong reason.

It can also be counted on the client side. NVMe/TCP controllers have different destination
addresses, so aggregating established sockets by destination separates the paths. That is a second
source, independent of the ONTAP side.

ss -tin state established | awk '/:4420/{...}'   # sum bytes_sent / bytes_received per destination

Counted properly, the second path was not running

I ran four workloads under queue-depth and took before/after differences.

Path	Read bytes	Write bytes	Total ops
optimized	+about 1,012 GiB	+about 730 GiB	641,908 → 120,008,866
non-optimized	±0	±0	4 → 4

Neither reads nor writes went to the second path. The ONTAP side and the client side agreed, and
it was the same on Rocky and on RHEL (the second path's increment across all workloads was about
20 KB).

Further, switching iopolicy within a single run produces no step change. With a sequential read
running I changed queue-depth → numa → queue-depth, and the values sampled at one-second
intervals stayed inside the same 1,150–1,400 MB/s range both before and after each switch.

Summary

All three kernel series returned by al2023-ami-kernel-default-x86_64 (6.1.186 / 6.12.103 / 6.18.48) had CONFIG_NVME_MULTIPATH unset. Measuring multipathing requires a different distribution. Asking AWS why AL2023 is excluded from the procedure (reply 2026-09-21) surfaced a likely reason: nvme connect-all establishes both paths and auto-discovers the alternate LIF even when only one is given, but without CONFIG_NVME_MULTIPATH the kernel never merges them into one namespace device. The procedure's later steps assume that merge already happened, so proceeding unmerged leaves the other path writable against the same namespace with no multipath software mediating it: a data-corruption risk
You can check before creating a file system. Extract the config from the kernel package in the repository and you know before billing starts. On a running instance it is the one command grep -i NVME_MULTIPATH /boot/config-$(uname -r)
Rocky Linux 9.7's figures can be read as RHEL 9.7's figures, but only for the reads. On the same namespace in the same state they agreed within 0.1%. Both write workloads had the Rocky side measured warm, so the states are not aligned. Counting the 0.3% difference on 4 KiB random write as agreement was my error, and for that shape I can say neither that it holds nor that it does not (corrected 2026-09-20). Sequential write does not agree: re-measured later in the same state, six runs each, RHEL's median was 7.0% higher and RHEL's minimum was above Rocky's maximum (spread 29.6% for Rocky against 2.1% for RHEL). When citing, say which of the two it was measured on
optimized / non-optimized in nvme list-subsys is a display of path existence. For whether they are in use, count with the nvmf_tcp_port counters. nvmf_lif returns no rows, and lif does not count NVMe-oF
In this configuration, not one byte went to the second path even under queue-depth. Two places agreed (the ONTAP-side counters and the client-side sockets), and it was the same on Rocky and on RHEL

AWS documentation notes and support inquiries

This article touches three places in the documentation. All three were submitted on 2026-09-17.

The source for all of them is
Provisioning NVMe/TCP for Linux.

Remark	State
Against "apart from installing packages, valid on other EC2 Linux AMIs too": `cat /etc/nvme/hostnqn` finds no file on AL2023	Submitted (2026-09-17). Reproduced in the same environment, confirming the wording doesn't hold as-is for the AL2023 environment checked. The two proposed fixes (scoping the wording, adding a generation step) have been shared as an improvement request. Adoption, content, and timing are unconfirmed
Against the same statement: `cat /sys/module/nvme_core/parameters/multipath` finds no file on AL2023 (kernel build option)	Submitted (2026-09-17). The observation was confirmed and filed as documentation feedback. Fix timing undecided. A follow-up question ("why is AL2023 excluded from scope?") also got a technical explanation: proceeding with the paths unmerged leaves the other path writable against the same namespace with no multipath software mediating it, a data-corruption risk, so the exclusion has a technical rationale
Step 5 directs confirming that `iopolicy` is `round-robin`, but the measurement on Rocky 9.7 / RHEL 9.7 is `queue-depth`. `nvme-cli`'s version boundary is 2.11 -> 2.12 (see below)	Submitted (2026-09-17). On 2026-09-30 it was acknowledged that a note is needed stating whether `iopolicy` resolves to `round-robin` or `queue-depth` depends on the environment, and this was fed to the responsible team (with a note that NetApp's own published documentation carries the same point). The content and timing of any change are not disclosed in advance, so they are unconfirmed

The form I submitted was this. Verbatim quotation (which line of which step), reproduction steps,
expected and measured values, and the range I confirmed across three kernel series, written as
such. Not generalising to "multipathing is not available on AL2023" is the same in the feedback: the
observation is limited to the three series returned by al2023-ami-kernel-default-x86_64.
For the second one I also included the control under which the procedure does hold (on a kernel
with CONFIG_NVME_MULTIPATH=y, Y is returned). It is not a report that the procedure itself is
wrong.

The third is not submitted as a performance problem. The difference between policies is 0.5% in
this configuration, and the report is the single point that the confirmation step returns a value
differing from the documentation.

The third changed shape before submission, after I looked at the package side. Reading
71-nvmf-netapp.rules out of four builds of the public nvme-cli package (the SRPMs for 2.11 /
2.13 / 2.16, plus the installed 2.16-1.el9) showed that 2.11 has round-robin and 2.13 changed it
to queue-depth (confirmed 2026-09-17). At that point 2.12 had not been examined, so the boundary
could only be written as "somewhere between 2.11 and 2.13".

Submission status across the three block-protocol articles and S3 Burst Part 2 is collected in
AWS documentation notes and inquiries across the block-protocol articles.

A pointer to the upstream v2.11 and v2.12 sources followed, showing the boundary is v2.12.
Fetching and checking that commit (2026-09-19, 71-nvmf-netapp.rules.in) confirmed that
queue-depth was already in place as of v2.12. So the version boundary is settled at
2.11 -> 2.12 (the original "somewhere between 2.11 and 2.13" narrowed to 2.12, from both that
pointer and my own re-check). This is agreed to be a version drift rather than an error in the
documentation.

On 2026-09-30, it was acknowledged for this point that a note is needed stating that whether
iopolicy resolves to round-robin or queue-depth depends on the environment, and it was fed to
the responsible team.
NetApp's NVMe-oF procedure for RHEL 9.x
carries the same point (from RHEL 9.7 the default is queue-depth; on 9.6 round-robin is the
default with queue-depth selectable), consistent with this understanding. The content and timing
of any change are not disclosed in advance, so its reflection in the documentation is unconfirmed.

Primary record of the figures

The primary record of the figures and their conditions is in
the protocol measurement results,
and the reproduction steps are in
the block measurement runbook.
What is unconfirmed is written as unconfirmed there, so please do not drop that when citing.

How to check whether multipathing is available, counter-table traps, and iopolicy version differences are
collected in
Considerations when measuring FSx for ONTAP's block protocols.

AWS Transform Now Supports Block Storage Migration to FSx for ONTAP — Benefits and Pitfalls from a Hands-On EC2 Test

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 28 Sep 2026 13:42:13 +0000

Introduction

This is Part 2 of the series "The Data Foundation for AWS Modernization." Part 1 set out the lens: migration is not the goal but the entry point, and putting FSx for ONTAP at the data foundation lets you re-choose the compute. This part takes that entry point through AWS Transform, the core of modernization, and verifies hands-on that the data area lands on FSx for ONTAP as part of the migration on AWS. Migration from an on-premises VMware environment will come separately; the idea here is to start modernizing from the tightly coupled EC2 × EBS configuration by decoupling the data store.

Until now, when you migrated servers with AWS Transform (MGN), every disk was placed on Amazon EBS. If the source was not ONTAP and you wanted the data area on Amazon FSx for NetApp ONTAP (FSx for ONTAP below), you needed a "two-step migration" — migrate to EBS first, then move the data — or a third-party block storage migration tool.

With the update of 30 August 2026, FSx for ONTAP can now be chosen directly as the MGN target storage type.

AWS Transform announces general availability of Amazon FSx for NetApp ONTAP support: https://aws.amazon.com/about-aws/whats-new/2026/09/aws-transform-fsx-netapp-ontap-support/

That looked useful, so I tried it hands-on straight away.
The source in this test was not an on-premises VMware environment but an EC2 instance already on AWS (Amazon Linux 2023). Using that EC2 as the source, I ran the new path that migrates the data area (EBS) directly to an FSx for ONTAP iSCSI LUN.

Running it for real turned up several pitfalls that reading the documentation alone does not reveal.
Besides sorting out what this new feature can and cannot do, this article records the traps I actually stepped on during the test — errors hidden behind a successful job, and an unexpected capacity spike.

Here are the highlights up front, especially the traps to watch for.

Only data volumes can be targeted. Boot is always Amazon EBS.
What became GA is "the MGN target storage type". It is a different thing from presenting FSx for ONTAP as a VMware datastore (the Amazon EVS side).
In MGN's phases, "SNAPSHOT" was a volume Snapshot done as a metadata operation, and "LAUNCH" was the creation of a FlexClone (about 44 seconds for 8 GiB).
[Trap 1] Finalize is not just clean-up; it is the step where physical capacity temporarily peaks (about 2x).
[Trap 2] Even when the job status becomes COMPLETED, a failed final snapshot (fallback) or the creation of an unbootable target can be hiding behind it.

Test Assumptions and Scope

The test was run with the following environment and scope.
(Region: ap-northeast-1 / Source: Amazon EC2 (AL2023) / ONTAP: 9.18.1P3D1)

[What was run]
The whole flow: agent installation → replication → test launch → cutover → Finalize → teardown.

[Out of scope for this article (not verified)]

Migration with VMware as the source (no vCenter was prepared, so EC2 stood in as the source)
FSx for ONTAP as an Amazon EVS datastore (a different feature)
Durations at production-scale data volumes (the test used a tiny 16 GiB environment, dominated by fixed overhead)
Windows sources, and access over SMB / NFS
Post-migration optimization with lun move start

The Shape of the Configuration

Figure 1: three EC2 instances and where each volume goes: the boot column is EBS in all three rows, and only the data column changes to FSx for ONTAP from staging on (dark theme)

Figure 2: the control path, from AWS Transform to the ONTAP management endpoint (dark theme)

The point is that the boot area and the data area go to different places. The boot column stays Amazon EBS in all three rows, and only the data column changes to FSx for ONTAP from the staging row on. The target instance boots from EBS and receives the data over iSCSI. The source in this test was EC2, but the lower two rows of the configuration are the same whether the source is a VMware datastore or on-premises block storage.

For the control path in Figure 2, the official blog says a PrivateLink connection is established automatically, but in practice a Network Load Balancer (NLB) and a VPC endpoint service are created inside your VPC. They incur hourly and LCU charges.

Layer	Normal operation (replicating)	After cutover
Source EC2	Running. The agent keeps sending blocks	Can be stopped
Replication server EC2	MGN launches it automatically	Terminated by Finalize
Amazon EBS	Boot staging	Becomes the target's root volume
FSx for ONTAP	Staging FlexVol and LUN	Target FlexVol (FlexClone)
Network Load Balancer	Running	Remains after Finalize (manual deletion needed)

Table 1: each layer after cutover: the one that needs manual deletion is the Network Load Balancer

The Most Important Distinction: What Became GA Is the MGN Target

Mix this up and you will evaluate the wrong product. It is easily confused with "FSx for ONTAP can be used as a VMware datastore".

Question	Current support
Can FSx for ONTAP be used as the MGN target storage type?	Yes (GA with this release)
Is FSx for ONTAP presented as a VMware datastore?	No. The datastore is a separate feature on the Amazon EVS side
Can the boot disk also go on FSx for ONTAP?	No. Boot is always Amazon EBS
Can EBS and FSx for ONTAP be mixed within one server?	No. All data volumes use the same type
Can you migrate agentlessly?	No. Agent-based replication only
Which protocol do clients connect with?	iSCSI. The data is placed as LUNs inside a FlexVol

Table 2: questions and current support: the agentless path is not supported

The Known limitations in the MGN documentation state Agent-based replication only. If you were planning around vCenter's agentless path, the plan has to change. The path depends on whether you leave VMs for EC2, or keep the VMs and move only the storage out.

Figure 3: the FSx for ONTAP settings in the replication template. Only the SVM ID and the Secret ARN are specified, and the note states that boot goes to EBS

Environments This Configuration Suits

Whether a migration fits depends on the shape of the workload, not on the product of the source storage.

The boot / root area and the data area are separate, and the data area is on block storage
You want capabilities such as Snapshot and thin provisioning on the data area after migration
You want to turn "migrate, then move the data" into a single step
You already run ONTAP and want the same mechanisms (Snapshot / FlexClone / SnapMirror) after migration

Most of these apply whatever the source storage is. Even when the source is ONTAP, however, permissions, quotas and Snapshot policies are not migrated and need to be set up again.

Environments Where Another Option Fits Better for Now

Boot and data sit on the same volume
You cannot install an agent (agentless is not supported)
The source is Windows and you want to move the data as SMB shares
File-level migration is enough and block-level replication is not needed (a file transfer such as AWS DataSync is sufficient)
The source is ONTAP and you want to take the volumes as they are (SnapMirror takes fewer steps)

A Mini Glossary

Term	Meaning	Where it appears in this article
SVM	A logical server inside the file system	Where MGN creates the FlexVol and LUN
FlexVol	A volume that holds data. LUNs live inside it too	MGN creates one for staging and one for the target
LUN	A unit of block storage, exposed over iSCSI	One source disk becomes one LUN
Snapshot	A point-in-time image inside a volume	MGN's SNAPSHOT phase creates it
FlexClone	A writable clone made from a Snapshot	The target FlexVol is one
Split	Detaching a FlexClone from its parent and materializing it	Finalize runs it

The Procedure and the Measured Time for Each Step

The entry point is the AWS Transform workspace. MGN runs the migration itself, but it is AWS Transform that handles everything from planning to landing zone and network as a single job.

#	Step	Measured
1	Initialize MGN	Done from the console. The CLI failed (see below)
4	Install the agent	Failed once (see below)
5	Initial full sync (16 GiB / 3 disks)	226 seconds
6	Test launch	SNAPSHOT 43 seconds, CONVERSION about 4 minutes 30 seconds
7	Cutover (correct order)	MGN job 649 seconds; 817 seconds from issuing it to confirming boot
9	Finalize	Split under 60 seconds; about 33 minutes until clean-up completed

(n=1 measurements at 16 GiB / 3 disks. Fixed overhead dominates, so they cannot be used as a basis for RTO.)

Snapshot and FlexClone: What AWS Transform's Phase Names Refer To

I lined up the ONTAP side by timestamp to see which ONTAP operation each AWS Transform phase name corresponds to.

AWS Transform phase	Corresponding ONTAP operation
SNAPSHOT	Creating a volume Snapshot of the staging FlexVol
LAUNCH	Creating a FlexClone from that Snapshot

The SNAPSHOT phase is not a plain data copy but a metadata operation (44 seconds for 8 GiB). The target volume is a clone with is_flexclone: true.

What matters most is that a FlexClone uses almost no physical capacity when it is created.
Measured just before Finalize, the target volume's logical size was 7.91 GiB while its physical consumption was only 35.5 MiB. Capacity planning has to look at physical values, not logical ones.

Finalize Is Not Clean-up; It Is Where Capacity Peaks

Running Finalize starts a split of the FlexClone (the operation that detaches it and makes it independent), which materializes the data.

Elapsed from T0	Observed change
Immediately	Lifecycle `CUTOVER`, replication `DISCONNECTED`
About 3 minutes	FlexClone split starts. Physical consumption rising from 35.5 MiB → 3.94 GiB
About 4 minutes	Split completes. Physical consumption 8.57 GiB
About 13 minutes	Staging FlexVol deleted, replication server EC2 terminated

There was a gap of about 9 minutes between the split completing and the staging volume being deleted.
During those 9 minutes, physical capacity holds twice the migrated data.

If you run Finalize while the aggregate's free space is below the migrated data size, it is likely to get stuck here. Keep in mind that Finalize is a capacity risk, not an availability risk.

Four Places I Stumbled

1. A Failed Final Snapshot Behind a Successful Job

To measure downtime, I shut down the source OS and then ran cutover. The job returned COMPLETED and the target LAUNCHED. It looked like a success, but the job log recorded this:

09:14:21  SNAPSHOT_START
09:19:22  SNAPSHOT_FAIL            timed out after 300 seconds
09:19:23  USING_PREVIOUS_SNAPSHOT

The final sync did not complete, and the job fell back to the previous snapshot. The cause was my procedure: shutting down the OS also stopped the replication agent, and taking the crash-consistent snapshot timed out.

The correct procedure is to stop only the application's writes and run cutover with the OS and the agent still running. "Stop writes on the source" in the official documentation does not mean shutting down the OS.

In this test there were no recent writes, so the data matched, but doing this in a real environment loses the latest data written after the snapshot it fell back to. What makes this risky is that it does not show up in the job status (status or launchStatus) at all. By the API's design, the job returns COMPLETED as long as processing finishes, even after a fallback.

Lesson: After cutover, always check event in describe-job-log-items. A COMPLETED job is not evidence that the final sync succeeded.

2. Two Unbootable Targets: NVMe Device Names Re-enumerated

I ran cutover in the correct order twice, and both produced instances that would not boot (a UEFI shell reboot loop). The job log showed no abnormal events, and the status was COMPLETED.

Examining the boot EBS volume of an unbootable instance, the 8 GiB boot volume held the contents of a 4 GiB data disk. The boot and data disk assignment had been swapped.

The cause: when the source was stopped and restarted as part of the test, NVMe device names were re-enumerated and the boot disk moved from /dev/nvme0n1 to /dev/nvme1n1. MGN's staging-type assignment appears to be keyed on the device name, and it is not re-evaluated after the name moves.

The official best practices say not to restart the source before cutover in the first place. If a restart does happen, check the disk assignment with a command like the one below and redo the test launch.

# Check which device is treated as the boot disk, and its assignment
aws mgn get-replication-configuration --source-server-id "$SRC" \
  --query 'replicatedDisks[?isBootDisk==`true`].[deviceName,stagingDiskType]' --output table

3. No Repair API, and Three Contradicting Errors

I called UpdateReplicationConfiguration to fix the disk assignment, but whatever parameters I passed, it returned contradicting errors, and the API could not fix it.

In the end I deleted the source server and reinstalled the agent, which turned out to be unnecessary: rerunning the installer is the documented procedure.
Rerunning the installer re-establishes the list of replicated disks and their mapping against the actual machine. To avoid losing launch template customizations, try rerunning the installer before deleting anything.

4. The Installer's Hint Did Not Match the Real Cause

When the agent installation failed, the installer displayed Are kernel linux headers installed correctly?. Reading the log closely, the real cause was lack of space in /tmp.

On Amazon Linux 2023, /tmp is tmpfs; on a t3.small it has only 955 MiB, which does not meet the installer's requirement (1 GB or more free).

sudo env TMPDIR=/var/tmp/mgn-build TEMP=/var/tmp/mgn-build TMP=/var/tmp/mgn-build \
  ./aws-replication-installer-init --region ap-northeast-1 --no-prompt

Pointing it at /var/tmp like this works around it. The installer's hint is not necessarily the real cause, so check the error code in the log.

Teardown Checklist

Finalize does not finish the clean-up. The correct teardown order for leaving nothing behind is as follows (steps 1–7 depend on each other).

Terminate the target and source EC2 instances
Delete the MGN source server
Reject the VPC endpoint connection, then delete the endpoint service
Delete the NLB and its target group (important: charges continue until you delete them by hand)
Delete the target FlexVol with fsx delete-volume
Delete the SVM with fsx delete-storage-virtual-machine
Delete the ONTAP security login

The staging EBS volumes are deleted automatically with a delay, but the target's root EBS volume has no DeleteOnTermination and must be deleted manually.

Initializing AWS Transform from the CLI: Create the IAM Roles First

If running aws mgn initialize-service from the API or CLI fails, the cause is that the IAM roles have not been created.
Initializing from the console creates the IAM roles for you, but on the CLI path you need to create eight roles (including AWSApplicationMigrationFsxProxyRole for FSx for ONTAP) and attach their policies first.

Closing

With this GA, a migration now finishes with the data area already on FSx for ONTAP as iSCSI LUNs, and the step of moving the data afterwards is gone.

Any server whose boot and data are separate fits the same shape, whatever the original storage.
On the other hand, watch for the configuration spanning two kinds of storage, the physical capacity spike during Finalize, and MGN-specific traps such as "a job status of COMPLETED does not mean the final sync succeeded".

I hope this test record helps anyone migrating servers with separate boot and data to EC2 while considering FSx for ONTAP as the home for the data area.

Resources

AWS What's New: AWS Transform announces general availability of Amazon FSx for NetApp ONTAP support
MGN: FSx for ONTAP configuration
MGN: Best practices for AWS Transform MGN
FSx for ONTAP: Using Amazon Elastic VMware Service with FSx for ONTAP
Verification report (GitHub): the test environment configuration, the measured teardown procedure, and the list of feedback sent to AWS, all omitted from this article

All test environments have been deleted. The figures are single measurements under a specific environment and conditions, and vary with data size and configuration. The configuration details are in the verification report above.

This article is Part 2 of the series "The Data Foundation for AWS Modernization." The Japanese original is on hatenablog.

Designing AWS Modernization with VMware Migration as the Entry Point — Why FSx for ONTAP as the Data Foundation

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 28 Sep 2026 13:42:08 +0000

Introduction

What comes "next" for VMware workloads? Hardly a day goes by without the question coming up.

The licensing changes that followed Broadcom's acquisition of VMware have pushed many organizations into a fundamental review of their virtualization strategy. But it is worth pausing here: "getting off VMware" is not the goal. The real destination is the modernization beyond the migration — reworking applications and data, up to and including serverless and cloud-native architectures. The AWS Storage Blog, too, frames the migration not as mere license avoidance but as an opportunity to modernize infrastructure. (Reference: AWS Storage Blog — Expedite VMware migration to Amazon EC2 and Amazon FSx for NetApp ONTAP using the BlueXP workload factory migration advisor)

The Core of Modernization on AWS — AWS Transform

What drives modernization on AWS today is AWS Transform, an agentic-AI service. Beyond rehosting VMware workloads, it can containerize source code (replatforming to Amazon ECS / Amazon EKS) and take .NET Framework applications to cross-platform .NET on Linux — capabilities that run migration and modernization in parallel. It lifts migration from "moving servers" to "reworking applications," and that is what modernization on AWS looks like now.

However you rework the compute — EC2, containers, or serverless — one thing is always needed: a place for the data. This is where Amazon FSx for NetApp ONTAP, which I work with, comes alongside.

I field questions about FSx for ONTAP every day, and one thing these migrations tend to overlook is the continuity and extensibility of the storage operations model. Keep the storage operations built up over years in VMware and ONTAP environments — Snapshot, Clone, Replication, Storage Efficiency — on AWS too, and let a multi-protocol data hub support everything from migration through to cloud-native. Keep the data layer consistent and you can re-choose the compute as many times as you like. Designing modernization with AWS migration as the entry point and the data foundation at the core — that is the lens of this series.

Five Destinations for VMware Workloads

AWS officially presents five pathways for VMware workloads. (Reference: AWS for VMware — Comprehensive Pathways)

AWS for VMware: The Pathways to Migrate and Modernize on AWS

I drew the five official pathways as five rows, combining AWS services with ONTAP:

Row 1 is Migrate to Amazon EC2: Amazon EC2 reaches Amazon FSx for NetApp ONTAP over iSCSI.
Row 2 is Modernize on AWS: Amazon Elastic Container Service, Amazon Elastic Kubernetes Service, AWS Fargate, Amazon WorkSpaces, AWS Lambda, AWS Batch and Amazon FSx for NetApp ONTAP.
Row 3 is Run VMware on AWS: Amazon Elastic VMware Service reaches Amazon FSx for NetApp ONTAP over NFS / SMB.
Row 4 is Run AWS on-premises: AWS Outposts and NetApp ONTAP.
Row 5 is Run third-party hypervisors on AWS: Red Hat OpenShift Service on AWS, Amazon FSx for NetApp ONTAP, and Nutanix Cloud Clusters on AWS.

(dark theme)

Source: AWS for VMware Partner Offerings
— "AWS offers the most comprehensive set of migration and modernization options for
VMware-based workloads - from relocating to Amazon EVS, to rehosting on Amazon EC2,
containerizing with Amazon EKS, or transitioning to running third-party hypervisors
in the cloud like ROSA and NC2 on AWS." (Summarized; rephrased for licensing constraints.)

Each Pathway in Detail

1. Rehost to Amazon EC2

The most straightforward path: migrating VMware VMs as EC2 instances. AWS Transform for VMware (an agentic-AI-based migration service) became generally available in May 2025. (Reference: AWS What's New)

Separately, on 30 August 2026, FSx for ONTAP became generally available as a target storage type for AWS Transform for migrations (MGN). The former is a service for migrating from VMware; the latter is a target-storage choice for server migration. They are different features, and Part 2 of this series covers the latter. (Reference: AWS What's New)

Storage configuration options:

EBS only: Simple. Automated with MGN / AWS Transform
EBS (OS) + FSx for ONTAP (Data): Keeps ONTAP capabilities. Supported by AWS Transform (MGN) / Shift Toolkit / Cirrus Migrate Cloud (the scope of this series)

2. Modernization (Containers / Serverless)

As the next step after rehosting to EC2, you can modernize according to the characteristics of each workload.

Target	Suitable workloads	FSx for ONTAP integration
ECS / EKS (EC2 mode)	Stateful containers (DB, middleware)	✅ iSCSI / NFS mount available
ECS / EKS (Fargate)	Stateless microservices	△ Through EFS only (no iSCSI)
AWS Lambda	Event-driven, short-running processing	△ EFS mount available, no iSCSI
AWS Batch	Batch processing / HPC	✅ iSCSI available in EC2 mode
Amazon WorkSpaces	VDI (virtual desktops)	✅ FSx for ONTAP file shares

This is not about containerizing VMs as they are. The journey is "rehost to EC2 → containerize the application → move step by step to Fargate/Lambda".

3. Amazon EVS (Keep VMware on AWS)

With Amazon Elastic VMware Service, you can deploy VMware Cloud Foundation (VCF) directly on EC2 bare metal inside your VPC. You keep your existing vSphere skill set and can connect FSx for ONTAP as an external datastore.
It suits cases with a large VMware-dependent application estate, where leaving VMware in a short time is impractical.

4. AWS Outposts (On-premises AWS + NetApp External Storage)

AWS Outposts is a fully managed service that places AWS infrastructure on premises.
Third-party block storage integration was announced in December 2024, and NetApp ONTAP and StorageGRID are available as partners validated through the AWS Service Ready Program. (Reference: AWS Blog) (Reference: NetApp)

NetApp ONTAP iSCSI LUNs can be attached directly from the AWS console as data volumes for EC2 instances, and boot volume support was added in July 2025.
This gives you separation of compute and storage: "compute is AWS-managed, storage is your existing ONTAP."

5. Partner Solutions (ROSA / Expanding Nutanix + NetApp Integration)

Red Hat OpenShift Service on AWS (ROSA): A fully managed environment based on OpenShift. With the NetApp Trident CSI driver, ROSA Pods can access FSx for ONTAP over NFS/iSCSI. (Reference)
Nutanix + NetApp ONTAP (on-premises Nutanix Cloud Platform): At .NEXT Chicago in April 2026, a partnership was announced that lets the Nutanix Cloud Platform (NCP / AHV) use NetApp ONTAP (AFF / FAS) as external storage. Over an NFS-based connection, compute (Nutanix AHV) and storage (ONTAP) scale independently. GA is planned for the second half of 2026 (NetApp targets Q3 2026, with Early Access underway). The partnership covers on-premises NetApp enterprise storage; integration with Nutanix Cloud Clusters (NC2) on AWS or with FSx for ONTAP is not announced or committed in public sources (NetApp mentions "NetApp cloud storage services for Nutanix Cloud Platform" as a future direction, with the timing and the services undetermined). (Reference: NetApp Blog, GA Q3 2026) (Reference: Nutanix press release)

The NetApp ecosystem as a whole:

Figure 2: the data layer stays the same when the compute changes. SnapMirror is what crosses the boundary (dark theme)

What this figure shows is that ONTAP is a data platform independent of the choice of hypervisor or cloud. Whether you move the compute layer from VMware to EC2 or to Nutanix, the data layer stays consistently available, and SnapMirror moves and protects the data.

Where This Series Sits

Figure 3: the phases beyond rehosting, and the alternatives to it. This series covers Phase 1 (dark theme)

This series concentrates on Phase 1 (rehost), but the EC2 + FSx for ONTAP configuration is designed not to close off the path to Phase 2 and beyond. FSx for ONTAP is reachable over NFS/iSCSI from ECS/EKS as well as from EC2, so the data layer can stay as it is after containerization.

Why "EC2 + FSx for ONTAP"

Among the five pathways, here is why this series looks at "EC2 + FSx for ONTAP" in particular.

An Entry Point for Rehosting and a Foundation for Modernization

This configuration is not just a rehost destination; it is designed not to close off future modernization.

Now (Phase 1): Rehost VMs to EC2. Data disks on FSx for ONTAP iSCSI
Next (Phase 2): Containerize the application. Keep accessing FSx for ONTAP over NFS/iSCSI from ECS/EKS
Later (Phase 3): Parts that became stateless move to Fargate/Lambda. The FSx for ONTAP data layer stays as it is

The multi-protocol access of FSx for ONTAP (NFS/SMB/iSCSI) is what supports this step-by-step migration. A volume used as an iSCSI LUN for EC2 can later be NFS-mounted by an EKS Pod.

The Value of ONTAP Is Capability, Not Capacity

Seen as just "large-capacity storage," FSx for ONTAP can look expensive next to EBS. But capacity is not where the value of ONTAP lies.

ONTAP feature	Use on AWS
Snapshot	Point-in-time copies in seconds. Instant test environments
FlexClone	Clones without copying data. Lower cost for dev/test
SnapMirror	Block-level replication. Cross-Region DR
Compression / Dedup	Lower effective capacity. Most effective for databases and logs
Thin Provisioning	Pay for what is used. Avoid over-provisioning
Multi-protocol	Access the same volume over NFS/SMB/iSCSI

FSx for ONTAP EC2 Integration Pattern

Figure 4: the boot column is Amazon EBS in all three rows; only the data column changes to FSx for ONTAP from staging on (dark theme)

This configuration has the following advantages.

Storage and compute scale independently: Change FSx for ONTAP capacity/throughput without stopping EC2
The ONTAP operations model carries over: Operate snapshots, clones and replication with the same CLI/API as on premises
The limit moves: The ceiling is no longer the per-instance Amazon EBS limit but the EC2 network bandwidth and the FSx for ONTAP throughput capacity.

Note: Performance Ceilings and Protocols

"The limit moves" above does not mean the limit goes away. This series has not measured it yet, but verification in a sibling project (F-1 iSCSI measurement and others) has shown the following.

Simple arithmetic between session count and bandwidth misses Estimating "required bandwidth ÷ 625 MBps = number of sessions" misses in both directions. One TCP connection came out at 1.82x the prediction and sixteen at 0.20x, so the ceiling is not set by the capacity of a single flow alone.
The performance gap between protocols (iSCSI vs NVMe/TCP) depends on the deployment Measured on the same file system, iSCSI and NVMe/TCP were 0.06% apart — practically the same — while on other configurations their order swapped. Quoting a past ratio as the basis for a protocol choice is unsafe.
ANA multipath is not available on Amazon Linux 2023 CONFIG_NVME_MULTIPATH is unset in all three AL2023 kernel series (details). The ANA path makes a large difference to sequential reads, so verifying it enabled needs a different distribution (RHEL 9.3 or similar).

These values come from tests against a raw device and include burst; they do not apply directly to this configuration, which puts a file system on the LUN.

Choosing a Migration Tool

Even when the destination is "EC2 + FSx for ONTAP," the tool depends on the environment.

Condition	Candidate tool	Notes
You want the data volumes on FSx for ONTAP (whatever the source storage is)	AWS Transform (MGN)	Set FSx for ONTAP as the target storage type. Generally available since 30 August 2026. Boot is always Amazon EBS. Verified hands-on in Part 2
Already on an ONTAP NFS datastore + small to medium scale	NetApp Shift Toolkit	Converts disks without copying them, using FlexClone. EC2 / FSx for ONTAP support is Early Preview. Verified hands-on in Part 4
Already on ONTAP + large scale (100+ VMs)	Cirrus Migrate Cloud	YAML-driven automation. Paid, through AWS Marketplace
Migration planning and sizing only	AWS Application Discovery Service / AWS Migration Evaluator	Inventory collection and cost estimation. AWS native

Where NetApp Shift Toolkit Fits

Shift Toolkit converts VMs on an ONTAP NFS datastore without copying their disks. Conversion is done with FlexClone and metadata operations, so the conversion itself is quick.
Migrations from VMware ESXi to Hyper-V, OpenShift Virtualization, Proxmox VE and OLVM are GA today, and support for VMware ESXi → Amazon EC2 / FSx for ONTAP is at the Early Preview stage.

⚠️ Early Preview note: The current scope is configurations that place data disks on FSx for ONTAP. Specifications and constraints may change.

What This Series Covers

Starting from migration as the entry point, the series runs four parts through to the modernization beyond it. Parts 2 and 4 are the two migration paths; Part 3 is the modernization that follows.

Part	Theme	Status	Source storage
Part 2	AWS Transform Now Supports Block Storage Migration to FSx for ONTAP — Benefits and Pitfalls from a Hands-On EC2 Test	GA (30 August 2026) / verified hands-on	Any
Part 3	Beyond the AWS migration of a VMware environment: serverless, cloud-native, and the data foundation (ECS / EKS, S3 Access Points, DR)	Upcoming	—
Part 4	NetApp Shift Toolkit	EC2 / FSx for ONTAP support is Early Preview / verified hands-on	An ONTAP NFS datastore

Of the two migration paths, AWS Transform is already GA and applies more widely, so the one a reader can try today (Part 2) comes first. Shift Toolkit (Part 4), which builds on existing ONTAP assets, comes later because its EC2 support is Early Preview. Part 3 sits between them and covers how to provide the data foundation when migrated workloads are reshaped into containers and serverless.

Across the parts, we plan to cover:

Environment setup: Design and build of VPC + FSx for ONTAP + VPN
Migration: Migrating Linux / Windows VMs to EC2
Modernization: Containerization on ECS / EKS, Fargate with S3 Access Points, DR and cyber resiliency
Operational continuity: Snapshot / Clone / SnapMirror behavior after migration
Cost comparison: TCO against an EBS-only configuration

Summary

Migrating from VMware is not only about "where to go" but about "how to carry the storage operations model forward".
For ONTAP users, FSx for ONTAP is a strong option for keeping, and making full use of, the value of ONTAP on AWS.
Choose the tool by source storage and scale. If the source storage does not matter, AWS Transform (MGN); for ONTAP NFS datastores at small to medium scale, Shift Toolkit.
Shift Toolkit's EC2 support is at the Early Preview stage, so those verification results may change at GA.

The next article (Part 2) covers the AWS Transform (MGN) path that sets FSx for ONTAP as the target storage, verified hands-on after GA. Part 3 covers the modernization beyond migration (serverless and cloud-native), and Part 4 explains how Shift Toolkit works — how FlexClone changes VM migration.

Reference Links

Verification repository (GitHub): VMware-Migration-EC2-ONTAP — procedures, configuration and figures for this series
Modernization journey map (FSx for ONTAP Adoption Playbook) — which material and implementation pattern to read at each stage beyond migration
AWS for VMware — Comprehensive Pathways
AWS Transform for VMware
AWS What's New: AWS Transform announces general availability of Amazon FSx for NetApp ONTAP support
MGN: FSx for ONTAP configuration
Nutanix Blog: NetApp and Nutanix Announce Technical Partnership
AWS for VMware Partner Offerings (ROSA, NC2)
NetApp Shift Toolkit Overview
AWS Storage Blog: Seamless migration from VMware to FSx for ONTAP and EC2
AWS Storage Blog: Expedite VMware migration to Amazon EC2 and Amazon FSx for NetApp ONTAP using the BlueXP workload factory migration advisor
Amazon FSx for NetApp ONTAP
Amazon Elastic VMware Service (EVS)
Red Hat OpenShift Service on AWS (ROSA)
Nutanix Cloud Clusters on AWS (NC2)
AWS VMware Migration Accelerator
NetApp Blog: Simplify VM migration with Shift Toolkit

This article is Part 1 of the series "The Data Foundation for AWS Modernization." The Japanese original is on hatenablog. Shift Toolkit's EC2 support is Early Preview, and its specifications are subject to change.

What to build a file portal on — Nx Plugin for AWS 1.0, AWS Blocks and Amplify Gen 2, same four features (File Portal Part 7)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 16 Sep 2026 14:34:33 +0000

Introduction

Nx Plugin for AWS reached 1.0 and AWS Blocks appeared in preview, so starting an application on AWS now has three front doors. I took all three to the same scope, counted the resources the same way, deployed them for real, implemented four features on each, and deleted everything at the end.

This article stands on its own. It continues a series, but the only thing it assumes is that a file portal is running on an FSx for ONTAP S3 Access Point.

What prompted it was reading a Zenn article by an AWS SA that compared all three. Someone else's comparison is useful, but as a basis for a decision in my own repository it is thin: you have to deploy and delete before you can say you compared anything.

Part 1 was about the application layer — what to build a file portal with. This one is about the tooling underneath it.

What the verification covers:

The resource count of each generated output, and the fixed cost its defaults imply
Deploying each one for real and exercising it
The differences that appear once the same four features are implemented on all three
What survives a stack deletion, and the order in which to clean it up

Out of scope: which tool is better, Nx or monorepos as concepts, and keeping generated code current over time.

The conclusions first:

Choosing a tool is a decision you can take back. The data path is common to all three, and switching tools does not mean rebuilding the FSx for ONTAP side.
The difference is not "can it be built" but what gets fixed in place once you have chosen. All three deployed without trouble.
The generated defaults mean fixed monthly cost at zero requests.
What a stack deletion leaves behind comes in two kinds. Retain and deletion protection are the generator's defaults and can be released on the generating side. Log groups that never appear in the template belong to CDK's internals, and there is no route to release those. The KMS waiting period is the only thing that cannot be avoided.
For some resources, notably KMS keys, DELETE_COMPLETE means "scheduled for deletion after a 30-day wait" rather than "gone".

Switching tools does not touch the data foundation underneath

The components side by side, matched to one scope: web, auth, API and a table. The data path below the S3 Access Point is this portal's real configuration and is identical in all three.

Figure 1: components of the file portal composed on Amplify Gen 2

Amplify Gen 2 is what this portal actually runs on. No perimeter resources — AWS WAF, customer managed KMS keys — are created by default. Authentication is Amazon Cognito, on the default ESSENTIALS tier.

Figure 2: components of the file portal composed on the AWS Blocks production preset

What to read in the AWS Blocks figure is the absence of Amazon Cognito. Authentication is AuthBasic (DynamoDB plus JWTs), so it moves out of a managed service and into the tables. Two of the four tables are for auth, and all four carry deletion protection.

Figure 3: components of the file portal composed on Nx Plugin for AWS

For Nx Plugin for AWS the difference is more perimeter in the figure: two AWS WAF web ACLs on the path (a third sits in a separate us-east-1 stack), Amazon Cognito on PLUS with MFA required, and four KMS keys. Those three are the fixed-cost difference described later.

That the three figures match below the access point matters: switching tools does not mean rebuilding the FSx for ONTAP side. Nothing was created or deleted on the FSx for ONTAP side during this verification.

Reaching it was also identical on all three. An Internet-origin access point is reached by passing the alias as a bucket name to the standard S3 SDK — no endpoint override, no VPC attachment. A file on the NAS can be found, opened and written from a browser with no VPN and no SMB mount, and through phase 3 the functions do not even need to sit in a VPC.

The same four features on all three tools (2026-09-14)

Everything above compares the outputs as generated. What someone choosing a tool actually wants to know is the difference once their own application sits on it. So a few days later I implemented the same four features — sign in, list, read, upload (listFiles / readFile / uploadFile) — on each and ran them against the same FSx for ONTAP S3 Access Point.

The target was fixed too: the same prefix (13 objects), the same file (1,615 bytes, text/markdown; charset=utf-8), the same 24-byte write. All three returned the same values.

The listing

.md rows are listed with sizes and modification times" width="800" height="476">
The listing on the AWS Blocks build: thirteen files.

The listing on the Nx Plugin for AWS build, with the generated shadcn sidebar.

The listing on the Amplify Gen 2 portal. Being a portal in actual use, it carries extra features such as stars and download icons.

Reading

.md sit 1615 bytes and text/markdown; charset=utf-8, the same report is rendered in the page, and an Upload section follows below it" width="800" height="476">

The two above render the report in the page. The Amplify Gen 2 portal below does not: it hands .md to an AI Assistant panel instead, because that is how I wrote it.

An aside: a by-product of the capture pass was one accessibility defect in my own portal. The document icon on a file row announces itself as aria-label="Download <filename>", yet what it actually does is select the file for AI processing. The title attribute mentions both, but a screen reader reads out only Download, so someone using assistive technology is told the button downloads a file when it starts AI processing instead. It is unfixed as of writing, recorded as something for the portal side to correct. Without lining up three builds in the same order, I probably would not have noticed.

Uploading

The two above are listings right after a successful write. The Amplify screen below is a refusal: the same file name was sent twice and the server side declined to overwrite. That is a difference in what I implemented, not in the tools.

How the IAM permission differs by tool

Reaching the S3 Access Point is the same on all three, but the IAM needs care: all three require the access-point ARN form.

arn:aws:s3:<region>:<account-id>:accesspoint/<name>          # ListBucket
arn:aws:s3:<region>:<account-id>:accesspoint/<name>/object/* # GetObject, PutObject

Written in the bucket form (arn:aws:s3:::<alias>), the deploy succeeds and list, get and put all return AccessDenied at run time — a failure that is hard to read.

AWS Blocks in particular has FileBucket.fromExisting(alias), which internally calls s3.Bucket.fromBucketName(alias).grantReadWrite() and therefore generates the bucket form. Trust the block's grant and add nothing, and it passes locally and fails once deployed.

Which layer refuses a malformed input

Trying an input that walks out of the prefix (../escape.txt) showed that the refusing layer differs by tool.

Input	Nx Plugin for AWS	AWS Blocks
`../escape.txt`	HTTP 403 `{"message":"Forbidden"}` (blocked by WAF)	The application's own domain error
An empty file name	HTTP 400 plus the application's error	The application's own domain error

On the Nx build, AWS WAF stopped the request before the Lambda. Defence in depth on by default is a good thing, but the refusal arrives as a generic HTTP 403, so the starting point of an investigation moves from your own code to the WAF metrics — worth knowing before you are debugging at speed.

What this round measured (deploy time and resource count)

	Deploy time	Resources	Application Lambdas
AWS Blocks (production preset)	318 s	82	1 (the whole API)
Nx Plugin for AWS (sandbox)	307 s	100	5 (one per procedure)

Amplify Gen 2 is excluded from this round's figures (deploy time, resource count) because the comparison used the portal already in production rather than a fresh build.

Worth noting: a green local e2e run on AWS Blocks says nothing about whether the access point is reachable, because the mocked storage closes the loop inside the mock.

Deploying the starters as generated (2026-09-13)

Stopping at synth leaves out everything you learn while deleting resources. So all three were deployed to a real AWS account in their generated state, before any features were added.

Output	Resources	Deploy	Exercised
AWS Blocks (sandbox preset)	83	—	signUp and friends returned 200 over JSON-RPC
AWS Blocks (production preset)	117	1,228 s	all five operations 200, CloudFront delivery included
Nx Plugin for AWS	86	317 s	TOTP enrolled, then 200 from the API via the identity pool

All three deployed without trouble. The Blocks production preset takes longer because DynamoDB builds its GSIs one at a time.

Fixed cost that accrues at zero requests

A generator's defaults reach the monthly bill even when the template says nothing about them.
The figure is settled at the moment the tooling is chosen, which makes it a number to read
before starting rather than after.

With not a single request served, the floor for each tool looks like this (FSx for ONTAP's own cost is not included).

Output	WAF	KMS CMK	Monthly floor
Nx Plugin for AWS	3 ACLs + 6 rules = $21	4 keys, $4	from $25
AWS Blocks (production preset)	none	1 key, $1	$1
Amplify Gen 2 (starter)	none	none	$0

Nx starts higher because what production needs is present from the start. It does bite when something is stood up for a test and left running.

What a production portal still needs

The feature set beyond these four, arranged in phases. Which phase your requirements stop at also changes how you choose a tool.

Phase	Feature	VPC attachment	Newly required
1	Paging, search, sorting	not needed	nothing
2	Downloads and share links	not needed	presigned URLs and an expiry policy
3	Large file uploads	not needed	multipart, or similar
4	Showing ONTAP information	needed	reaching the management LIF, splitting the functions
5	Operating ONTAP	needed	audit logs, approval for irreversible operations
6	AI processing	depends	confidence thresholds, data classification

From phase 4 the ONTAP management LIF has to be reachable, which is where the decision to put a Lambda in a VPC arrives, and with it fixed cost such as a NAT gateway. If phase 5 or later is a possibility, splitting the functions into "touches the S3 Access Point" and "touches ONTAP" from the start is the design that pays off.

Tearing down the evaluation — what a stack deletion leaves

This is about putting the evaluation environment back, not about running the portal. A portal that
gets built runs for years; the deletion happens while you are still trying things out. Even so, how
much is left behind is worth knowing before you start.

Listing what remained after each stack deletion turned up resources that static analysis of the template (synth) could not show.

Output	Left after the stack deletion
Blocks (sandbox preset)	5 log groups
Blocks (production preset)	4 DynamoDB tables (deletion protection), 8 log groups, 1 KMS key (waiting)
Nx Plugin for AWS	1 Cognito user pool, 1 DynamoDB table, 4 KMS keys, 2 IAM roles, 7 log groups

Those eight and nine are the generator's defaults, not an AWS constraint. The generated code
is what applies Retain and deletion protection, so anything that may go can be released on the
generating side (RemovalPolicy.DESTROY in CDK; the deletion-protection properties on DynamoDB
and Cognito set to false). Defaults leaning towards keeping data is a production-shaped choice;
it conflicts with intent only when the deployment is a test.

Log groups are the awkward ones. The Lambda functions behind CDK and tooling custom resources create them on first execution with the default retention (never expire), which puts them outside CloudFormation's control and leaves them behind. One sweep at the end of the teardown catches them:

aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/<stack-name>" \
  --query "logGroups[].[logGroupName,retentionInDays]" --output text

The removal order matters too. Nx's user pool cannot be deleted until deletion protection is off, and the SMS configuration needed to switch it off references one of the two IAM roles listed above as survivors. Delete those IAM roles first, while clearing up what the stack left behind, and the route to switching off deletion protection is gone. Delete the user pool first, then the roles.

Your own functions are a different case from CDK's. A Lambda you write can declare its log
group in the template, so it does not survive — measured, the declared ones went away on all
three. What survives belongs to CDK's internal custom resources, and there is no route to declare
those from your own code: the CustomResourceProvider base class declares an IAM role and a
Lambda function and no log group (custom-resource-provider-base.ts,
read in full on main, 2026-09-16).

No released version fixes this (checked 2026-09-16). Two feature requests are open upstream
(aws-cdk #26553,
aws-cdk #23909). A third,
aws-cdk #24815, is closed as completed — but
its closing comment, from an AWS maintainer, says the fixing PR was reverted, that the problem
is general to all custom resources, and that tracking moved to #23909. Closed is not shipped.

AWS Support has not been asked. Lambda documents the implicit creation and the never-expire
default, the CDK source shows what the provider declares, and CDK's primary channel is its
GitHub repository rather than AWS Support. What would be worth filing is the per-output residue
count measured here, which appears in neither issue.

Two behaviours that are easy to misread in operation

Both are documented, and both are still easy to read the wrong way.

Trap 1: a deployment mode cannot be dropped later

Nx generates deploy-sandbox as cdk deploy ... --express under the hood. If a deploy breaks, "drop --express and run it again" does not work: a stack last updated in express mode requires its later operations to be express too. The remedy is to pair --express --rollback from the start.

Trap 2: DELETE_COMPLETE does not always mean gone

DELETE_COMPLETE in CloudFormation does not always mean the resource is gone. For a KMS key it means deletion was scheduled — a 30-day wait by default — and that wait cannot be shortened afterwards. Confirm a teardown from the listed state rather than from the status.

Choosing between them

The three are not mutually exclusive; they suit different purposes.

Nx Plugin for AWS: production-leaning defaults and dependency management that holds up as projects multiply. Fixed cost is the highest of the three.
AWS Blocks: running the whole application locally with no AWS account (it is in preview, so treat a Block ID as immutable after deployment).
Amplify Gen 2: treating the backend definition, the sandbox and hosting as one thing.

Part 1's conclusion stands unchanged — Nextcloud when file sharing is the centre, Amplify Gen 2 when AI processing and ONTAP operations are. What this part settled is which tool the "build" side sits on.

Test environment

Item	Value
Measured	2026-09-13 (starters, live), 2026-09-14 (live, with the four features implemented)
Region	ap-northeast-1 (only Nx's CloudFront web ACL in us-east-1)
Outputs	Nx Plugin for AWS 1.0 (`@aws/nx-plugin@1.0.0`), AWS Blocks (preview), Amplify Gen 2 (`npm create amplify`)
Local	Node.js v26.4.0, npm 11.17.0
Pricing	AWS Price List API, ap-northeast-1, 2026-09-07
Teardown	everything deleted; only 5 KMS keys remain in their waiting period (`PendingDeletion` is not billed)

The record in the repository

The numbers, the sources and the teardown procedure live in the repository, in more detail than this article.

Choosing an application foundation — where the three sit, measurements up to synth, fixed cost, reproduction steps
Deploy verification and teardown — the 15-entry pitfall register with a source on every entry, the teardown order, and the live record
The same four features on all three — twelve captures across three builds by four screens, how the permission differs, which layer refuses, 9 more pitfalls, reproduction steps
Beyond the four features — what each phase assumes, the nature of the cost, which phases can be skipped
Documentation index

Closing

Everything in this part is about defaults. Choosing a tool fixes your monthly floor and your teardown procedure, and neither is written in the template.

What worked as a method was writing the same four features three times. The difference shows up not as how much code you write but as where you write it, and which layer refuses a bad request. Choosing a tool is, I think, choosing where you look first when something breaks.

I hope this record helps anyone weighing the same choice.

Everything in the test environment has been deleted (only KMS keys remain in their waiting period). The figures come from one specific environment and configuration and will differ with workload and setup. Prices are the published ap-northeast-1 rates as of 2026-09-07.

How far ONTAP features reach through an S3 Access Point — measuring qtree, quota and FlexClone (File Portal Part 6)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Fri, 04 Sep 2026 00:59:30 +0000

Introduction

This is the sixth part of the series, and Part 7 continues it. Part 1 built a file portal on Amazon FSx for NetApp ONTAP (FSx for ONTAP) S3 access points, Part 2 added ransomware response and WORM retention, Part 3 put AI agents on it, Part 4 delegated 182 storage operations, and Part 5 collected what the running cluster refused.

This part is what was left off the screen. That is the difference from last time: where Parts 4 and 5 were about operations you run from a screen, this one has two subjects. How much of ONTAP stays usable without going through the portal at all, and the operational work that was handed to a schedule instead of a button.

What this part does not cover: performance figures (this measures whether things are possible and under what condition, not throughput), running SnapLock or Object Lock (irreversible, so not enabled without an instruction naming the retention), and actual tiering into a capacity pool (needs a different aggregate configuration, which was not on this path).

The reach of ONTAP features on this path

When you read and write files over the S3 API through an FSx for ONTAP S3 access point, how much of ONTAP stays usable alongside it? You want qtrees to carve up directories, quotas to stop a tenant running away with the space, FlexClone to spin a verification copy. Does any of that sound like your environment?

Start looking into it and NetApp's documentation gives you pause. ONTAP S3 interoperability records qtrees, quotas, FlexClone, and a volume clone of a FlexGroup containing ONTAP S3 buckets as not supported — for the ONTAP S3 server.

That is where it stalls. The FSx for ONTAP S3 access point is an AWS mechanism, and what it attaches to is a volume, not a bucket. Attaching one does stand up an ONTAP S3 server on the SVM, and I/O through the access point goes through ONTAP's S3 protocol stack. So the two are not unrelated. Does the same restriction appear?

I measured it. Up front: everything I measured worked.

ONTAP feature	Result on the S3 access point path
Qtree	Can be created. Appears as an S3 prefix, and an object PUT into it lands inside the qtree
Quota	A tree quota refuses an S3 PUT
FlexClone (volume granularity)	The clone can be made, and the clone itself can take an S3 access point
FlexClone (file granularity)	A file written over S3 can be cloned, and the clone is visible as an object through the access point
FlexGroup volume	An S3 access point can be attached. PUT / GET / LIST and multipart all work
Clone of a FlexGroup	The clone can be created

The four items that table lists did not appear as restrictions on this path. Reading that table as "not supported there, therefore not supported here" is wrong, and so is the reverse: a "supported" row is not evidence about this path either. The subjects differ, so nothing is known until it is measured.

What actually tripped me up was not in the table at all. Three things.

An NTFS volume with a UNIX identity on an SVM with no CIFS server attaches, reaches AVAILABLE, and then refuses every data operation with AccessDenied. AVAILABLE says nothing about the health of the file system layer
A volume created on the ONTAP side takes ten to twenty minutes to appear in the AWS API. The documentation says "several minutes"
A volume that has once carried an S3 access point cannot be deleted from the ONTAP side, even after every access point is detached. Only aws fsx delete-volume removes it

Below is the walkthrough, and how the controls were set up.

Test environment

Item	Value
Date	2026-08-26 (UTC)
Region	ap-northeast-1
ONTAP version	9.18.1P3D1
Deployment type / throughput	SINGLE_AZ_1 / 128 MBps
Test volumes	UNIX security style 1 GiB × 2 (one with an access point, one as the control), NTFS 1 GiB × 1, FlexGroup 400 GiB × 1
Access point identity	UNIX `root`, internet origin
Cleanup	Throwaway volumes only. All deleted

A note on the identity: UNIX root was used to keep the measurement simple. Every request through an access point is authorized as that one file system identity, so specifying root removes any narrowing by file permissions. This is not a recommended configuration. Use a dedicated user with only the permissions the writes need, and split access points per purpose (FileSystemIdentity cannot be changed after creation).

The distinction that matters most — different subjects

The results below have to be read with these two kept apart.

Mechanism	What it is	Whose
ONTAP S3	The S3 server with which ONTAP serves buckets. This is what NetApp's table is about	NetApp
FSx for ONTAP S3 access point	An AWS access point attached to an FSx for ONTAP volume	AWS

They are not unrelated. Attaching one stands up an ONTAP S3 server on the SVM, and the I/O goes through ONTAP's S3 protocol stack (measured). That is exactly why NetApp's table is the right place to start.

Being the right place to start is not the same as being the conclusion. The target is a volume rather than a bucket, and the namespace and the authorization path both differ. "Not supported there, therefore not supported here" turns something unchecked into something checked. The reverse reading is the same error.

The walkthrough

Qtree — usable

Step	Result
Create a qtree in the volume with an access point	Succeeded
Control: the same qtree in the volume with no access point	Succeeded. No difference
`list-objects-v2 --delimiter /`	The qtree appears in CommonPrefixes
PUT into the qtree prefix	Succeeded
Inspect the qtree from ONTAP	The written file is inside the qtree directory

NetApp's "Qtrees: not supported" does not manifest as a refusal on this path. The qtree is created, it is visible as a prefix in the S3 namespace, and an object written into that prefix lands inside it. Together with the quota result below, it behaves as a qtree.

Incidentally, a volume with an access point attached carries an internal ____NTAP_S3_MAPPING directory at its root. It is visible over NFS and SMB, so anyone who inspects a collect-layer volume by hand will see it.

Quota — enforced against an S3 PUT

Step	Result
Set a tree quota on the qtree (space 1 MiB, files 10) and enable quotas on the volume	Succeeded
`quota report`	The rule is active, and the file written over S3 is counted in files used
PUT 15 small objects	8 succeeded, 7 refused. files used stopped at exactly 10/10
What the S3 client received on refusal	HTTP 507 `InsufficientCapacity` / `Maximum storage capacity of file system has been reached.`
Control: raise the files limit 10 → 50 and re-PUT the refused key with the same body	Succeeded

The control carries the conclusion. Raising the limit alone made the identical PUT succeed, so the refusal was the quota, not a capacity shortage and not a permissions artefact.

That the response mis-describes its cause matters to the design. The file system was not full; a qtree file-count quota was reached. An operator who sees a 507 and "maximum storage capacity of file system" will consider growing the file system. If you use quotas on this path, write down what that response actually means in the runbook.

FlexClone — both volume and file granularity

There are two granularities, and they behave differently.

	File granularity	Volume granularity
Where it lands	The same volume. It appears as a new key on the same access point	A separate volume
Authorization boundary	Same as the parent	Its own access point, its own policy, its own identity
Time before it is usable	None	599 to 1,177 s measured, for a volume created through the ONTAP API to appear on the AWS side
Visible over S3	Yes. `StorageClass=FSX_ONTAP`, sha256 matches the source	Yes. LIST / GET / PUT through the clone's own access point

At either granularity the clone shares data blocks with its parent. Cloning a 256 MiB file four times:

Point	Logical	Physical
Just after one 256 MiB PUT	—	253,534,208 B
After four file-granularity clones	1,350,942,720 B	277,200,896 B

Logical grew to five copies, physical by about 23 MB. FSx for ONTAP bills provisioned SSD capacity, so there is no capacity to provision per duplicate.

NetApp states that copies consume no storage except what is required for metadata until changes are written to the copy (FlexClone volumes, files, and LUNs). The operation that ends the sharing is a split (volume clone split start), which I did not measure. From ONTAP 9.4, on AFF systems where the volume guarantee is none, the documentation states that the split shares the physical blocks rather than copying data (splitting procedure).

A file-granularity clone gives no observable failure, however. POST /api/storage/file/clone returns 202 and a job UUID, but the UUID resolved to 404 entry doesn't exist and appeared in no job listing. The same fsxadmin retrieves volume-create and volume-clone jobs as state=success, so this is not a permissions problem. A call naming a destination directory that does not exist also returned 202 and created nothing. Judge the outcome by inspecting the destination file.

FlexGroup — access point attachment

Creation has conditions, learned by failing in order.

Step	Result
Create with default parameters	Failed. `Volumes of this type must be at least 50GB`
Retry at 50 GiB	Failed. `Aggregates not matching FabricPool requirements: aggr1`
Retry with the aggregate named explicitly	Failed. `Minimum size is "400GB"`
400 GiB, explicit aggregate, `tiering.policy=none`, thin	Succeeded

FlexGroup volumes can be created from the AWS Management Console, the AWS CLI and the Amazon FSx API (Managing volumes). Naming the aggregate explicitly was required because this one was created through the ONTAP REST API; that constraint does not appear on the AWS-side path.

What follows is the point: whether an access point on the AWS side attaches to a FlexGroup created on the ONTAP side.

Step	Result
Appears on the FSx for ONTAP side as `VolumeStyle=FLEXGROUP`	It does, with an `fsvol-` identifier
Attach an S3 access point to the FlexGroup	Succeeded. `Lifecycle=AVAILABLE`
PUT / GET / LIST	Succeeded. The GET content matched
12 MiB multipart upload	Succeeded. `StorageClass=FSX_ONTAP`
Snapshot the FlexGroup	Succeeded
Clone that FlexGroup	Succeeded. `style=flexgroup`, `is_flexclone=true`, online

That last row is the item NetApp's table records as "volume clone of the FlexGroup volume containing ONTAP S3 buckets: not supported".

What actually failed on this path

The constraints were not on NetApp's side of the table. Within what I measured, these three are the real traps.

A security-style and identity mismatch — failure after a successful attachment

Step	Result
Attach with an NTFS-security-style volume + UNIX identity (`root`)	Succeeded. `Lifecycle=AVAILABLE`
PUT through that access point	Refused. `AccessDenied`, body only `Access Denied`
Control: same identity, same caller, UNIX volume	PUT / GET / LIST all succeeded
CIFS server on this SVM	None. The UNIX-to-Windows mapping cannot resolve

AVAILABLE does not mean the file system layer is healthy. Attachable and usable are different properties: IAM and the access point policy are both passed, and the file system layer refuses after that. The body is only Access Denied and does not name the layer. A refusal at the identity layer says no identity-based policy allows ... instead, so the difference in the body is the signal.

Trying WINDOWS identity on the same SVM does not even complete the attachment.

Step	Result
Attach with WINDOWS identity on an SVM with no CIFS server	Failed. `did not stabilize` (`NotStabilized`), stack rolled back
Whether an attachment survives the rollback	It does not. Nothing was orphaned

The wait for something created in ONTAP to appear on the AWS side

A volume created through the ONTAP API is not immediately in the AWS-side describe-volumes. Without an fsvol- identifier, neither AWS::FSx::S3AccessPointAttachment nor create-and-attach-s3-access-point can reference it.

Measurement	Result
FlexGroup, polled every 20 s with no gaps	Appeared at 599 s (about 10 min), with an `fsvol-` identifier
A FlexClone volume, polled every 20 s with no gaps	Appeared at 1,177 s (about 19.6 min)
A separate run	Still absent at 1,258 s (about 21 min)

The three observations disagree, so this is not an upper bound. The order of magnitude is tens of minutes, which is not enough to bake a fixed wait into a design. AWS documents the following, and the measurement is longer than its "several minutes".

Amazon FSx periodically syncs with ONTAP to ensure consistency. If you create or modify volumes using NetApp applications, it may take up to several minutes for these changes to be reflected in the AWS Management Console, AWS CLI, API and SDKs.

— Managing FSx for ONTAP resources using NetApp applications

This delay has to be kept apart from absence. In an observation of about 2.5 minutes, a volume created through the ONTAP API shows no fsvol- identifier, and it would be easy to conclude from that that neither a clone nor a FlexGroup can take an access point. Wait, and it appears, and it attaches. 2.5 minutes sits inside the "a few minutes" AWS documents, so what is observable in that window is "not yet appeared".

It lags in the other direction too. Setting a junction path on the ONTAP side and attaching immediately fails with Amazon FSx is unable to attach S3access point because the volume is not mounted. The AWS-side JunctionPath was None at that moment and still was two minutes later. Setting it with aws fsx update-volume made it visible on the AWS side in about 40 seconds, and the attachment then succeeded. The error message was telling the truth. What was wrong was my assumption that setting it in ONTAP made it visible to AWS.

A residual NAS bucket refusing volume deletion on the ONTAP side

First, the terms. An S3 access point is an AWS-side mechanism and an object store NAS bucket is an ONTAP-side object, and the two are different things. Attaching an access point creates a corresponding NAS bucket on the ONTAP side.

What happened here is residue of the latter. After every access point was detached, the NAS bucket remained on the ONTAP side, and ONTAP refused to delete the volume because of it.

Cannot delete volume "..." in SVM "..." because it is associated with the following
object store NAS buckets: "amazon-fsx-<volume-id>"

What was checked	Result
Where the bucket name comes from	An exact match for the volume ID. It is created per volume, not per access point
After deleting every access point	Still refused, and still refused hours later
Bringing the volume back online and mounted	Still refused
Re-attaching an access point and detaching it in the correct order	Still refused. It is independent of the access point lifecycle
`aws fsx delete-volume`	Succeeded. Both the volume and the bucket went

NetApp's documented procedure for removing a NAS bucket configuration uses vserver object-store-server bucket delete, which could not be applied here because the target could not be listed.

Whether this residue is by design or a defect is unconfirmed. No AWS documentation I could find describes a NAS bucket outliving its access points and blocking a volume delete; the documented removal path is detach-and-delete-s3-access-point. If you hit the same state, delete-volume on the AWS side will complete the teardown, but ask support to establish the cause.

Not being listed is not the same as not existing. Laid out by reader:

Reader	How it shows this bucket / S3 server
ONTAP REST `/protocols/s3/buckets`	Does not list it
ONTAP CLI `vserver object-store-server bucket show`	Does not list it
ONTAP REST `/svm/svms` (the `s3` field)	Reports the S3 server as `enabled`
ONTAP REST `/protocols/s3/services`	Does not list the same SVM

Objects that AWS manages are hidden from the standard ONTAP S3 views. I confirmed the bucket's absence across two readers, and both shared the same blind spot.

If you have made clones, the teardown order needs one more step of care. delete-volume returned DELETING and then silently went back to CREATED (observed twice). The reason appears only in LifecycleTransitionReason on describe-volumes: Failed to delete volume because it has one or more clones. The clones had been deleted, and were sitting in ONTAP's volume recovery queue. The parent kept clone.has_flexclone at true, and the AWS-side delete reads that flag and refuses. Purging the recovery queue cleared the flag and the same delete-volume worked. That queue appears in neither the console nor the FSx for ONTAP API.

Putting it in a runbook

Translated into operational terms, the measurements come to five points.

#	What to write in the runbook
1	Match the security style to the identity. UNIX identity for UNIX volumes, WINDOWS identity for NTFS ones. Avoid a UNIX identity on NTFS even where the SVM does have a CIFS server
2	Do not use `AVAILABLE` as a connectivity check. Make the acceptance criterion a PUT and a read-back after attaching
3	Budget tens of minutes into automation that creates a resource in ONTAP and then handles it through the AWS API. Poll for the `fsvol-` identifier rather than sleeping a fixed interval
4	Write through the AWS management plane. Where a setting can be made from either side, as with the junction path, the AWS side reflects faster (about 40 seconds against more than two minutes)
5	Tear down through the AWS-side API. A volume that has carried an access point can keep a NAS bucket on the ONTAP side, so a procedure built on ONTAP's `volume delete` gets stuck there (`aws fsx delete-volume` completes)

What the schedule took over

Everything so far was about whether an operation is possible. Operations has another kind of work: checking capacity, taking stock of snapshots, revisiting tiering policy, confirming throughput is still right-sized. All of it is daily or weekly, and a button for it does not get pressed. That work suits a schedule better than a screen, so six patterns live in operations/.

ID	Pattern	What it does
OPS1	capacity-rightsizing	Capacity and throughput monitoring, tier recommendation, what-if cost
OPS2	storage-efficiency	Tracks deduplication and compression efficiency, recommends enabling them
OPS3	tiering-optimizer	Cold-data analysis, tiering policy recommendation and savings estimate
OPS4	snapshot-lifecycle	Retention policy compliance check and drift detection
OPS5	cost-optimization	Cost breakdown, unit economics ($/GB), growth projection
OPS6	qos-monitoring	QoS policy adherence, bandwidth contention risk

The shape is the same in all six. An EventBridge schedule starts a Step Functions workflow, a VPC Lambda collects from the ONTAP REST API and CloudWatch, another analyses, and a third writes the report. Unit tests total 74 across the six (capacity-rightsizing 27, snapshot-lifecycle 19, tiering-optimizer 11, storage-efficiency 6, cost-optimization 6, qos-monitoring 5).

The report has two destinations. With OutputDestination=FSXN_S3AP it is written to an FSx for ONTAP volume through an S3 access point, which means anyone with the NFS or SMB mount reads it in the file explorer they already use. On the real cluster the same 4,240-byte JSON appeared on the ONTAP volume as in the S3 bucket, the Step Functions execution took around 15 seconds, and four metrics were published to the FSxOps namespace.

A recommendation comes out like this.

{
  "recommendation_type": "downsize",
  "target": "vol_backup_temp",
  "current_value": "10.0% (1024 GB allocated, 102 GB used)",
  "recommended_value": "Shrink to 205 GB or enable autosize (grow_shrink)",
  "monthly_cost_delta_usd": -102.4
}

DemoMode=true runs the whole workflow on mock data with no file system, so the report layout and the shape of the recommendations can be reviewed first. Running cost itself — Lambda, Step Functions and S3 storage — came to roughly $1-5 a month for a daily run over a handful of file systems.

Progressive automation — four levels by design, two in the code

Honestly, here. The template's AutomationLevel accepts 0 through 3.

Level	Intended behaviour	Implemented
0	Report only	Yes (default)
1	Notify over SNS	Yes
2	Execute after human review	No
3	Auto-execute within guardrails	No

The code branches in exactly one place, automation_level >= 1, deciding whether to send SNS. So 2 or 3 deploys successfully and behaves identically to Level 1. The values the parameter accepts and the values the code distinguishes do not match.

The values listed in AllowedValues are not a declaration of what is implemented — accepting a value and honouring it are different things. It is the same point Part 4 made about not shipping buttons that cannot be pressed, showing up on the parameter side instead. Until the approval flow exists, narrowing AllowedValues to 0 and 1 would be the accurate move.

None of this replaces existing tooling. The vendor's CloudWatch dashboards and auto-resize solutions, and the SSD capacity auto-expansion AWS documents, differ in immediacy and in scope. The six here suit a flow where analysis and a recommendation come first and a person decides. Choose on the use case.

On how recommendations are treated: OPS4's snapshot deletion recommendations exclude anything below the minimum retention. A recommendation is an assistive signal, not a final decision. Which retention period satisfies your regulatory requirements is a legal and compliance judgement, and neither this article nor the pattern stands in for it.

What I did not measure

"Unverified" is not "cannot be done". Every result above turned out that way, so on this path nothing is known until it is measured.

Item	State
FlexClone at LUN granularity	Unverified. Creating a LUN needs an iSCSI configuration, which is not on this path
The FlexClone split	Unverified. Documentation only
An upper bound on the ONTAP-to-AWS reflection time	Unverified. The three observations disagree
FabricPool tiering	Unverified. Needs a different aggregate configuration
QoS / deduplication and compression / SnapMirror / Vscan	Unverified
SnapLock / Object Lock	Unverified. Irreversible, so not enabled without an instruction naming the retention value
Attaching on a FlexCache cache volume	Unverified. Requires a cluster peer and an SVM peer
OPS Levels 2 and 3	Not implemented. The approval flow and auto-execution are not in the code
Whether the OPS recommendations are sound	Not evaluated. They are produced; none was applied and measured
OPS across many file systems	Unmeasured. Only a handful were run

Conclusion

Two things were left off the screen, and both were written up here.

On the ONTAP side I took NetApp's table as the starting point and measured on the S3 access point path. The four items it lists as unsupported did not appear as restrictions here. Qtrees work, a quota refuses an S3 PUT, FlexClone works at both volume and file granularity, a FlexGroup takes an access point, and a clone of that FlexGroup can be created. The constraints were elsewhere: AVAILABLE says nothing about the file system layer, something created in ONTAP takes tens of minutes to appear on the AWS side, and a volume that has carried an access point can only be deleted from the AWS side.

On the operations side, the daily work went to a schedule rather than a screen. There is one mismatch between the declaration and the code here: AutomationLevel accepts 0 through 3 while the code distinguishes only up to 1.

Every failure in these six parts looked like a correct conclusion on its own. A conclusion is not settled until one more control, or a second reader, is added.

Thank you for reading these six parts. It started with showing files in a browser and ended outside the screen entirely. Part 7 picks up the layer underneath the portal: which tooling to build it on, and what a stack deletion leaves behind. I hope the series is useful to anyone operating FSx for ONTAP, or building this kind of admin screen.

All test resources were deleted. The figures come from one specific environment and configuration and will differ with workload and setup. This article does not stand in for governance or regulatory judgement.

FlexGroup creation and capacity rebalancing failed on the defaults — what the error codes said (File Portal Part 5)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Fri, 04 Sep 2026 00:59:05 +0000

Introduction

Part 4 described putting 182 ONTAP operations on a file portal: the design of delegation and record, and the treatment of buttons that cannot be pressed and buttons that were never built.

This part is what the running cluster refused while those 182 were being built. None of it was visible while reading the documentation; each one arrived as an error code. That is the difference from last time: Part 4 was about design, this one is about measurement.

The readers I have in mind are people writing code against the FSx for ONTAP ONTAP REST API. You do not need to be building a portal — if you call the same API, you stop in the same places.

What this part covers: FlexGroup creation, FlexGroup capacity rebalance, how to read the capacity numbers, deletes that refuse, and four pitfalls in the S3-compatible API. Each with the response the system actually returned.

What this part does not cover:

ONTAP internals
Absolute throughput figures (the subject here is whether something is possible and under what condition — but the defaults that decide whether a measured figure can be trusted are covered in a section below)
Running SnapLock compliance (irreversible, and it would make a test file system undeletable for months, so it was deliberately not done)

Up front:

Creating a FlexGroup through the ONTAP REST API requires naming the target aggregate. FSx for ONTAP aggregates have tiering enabled, so ONTAP's automatic placement skips them. Creating one from the AWS console or the Amazon FSx API does not need this step
The same root cause is documented on the FlexCache side as a flag (use_tiered_aggregate), and it had not reached the volume-creation side
The capacity rebalance max_runtime has a lower and an upper bound that are not in the API reference. With ONTAP's defaults it never starts
Starting a rebalance enables an irreversible setting. Stopping it does not undo it
space.used means a different quantity depending on the volume. Snapshots inside the reserve are not counted
A deleted FlexClone child volume stays invisible to the API while still blocking the parent volume's delete. There are two ways out, and both were measured
Two places the documentation and the running system disagreed, and both changed the screen's wording
Ten combinations of defaults return a plausible figure while measuring something other than what you intended. Three of them come from reads being served out of cache, two from when you measured, and one is not a setting at all but a way of measuring (a single client at low concurrency)

Test environment

Item	Value
Dates	mid-August 2026 to 2026-09-02 (JST)
Region	ap-northeast-1
ONTAP version	9.18.1P3D1
Deployment type	SINGLE_AZ_1
Caller	ONTAP REST API from a VPC Lambda (Python 3.13)
Privilege	`fsxadmin`
Cleanup	All test volumes deleted

The distinction that matters most — ONTAP's words and AWS's words

Mix these up and searching for the error text will not find the answer.

Term	Where it appears
FabricPool, `use_tiered_aggregate`	ONTAP error text and REST field names
tiering, capacity pool tier	AWS documentation and console

They refer to the same mechanism. But the error arrives in ONTAP's vocabulary while the documentation explains it in AWS's, so searching the AWS documentation for the error string returns nothing. The portal's wording follows the AWS side; only API strings are quoted as-is.

What to watch when creating a FlexGroup

A FlexGroup presents several "constituents" — internally split real volumes — as one namespace.

To set the scope: FlexGroup volumes can be created from the AWS console, the AWS CLI and the Amazon FSx API (Managing volumes). What follows is the ONTAP REST API path, which is the one the portal uses.

ONTAP normally picks the aggregates if you do not name them. On FSx for ONTAP that automatic selection does not resolve.

Aggregates not matching FabricPool requirements: aggr1

An FSx for ONTAP file system has a primary storage tier (SSD) and a capacity pool storage tier, and infrequently accessed data is tiered to the latter (AWS documentation). Aggregates in that configuration fall outside ONTAP's automatic placement.

The conditions came out one failure at a time.

Step	Result
Create with default parameters	Failed. `Volumes of this type must be at least 50GB`
Retry at 50 GiB	Failed. `Aggregates not matching FabricPool requirements: aggr1`
Retry naming the aggregate	Failed. `Minimum size is "400GB"`
400 GiB, aggregate named, `tiering.policy=none`, thin	Succeeded

Naming the aggregate works. But that aggregate name appears in neither the AWS console nor the Amazon FSx API. It has to be fetched with GET /storage/aggregates. Nobody hands the tenant an aggregate name, so that extra call is mandatory.

The 400 GB floor is configuration-dependent: the documented minimum is 100 GB per constituent, so the floor moves with the constituent count (the default is 8 per HA pair). And this extra step is only needed on the ONTAP REST API path — if you do not need to choose the aggregate yourself, creating the volume from the Amazon FSx API is shorter.

The same root cause was already known on the FlexCache side. There is a flag, use_tiered_aggregate, defaulting to false. It had not reached the volume-creation side. The same cause surfacing elsewhere with a different face is a familiar shape in a managed service that wraps upstream software.

The screen carries this note:

When creating a FlexGroup from this screen, the aggregate is named explicitly rather than placed automatically. FSx for ONTAP aggregates have tiering enabled and are excluded from automatic placement (measured: without it, creation fails with "Aggregates not matching FabricPool requirements"). With the default four constituents the minimum is 400 GB.

The capacity rebalance that never starts with the defaults

A FlexGroup places files across constituents by hash, so they drift apart over time. When one fills, the whole volume returns "no space" even though the others have room. A capacity rebalance is what corrects that.

Running it against the real cluster produced two constraints on max_runtime that are not in the API reference.

Constraint	Detail	Error code
Lower bound	Under 30 minutes is refused	`144182221`
Upper bound	Must be shorter than the time remaining until the next snapshot	`13107433`

Together, the window to start is:

30 min <= max_runtime < (time from start until the next snapshot)

ONTAP's default max_runtime is 6 hours. On a volume with the default snapshot policy (hourly at :05), 6 hours always violates the upper bound. With the value the portal offered as its default, it never started once. With the default policy the only window is the 30 minutes from :05 to :35 each hour. On a frequent schedule such as 5min, nothing starts until the policy is detached.

The boundary was pinned with a one-second A/B.

Time	`maxRuntime`	Until next snapshot	Result
16:06:12	`PT1H` (60 min)	58m47s	Refused (`13107433`)
16:06:13	`PT30M` (30 min)	58m46s	Succeeded

60 minutes > 58m47s refused; 30 minutes < 58m46s succeeded. Because the two calls are one second apart, the deciding factor is the max_runtime value and not elapsed time. ONTAP's own message names both remedies — "reduce the -max-runtime or disable the snapshot policy" — because of this structure.

The full observation log is in the FlexGroup capacity rebalance record. State values absent from the reference's volume-side list turned up there too: idle means running with nothing to move, scheduled means a reservation exists. The portal had been displaying "running" as "unknown".

A note on irreversibility

Starting a rebalance enables granular data on that volume. It is required for the rebalance, and it creates two multi-part inodes per moved file (ONTAP documentation). Stopping does not undo it. The vendor's documentation lists no way back other than deleting the volume or restoring from a snapshot taken before it was enabled.

Stopping is not a rollback either: files already moved stay where they are. Both were measured (granular_data remains true after a stop). The screen's confirmation says both.

The space that does not come back after a delete

Read the usage figure in a volume list naively and you will get it wrong, because space.used means a different quantity depending on the volume.

Snapshots that fit inside the snapshot reserve (5% by default) are not counted in space.used
Anything past the reserve consumes the same space as live data and is counted

The measurements. A 100 GiB volume reported used of 18.1 MiB while holding 77.3 MiB of snapshots. That sits inside the 5% reserve (5 GiB), so the usage bar does not move. On a different volume with a 0% reserve, used was 83,677 MiB: 81,934 MiB of live data and 1,743 MiB of snapshots. On 8 of 11 volumes, snapshots exceeded live data.

So the list was split from one usage figure into three: live data, snapshots, and reserve spill. Whether the space is not coming back because of snapshots or because live data fills it can now be read off the screen.

Sometimes the cause is on another volume and its own row does not show it. That is FlexClone. A clone is based on one of the parent's snapshots, and that snapshot stays locked while the clone exists. Deleting files on the parent frees nothing while the locked snapshot still references the blocks. Reading the parent's row alone cannot tell you that, so it is written on the FlexClone panel instead.

What to check when a FlexClone volume will not delete

After deleting a child volume created with FlexClone, deleting the parent volume is refused with has one or more clones. I reproduced that on real hardware. The awkward part is that the API cannot see that child volume. It is not in the volume listing, fetching it by name returns entry doesn't exist, and yet only the parent's delete refuses, saying a clone exists.

The cause is ONTAP's volume recovery queue, which retains deleted volumes for 12 hours by default. An A/B in the same environment, varying only whether the clone was split first:

Steps	Parent delete
Delete an unsplit clone, then delete the parent	Failed (still failing after 7 and 15 minutes)
Split the clone, delete it, then delete the parent	Succeeded within seconds

The exit was measured too. GET /api/private/cli/volume/recovery-queue reads the queue. POST .../purge goes through with fsxadmin privilege, the entry left the queue in about 20 seconds, and the parent delete succeeded immediately afterwards.

This is a place where I published something wrong and have corrected it. The earlier text said purge needs diag privilege, is unreachable as fsxadmin, and therefore the only option is to wait 12 hours. That was wrong. I blamed privilege without trying it.

The screen's error carries the causation and both exits (split first, or purge). Purge cannot be undone, so it also says to confirm the volume was yours before running it.

Four pitfalls in the S3-compatible API

Behaviours worth knowing before you call an FSx for ONTAP S3 access point from a browser or an SDK. Knowing them changes how long the investigation takes.

The 501 failure of a "write only if absent" request

PutObject with if-none-match: * returns 501 NotImplemented, with the body A header you provided implies functionality that is not implemented.

Request	Result
`PUT` + `if-none-match: *`	501 NotImplemented
`PUT` + `x-amz-checksum-crc32` (no `if-none-match`)	200 OK
`GET` / `ListObjectsV2`	200 (neither header is sent)

CRC32 flexible checksums pass. Only if-none-match fails. Reads are unaffected, so the symptom presents as "the listing works but every write fails."

This is not somebody else's problem. Amplify's Storage Browser translates its "do not overwrite" option into that header, so dropping it in unchanged breaks every upload and folder creation. If you need overwrite protection, look the key up before writing instead. That does not give you if-none-match's atomicity, and two concurrent writes to the same key can both decide it is absent.

GOVERNANCE and COMPLIANCE, indistinguishable from the error text

S3 Object Lock has two retention modes. GOVERNANCE lets a user holding a specific permission (s3:BypassGovernanceRetention) delete inside the retention period; COMPLIANCE lets nobody delete until the retention expires. Deleting an object written with one day of retention, compared across both modes:

Operation	GOVERNANCE	COMPLIANCE
Delete without bypass	`AccessDenied ... object protected by object lock`	Identical text
Delete with `BypassGovernanceRetention`	Succeeds	Refused with identical text

Because the text is identical, the response does not tell you why the bypass did not work. The first things to read are get-object-lock-configuration and head-object's ObjectLockMode rather than the IAM policy, because the same text covers both a missing permission and a mode that has no bypass.

The lock indicator that does not change under automatic locking

Assigning a policy whose schedules carry a retentionPeriod leaves the locking feature flag at false; only the assigned policy name changes. Reading the state alone suggests nothing will be locked, while every snapshot that policy takes is in fact locked. The judgement needs the assigned policy's retentionPeriod as well.

The usable endpoints, derived from the API rather than a config file

An alias copied into a config file looks equally correct for an access point that has been deleted and for one that is MISCONFIGURED. fsx describe-s3-access-point-attachments returns Lifecycle and the Internet or VPC origin, so derive the inventory from there. Always follow the pagination: read only the first page and an access point that exists goes missing in a way indistinguishable from one that does not.

Two places the documentation and the running system disagreed

Two things behaved differently from what I read before implementing them. Both changed the screen's wording.

Subject	Expected from the documentation	Measured (ONTAP 9.18.1P3D1)
Deleting a QoS policy while assigned	The CLI reference says it is refused without `-force`	REST accepts it, and the volume's assignment is silently removed (every ceiling becomes 0, meaning unlimited)
Enforcement after deleting a quota rule	The REST reference says it stays in effect until enforcement is cycled off and on	The deleted rule's ceiling disappeared from the usage report immediately

The QoS confirmation text was rewritten. The correct warning is not "it cannot be deleted" but "deleting it removes the ceilings from every volume using that policy". A note was added that releasing a single volume means assigning none, not deleting the policy.

The ten defaults that can invalidate a measurement

Up to here the defaults decided whether an operation goes through. Measurement has defaults of the same shape. None of them returns an error: each returns a plausible figure while measuring something other than what you intended.

Where these figures come from: a different measurement from the environment above. 2026-09-01 to 02, ap-northeast-1, ONTAP 9.18.1P3D1, SINGLE_AZ_1, throughput tiers 128 MBps and 2048 MBps, client c5n.9xlarge. Repeatability across repeat runs of the same measurement was 0.02–0.14%.

The sixth is from a different measurement. 2026-09-06, ap-northeast-1, ONTAP 9.18.1P3D1, SINGLE_AZ_2, throughput 6,144 MBps, an SMB SVM joined to AWS Managed AD, clients c5n.2xlarge and c5n.9xlarge. A different generation and a different tier from the five above, so do not put the figures side by side.

The sixth does not record the NVMe read cache state. The second generation at 6,144 MBps has 1,900 GB of NVMe read cache and it ships enabled (confirmed on a live file system on 2026-09-10). The difference Multichannel makes is a channel count and is unaffected by this gap, but do not cite 942 and 1,824 as disk-path figures.

The seventh and eighth are from a third measurement. 2026-09-10, ap-northeast-1, ONTAP 9.18.1P6, SINGLE_AZ_2, 1,536 MBps and 6,144 MBps, 8,192 GiB of SSD, client c5n.9xlarge. The NVMe read cache state is part of the recorded conditions. It ships enabled at 6,144, so it was disabled on both nodes and read back before measuring.

The ninth and tenth are from a fourth measurement. 2026-09-19/20, ap-northeast-1, ONTAP 9.18.1P6, SINGLE_AZ_1, provisioned 2,048 MBps, SSD IOPS 40,000, NVMe read cache disabled, an 800 GiB volume, 300 GiB read once, client c5n.9xlarge. It differs from every item above in generation, provisioned value, or version, so do not put these figures side by side with the others either.

Default	What happens	How to notice	What to do
Linux NFS uses one TCP connection	Stops around 590 MB/s and does not respond to stream count	1 / 4 / 8 give the same figure	`nconnect=16`
`dd if=/dev/zero`	Zero blocks skip disk and return at 4x the published ceiling	Exceeds the published ceiling	Use incompressible data
Volume inline efficiency is on	Compressible or identical payloads collapse (4x on reads, under 5% on writes)	`space_savings.dedupe_percent` is high	Turn it off before writing
`DiskIopsConfiguration: AUTOMATIC`	3 IOPS/GiB, so IOPS becomes the limit	MB/s divided by IOPS is implausibly small	`USER_PROVISIONED`
A read only slightly over the cache size	The cache-served share dominates and the disk path is not measured	`DiskReadBytes` divided by `DataReadBytes` is a few per cent or less	Read at least twice the cache in one pass
SMB Multichannel ships disabled	One channel, and it holds while the session reports `dialect=3.1.1` (1 MiB sequential read 942 against 1,824 MB/s)	One established TCP connection to the SVM; `Get-SmbMultichannelConnection` returns nothing	`vserver cifs options modify -is-multichannel-enabled true`. It does not reach existing connections — restart the client's SMB service and remap
No window length decided	Measures while the burst allowance is still there. 2.0x on the same environment and the same shape (2,882 MB/s for 27 minutes, then 1,439 MB/s)	`FileServerDiskThroughputUtilization` is above 100%. It is a ratio against baseline, so it can be (median 203.5% measured)	Run until the allowance is gone. Read `FileServerDiskThroughputBalance` over the same window and record the balance
Reading straight after a fill	Competes with the fill's write-back and runs about 20% low for 90 seconds	Disagrees with the same shape measured a second time (39% measured)	Leave two minutes after the write finishes. Put the wait in the parameter file, not in a note in the runbook
`tcp-max-xfer-size` ships at 64 KiB	Measures at a truncated transfer size. A mount that requested `rsize=1048576` succeeds but still runs at 64 KiB	`findmnt -no OPTIONS <mnt>` still shows `rsize` at 65536	Raise it to 1,048,576 on the SVM side, then remount (it is fixed at connect time). The same measurement gained +21%
A single client at low concurrency	Measures that setting's value, not a ceiling (measured 1,195.27 → 1,446.66 at 1 MiB transfer size → 1,646.17 at 16 threads)	None of the utilization metrics are saturated (disk throughput 63–77%)	Raise transfer size and concurrency, or add clients. Both reached the same ceiling in this measurement (1 client at 16 threads: 1,646.17; 2 clients at 8 threads: 1,901.48; both at 102.6% disk throughput)

The tenth is different in kind from the rest. The other nine go away once you change a setting; this one is a procedure. The fix is either "raise transfer size and concurrency" or "add clients", and in this configuration both reached the same ceiling (1 client at 16 threads: 1,646.17; 2 clients at 8 threads: 1,901.48; both at 102.6% disk throughput utilization).

Do not write "raising concurrency is enough", though. On a different configuration (second generation, 6,144 MBps / 200,000 IOPS, 4 KiB random reads), doubling threads moved the figure by only +8.9%, adding SVMs moved it +26.3%, and adding clients moved it +52.7%. Which lever works depends on the configuration. What is common is only this: do not cite a value measured on a single client as a ceiling (measured 2026-09-20).

How the fifth one surfaced. I assumed 280 GiB would not fit in cache. Read through Amazon CloudWatch's AWS/FSx namespace, DiskReadBytes divided by DataReadBytes came to 1.4 / 0.9 / 1.5%. 98.5-99.9% of the bytes read never touched disk.

The 2048 MBps tier's in-memory cache is 256 GB, that is 238 GiB (FSx for ONTAP performance; ap-northeast-1 falls under the first-generation Single-AZ "all other Regions" table). 280 GiB exceeds it by 18%, not by a factor of two or three. "Past the cache" and "not served from the cache" are different things.

And clearing the cache by a wide margin does not help if the window is short. In the seventh measurement the working set was 30 times the in-memory cache, and the figure still came back 2.0x high for the 27 minutes the burst allowance lasted. "Read at least twice the cache" answers the fifth, not the seventh.

Inline efficiency can be restored, provided you wait.

# Rejected unless efficiency.op_state is idle
PATCH /api/storage/volumes/{uuid}  {"efficiency": {"compression": "inline"}}
PATCH /api/storage/volumes/{uuid}  {"efficiency": {"dedupe": "both"}}

But only turn it off on a volume you can discard. Immediately after writing, a background efficiency operation is running and the wait is not predictable.

Putting it in a runbook

#	What to write down
1	To create a FlexGroup through the ONTAP REST API, fetch the aggregate with `GET /storage/aggregates` and name it. If you do not need to choose the aggregate, create the volume from the Amazon FSx API instead
2	Set the rebalance `max_runtime` to at least 30 minutes and below the time to the next snapshot. Do not pass the default through
3	Do not treat starting a rebalance as an operation you can press at any time. `granular data` is enabled irreversibly
4	Do not show usage as a single number. Separate live data from snapshots, or nobody can reach the cause of "deleting files did not free space"
5	A teardown procedure for an environment with clones must include the split, or the recovery queue purge. "The clone was deleted, so the parent can go" does not hold
6	Errors arrive in ONTAP's vocabulary. Keep the AWS-to-ONTAP term mapping at hand before searching
7	A runbook for measuring performance states how to drop the defaults first. Leave any one of zero-filled data, inline efficiency, an unspecified `nconnect` or `AUTOMATIC` IOPS in place, and the figure produced is measuring something else

What I did not measure

"Unverified" is not "impossible". As last time, the boundary stated as a boundary:

Item	State
The actual effect of a rebalance (how far the skew is evened out)	Unmeasured. Only the conditions to start were measured
Performance impact after `granular data` is enabled	Unmeasured
A recovery queue with a changed retention	Unverified. Measured at the 12-hour default
Actual tiering into the capacity pool	Unverified. Needs a different aggregate configuration
Running SnapLock compliance	Deliberately not done. Irreversible, up to the file system
An alternative path for copies over 5 GiB (NFS or SMB)	Unmeasured. The portal stops at refusing

Part 6 covers the other side: what was left off the screen. How much of ONTAP stays usable alongside the S3 access point path, and the operational work that was handed to a schedule instead of a button.

Conclusion

Nothing in this part was visible while reading the documentation. Creating a FlexGroup through the ONTAP REST API needs the aggregate named, a capacity rebalance never starts with the defaults, space.used means a different quantity per volume, and a deleted FlexClone volume blocks the parent's delete while invisible through the API.

What they share is that the error text does not name the cause. That is why the causation and the exits live on the screen. As with the one-second A/B on the rebalance, another explanation looked just as correct until one control was added.

There is also one correction. I published "purge needs diag privilege, so wait 12 hours" and later found it goes through. I blamed privilege without trying it.

The figures come from one specific environment and configuration and will differ with workload and setup. The error codes were observed on ONTAP 9.18.1P3D1.

Putting FSx for ONTAP operations in a browser — the buttons I disabled, and the ones I never built (File Portal Part 4)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Fri, 04 Sep 2026 00:57:42 +0000

Introduction

Part 1 built the foundation of a file portal on top of Amazon FSx for NetApp ONTAP (FSx for ONTAP) S3 access points. Part 2 added ransomware response and WORM retention. Part 3 put AI agents on it.

Those parts were screens for people who handle files. From here the subject is the people who operate the storage itself. Volume create and resize, FlexCache, FlexClone, SnapMirror, snapshot policies, QoS, quotas, SMB shares. The portal now reaches 182 operations.

The readers I have in mind are infrastructure owners running FSx for ONTAP, and developers building this kind of admin screen with Amplify Gen2. ONTAP terms get a one-line gloss on first use, so ONTAP operations experience is not assumed. You can start here without Parts 1-3.

What this part covers is what had to be decided to put operations on a screen: the design of delegation and record, and the treatment of buttons that cannot be pressed and buttons that were never built.

What this part does not cover:

The ONTAP features themselves (ONTAP's own documentation is more accurate)
The failures the real cluster produced (that is an article's worth on its own, so it became Part 5)
Which retention period satisfies your regulatory requirements (a legal and compliance judgement that neither I nor the portal can stand in for)
Performance and scale limits of the screen (unmeasured; no concurrent-user test was run)

Up front:

Three management interfaces reach the file system, and two of them are for administrators inside the VPC. What the portal adds is not a path but delegation and record
The portal does not remove the path into the VPC. The path did not disappear; what holds it moved from a person to a Lambda
The number 182 is not maintained by hand. A script counts it and CI fails when it disagrees
A button that cannot be pressed says what is missing. A button that does nothing is indistinguishable from a broken one
The confirmation screen for an irreversible operation states what becomes undeletable, for how long, and what it costs meanwhile, because the effect does not stop at the resource named in the call
After writing this article I ran the screen and found the lock indicator was broken. While only one panel was being read, it looked correct

The repository is solutions/amplify-portal/.

Test environment

Item	Value
Date	2026-09-02 (JST)
Region	ap-northeast-1
ONTAP version	9.18.1P3D1
Portal	Amplify Gen2 sandbox: AppSync + Cognito + VPC Lambdas
Lambda runtime	Python 3.13
How the action count was taken	`scripts/portal_action_types.py --check` and `scripts/check_portal_action_params.py`
Cleanup	Test sandboxes deleted. One working stack retained

The distinction that matters most — a path that persists, a holder that moved

First, the premise. Without routing through an additional third-party SaaS, three management interfaces reach FSx for ONTAP.

Interface	Path	Reach
AWS Management Console / FSx API	IAM	The AWS-side operations on file systems, SVMs and volumes; backups
ONTAP CLI (SSH)	SSH to the management endpoint	Roughly an ONTAP cluster administrator
ONTAP REST API	HTTPS to the management endpoint	The CLI's operations, programmatically

The sources, and how anything outside this list is treated, are in Management interfaces. The CLI and REST management endpoints are reachable only from inside the VPC or through a Transit Gateway peered network.

One consequence follows directly. Consider an everyday request: add a volume, or raise one quota. Answering it needs fsxadmin-equivalent credentials and a path into the VPC. So every request lands on the infrastructure owner, and none of it can be handed to anyone else.

That separation is what the portal does.

Delegation. A Cognito group (storage-admin) decides who may run what, per operation. No SSH key and no ONTAP credential is handed out. The credential lives in Secrets Manager and only the Lambda reads it
A record. Who ran which operation is retained with the Cognito principal. Unlike an SSH session on a shared account, it attaches to a person

The portal does not remove the path into the VPC: the Lambda that calls the ONTAP REST API sits inside it. The path did not disappear; what holds it moved from a person to a Lambda. That distinction affects operational design, so the repository documentation words it the same way.

The reach of the screen

The portal reaches 182 actions across 12 endpoints. That number is not maintained by hand. A script reconciles the generated types against the handlers and prints it.

$ python3 scripts/portal_action_types.py --check
DISPATCH ACTION TYPES: PASS (182 actions declared, matching the handlers)

$ python3 scripts/check_portal_action_params.py
PORTAL ACTION PARAMS: PASS (187 literal call sites, 182 actions
across 12 endpoints, 1 call(s) not statically readable)

The reason it is not counted by hand is not that the number grew. Types do not cross the boundary between the screen and the Lambda. The portal's dispatch takes an untyped params, so code that sends {snapshotName, retentionDays} to an action reading snapshotId and expiryTime compiles, lints, renders a button, and fails on every click. CI now fails until the reconciliation agrees.

By domain.

Domain	Operations
Volume lifecycle	Create / resize / delete / bring online, for both FlexVol and FlexGroup
FlexGroup capacity rebalance	Read state / start / stop / schedule
FlexCache	Create / list / resize / write-back toggle / delete
FlexClone	Create / list / split
SnapMirror	Create / list / transfer history / update now / quiesce / resume / break / resync / abort / delete
Snapshots	List / create / delete / lock / policy create / assign / enable locking
SnapLock and WORM	Read configuration / change retention, S3 Object Lock retention
ARP/AI (ransomware protection)	State change / bulk enable / suspect list and clear / surge parameters
Containment	Block NFS IP / block SMB user / disconnect sessions / list active blocks
QoS	Policy create / update / delete / assign to and release from a volume
Quotas	Rule create / update / delete / usage report / per-volume enforcement
SMB	Share create, update, delete; local users and groups; members; name mapping
Qtrees	Create / update / rename / delete
Export policies	Policy and rule create / delete
FPolicy and Vscan	Policy and event create / enable / delete
Peering	Cluster peer and SVM peer create / accept / delete, intercluster LIFs
Cluster inventory	Nodes / licenses / LIFs / protocols / DNS / jobs, EMS events
S3	Bucket list, Object Lock status and retention

182 is not every ONTAP operation. Operations that need advanced privilege are absent from the REST API, so they cannot appear on this screen.

The plain statement of what cannot be done

The copy limit is the clearest case. It is easy to confuse with the upload limit, so the measured values are split out.

Operation	Limit	Measured
Upload (single `PutObject`)	5 GiB	`MaxSizeAllowed` = 5,368,709,120
Upload (multipart, whole object)	50 GiB	53,687,091,200. Checked only at `CompleteMultipartUpload`
Download (`GetObject`)	No limit	Objects over 50 GiB were retrieved
Copy (single `CopyObject`)	5 GiB	As above
Copy (`UploadPartCopy`)	Unusable	Documented as Supported; every form returned `NoSuchKey`

Uploads do reach 50 GiB (measured with multipart). Copies, though, have no path past 5 GiB. UploadPartCopy is listed as Supported in Access point compatibility and returned NoSuchKey in every form measured. Rename, move, copy, trash and restore in the portal are all CopyObject, so that 5 GiB becomes the screen's limit.

The units caught me out as well. The AWS documentation's "5 GB" and "50 GB" both turned out to be binary when measured. MaxSizeAllowed from PutObject is 5,368,709,120 = 5 × 1024³, not the decimal 5,000,000,000. The detail is in the object size limit verification.

Rather than surfacing an S3 error halfway through, the portal checks the size before the copy and refuses with a reason.

{key} is 6.2 GiB. A single copy is limited to 5 GiB, and the multipart copy that would lift the limit is not usable on this Access Point, so this operation cannot be completed from the portal. Move the file over NFS or SMB instead.

The guard's 5 GiB threshold is the same value as the measured MaxSizeAllowed.

On the same reasoning, no button was built for converting a FlexVol to a FlexGroup. The conversion is ONTAP CLI advanced-privilege only, with no REST equivalent. Instead the screen carries the prerequisites, the irreversibility, the treatment of existing snapshots, and what AWS recommends — a copy into a new FlexGroup with AWS DataSync rather than an in-place conversion. Writing down that something is absent, and why, is more useful than a button that cannot be pressed.

The free-space estimate for a clone split (volume clone split estimate) is the same. It exists only in the CLI, so it cannot appear here. Rather than omitting it silently, the screen says why it is missing.

What the confirmation screen states for an irreversible operation

SnapLock and snapshot locking cannot be undone. "Are you sure?" on its own does not tell the person pressing it what becomes undeletable, so the screen states this:

The SnapLock type (compliance) can only be set at creation and cannot be changed or removed afterwards

Under compliance, nobody can delete until retention expires. Not the account administrator, and not AWS

While one unexpired WORM file remains, this volume, its SVM, and the file system cannot be deleted

While the file system cannot be deleted, it keeps billing

A newly created empty volume can still be deleted. It stops being deletable when the first file is committed to WORM

The last two lines are the point. The effect of an irreversible operation does not stop at the resource; it propagates to the parent and to the bill. A test environment is the worst place to put a file system you cannot delete for six months.

Snapshot locking is written up with its asymmetry intact. A lock can only be extended, never shortened or released. The policy side is reversible: removing the retention, switching policies, or detaching stops new locks. But snapshots already locked stay until they expire. Compressing that asymmetry into one sentence makes one half of it wrong.

A retention count does not act as a ceiling either. Locked snapshots cannot be rotated out, so they accumulate past the count until expiry and consume the space. That is on the screen too.

Irreversible operations carry a type-to-confirm (the keyword typed in capitals). The API also requires acknowledgeIrreversible, so a call that bypasses the UI passes the same check.

What is written here is the technical blast radius of an operation. Which retention period satisfies your regulatory requirements is a legal and compliance judgement, and neither this article nor the portal stands in for it.

What I did not measure

"Unverified" is not "impossible". The boundary of this article, stated as a boundary:

Item	State
Behaviour under concurrent users	Unmeasured. Only single-operation latency was observed
Operations needing advanced privilege	Out of scope. Absent from the REST API, so they cannot be on the screen
Enabling SnapLock compliance from the screen	Deliberately not done. Irreversible, and it would make a test file system undeletable for months
Multi-tier delegation with more Cognito groups	Unverified. There is one tier, `storage-admin`
Long-term retention and search of the audit record	Unverified. The record is kept; no retention design was done

Part 5 covers what the real cluster refused while these 182 actions were being built: why creating a FlexGroup through the ONTAP REST API needs the target aggregate named, why a capacity rebalance never starts with ONTAP's defaults, and the two places the documentation and the running system disagreed.

Conclusion

Putting operations on a screen was not a matter of listing features. Three paths already reach the file system and none of them went away, so what a screen adds is delegation and record, not reach.

Half the screen's job turned out to be saying what cannot be done rather than showing what can. Name the missing input on a button that cannot be pressed, write down absent features with the reason, and give irreversible operations their blast radius. Where a limit is measurable — the 5 GiB copy — using the measured value directly in the guard keeps the screen and the storage saying the same thing.

And one display defect surfaced after the article was written. Until two panels were reconciled, the broken indicator looked correct.

The figures come from one specific environment and configuration and will differ with workload and setup. This article does not stand in for governance or regulatory judgement.

AWS Backup now copies FSx for ONTAP backups across Regions and accounts — a new option, measured Tokyo to Osaka

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sat, 29 Aug 2026 05:02:38 +0000

Update (2026-10)

Since September 2026, AWS Backup logically air-gapped vaults support Amazon FSx for NetApp ONTAP volume backups (What's New). The vault lives in an AWS Backup service-owned account, is always locked in Vault Lock compliance mode, can be shared with a recovery account through AWS RAM, and can be combined with multi-party approval. That gives recovery points a place outside your own account's management boundary, which the "As a ransomware layer" section below did not cover.

Two prerequisites connect to this post. Backups of a file system encrypted with the AWS managed key (aws/fsx) are not copied into the vault; the backup job does not fail but ends with the status "Completed with issues", and the recovery point stays in the standard vault only. So the customer managed key decision, made before the file system exists, applies here too. Only RW volumes are backed up.

I have not tried the vault yet. The prerequisites and how to choose between isolation options are collected from AWS documentation in the logically air-gapped vault guide.

The Tokyo-to-Osaka copy measured in this post (within one account) is unchanged. If you plan data protection before a migration, AWS Storage Assessments treat isolation level (standard vault or logically air-gapped vault) as one of the cost dimensions (AWS Storage Blog).

Introduction

How do you approach disaster recovery and backup for your file systems on AWS? Take FSx for ONTAP and a pair of Japanese Regions as the example: if Tokyo is primary, standing up a file system in Osaka and replicating into it continuously with SnapMirror is probably the first shape that comes to mind for anyone who has run ONTAP, or is about to evaluate it. Plenty of AWS users run exactly that in production, and as a way to hold availability it is still the best answer. RPO comes down from the 60 minutes AWS Backup offers at its shortest to SnapMirror's 5-minute minimum interval, switching between active and standby is breaking the relationship and promoting the destination, and cutting data back the other way is a snapmirror resync in the reverse direction.

The trouble was that "I just want the data sitting on the DR side, without operating a file system there until I actually restore" required the same shape. Amazon FSx for NetApp ONTAP volume backups were created and restored inside the file system's own Region and account, and could not cross a Region boundary. AWS Backup could already copy backups across Regions and accounts for FSx for Windows File Server, FSx for OpenZFS, and FSx for Lustre (AWS Backup extends Amazon FSx support to 5 additional AWS Regions and expands cross-Region and cross-account copy to 14 AWS Regions, April 2026), but FSx for ONTAP was not in that set at the time.

Then on 27 August 2026 that gap closed along the two paths below. One more shape became available to choose from in a backup strategy. The two were announced separately.

Path	What it does	Scheduling	Cross-account
FSx for ONTAP backup copy (`CopyBackup`)	Copies a volume backup to another Region	None. Run it each time	No. Same account only
AWS Backup (policy-based backup management service)	Copies FSx for ONTAP backups to another Region and another account	Yes. Copy rules in a backup plan	Yes. Requires AWS Organizations

So I copied a backup from Tokyo to Osaka, restored it at the destination, and confirmed the content matched. This post covers that procedure, along with the design and operational considerations that came with it.

Up front:

AWS Backup now copies FSx for ONTAP backups to another Region and another account (27 August 2026). The FSx for ONTAP CopyBackup also reaches another Region, but it is same-account and manual.
Moving them to another account needs a CMK. AWS managed keys cannot be used for a cross-account copy, and a file system's KMS key is set at creation — so it is a decision made before the file system exists (below).
What matters is that you no longer need a file system at the destination until you recover. The restore target is still the Region the backup lives in, so creating that file system at recovery time lands on your RTO — 20 minutes, measured (below).
CopyBackup has no scheduler. Recurring copies mean an AWS Backup backup plan or your own automation. The AWS Backup copy and restore were measured too — 6 m 31 s to 8 m 35 s to copy, 16 m 16 s to restore, at 9 MB.
This does not replace SnapMirror. If a minutes-level RPO and a failback procedure are requirements, SnapMirror still is the answer. Replication and copy are different operations and leave different things at the destination (below).
Restore time is proportional to capacity. My 13 minutes was a figure for 9.4 MiB. Restoring 10 TB on a minimum configuration is 22 hours at the floor, calculated from AWS published rates. I could not find a measured figure at 10 TB or above in public sources (below).
Cost inverts with scale. A Single-AZ capacity pool is 47.6% of backup storage ($0.0238 against $0.050, retrieved from the Price List API), so past roughly 4.8 TB a standing destination file system is lower monthly. Choosing to keep one running is not choosing the more expensive option (below).

Scope: within one account, copying a volume backup from ap-northeast-1 to ap-northeast-3, restoring it into a file system there, and verifying the content. Both CopyBackup and AWS Backup were measured.

Out of scope: cross-account copies (they require AWS Backup with AWS Organizations — I show the shape but did not measure it), building or failing back a SnapMirror relationship, durations at production data volumes, and reconciling inter-Region transfer charges against a bill.

Shape of the setup

(Dark theme: backup-copy-cross-region-en-dark@2x.png)

The right-hand side being empty in normal operation is the whole point. With SnapMirror, that right-hand side holds an equivalent file system at all times.

Layer	Day to day	When recovering
Tokyo (production)	file system + RW volume + backups	unchanged, or lost if the Region is affected
Osaka (destination)	the backup copy only. No file system	file system → SVM → restore into a new volume
Billing	Osaka pays backup storage only	Osaka's file system starts billing once it exists
Running the copy	`CopyBackup` each time, or an AWS Backup copy rule	—

On vendor neutrality: this post evaluates FSx for ONTAP backup copies on technical grounds only. The section on existing backup products touches other options, but not in a framing where one is better than another. Each suits a different context, and the goal here is to record what actually works and what does not.

Test environment

Item	Value
Date	2026-08-28 (`CopyBackup` path), 2026-08-29 (AWS Backup path)
Runs	One per step (n=1). Variance was not measured
Interface	AWS CLI v2 and the AWS Management Console. Japanese locale for the `CopyBackup` path, English locale for the AWS Backup path (the setting at capture time). CloudFormation untested
Source	`ap-northeast-1`, first-generation `SINGLE_AZ_1`, 1,024 GiB SSD, 128 MBps
Destination	`ap-northeast-3`, same shape, created for this test
ONTAP version	`9.17.1P7D1`
Source volume	FlexVol RW, 1 GiB, tiering `NONE`, storage efficiency off
Dataset	5 files / 9.4 MiB (nested directory, symlink, UTF-8 filename, one file at `0640`). The AWS Backup run used an equivalent 9,458,747 B set
AWS Backup	one vault per Region, one backup plan (one rule, one copy rule), selection scoped to the single verification volume ARN, existing `AWSBackupDefaultServiceRole`
KMS	Defaults in each Region: `aws/fsx` for the source file system, `alias/aws/backup` (`KeyManager: AWS`) for the AWS Backup vaults. CMKs untested, and this configuration cannot do a cross-account copy (below)
Clients	one `t3.micro` (AL2023) per Region, NFS
Actual spend	~$0.55 for the `CopyBackup` run (destination file system ran 1 h 24 m); under $2 for the AWS Backup run (destination file system ran ~1 h 21 m, two `t3.micro`)
Data used	Synthetic data from `/dev/urandom`. No real or personal data
Not verified	Cross-account copies (covered by the earlier post), CMKs, copying `AUTOMATIC` backups, incrementality, residue from a failed copy, deploying the CloudFormation template, restoring into an AD-joined SVM, reconciling inter-Region transfer charges, AWS Backup restore testing

Do not use these durations as an RTO basis for your environment. They are measured against 9.4 MiB, where fixed overhead dominates.

When this fits

You want a copy of your data in another Region but want to avoid running a file system there
Compliance requires retaining a copy that is logically separated from production
You want an isolated-account layer against credential compromise or a KMS key compromise
Recovery measured in tens of minutes to hours is acceptable
You already take volume backups and only want to add a destination for them
SnapMirror already covers availability, and you want a separate retention layer detached from production

When to look at something else for now

RPO in minutes is required (the guidance figure for backups is 60 minutes)
RTO in single-digit minutes is required (creating the destination file system took 20 minutes)
You need a documented path back to production (backup copies have none — see below)
FlexGroup volumes are in scope (copy is unsupported, and creation failed in my environment — see below)
You need copies spanning China Regions, or cross-account copies in the China Regions (unsupported)

If any of those apply, a SnapMirror-based design fits the requirement better. Running both together is also coherent, covered below.

The distinction that matters most — what changed is where a backup can live

Blur this and the runbook comes out wrong.

Claim	Status
A backup can be stored in another Region	now possible (`CopyBackup` or AWS Backup)
A backup can be stored in another account	now possible, but only through AWS Backup. The FSx for ONTAP `CopyBackup` stays inside one account
A backup can be restored to a file system in a different Region than the backup	no (unchanged)
A restore produces a new volume	unchanged. There is no restore-in-place path

using-backups.html still states that a backup can only be restored to a file system in the Region where the backup is stored. That is not a contradiction — copying increases the set of Regions where a backup is stored.

So recovery in Osaka is three steps:

A copied backup is already in Osaka (this is what you prepare in advance)
Create a file system and an SVM in Osaka
Restore as a new volume

Step 2 lands on your RTO. Not paying for a standby file system means paying for its creation when you need it. That is the trade you are choosing.

How `CopyBackup` and AWS Backup relate

Two different things are called "copy a backup to another Region" here, so it is worth separating them first. They are separate mechanisms. Neither calls the other, and their control planes and storage locations differ.

	FSx for ONTAP `CopyBackup`	AWS Backup
Whose API	Amazon FSx (`fsx:CopyBackup`)	AWS Backup (backup plans / copy jobs)
What it copies	An existing FSx for ONTAP volume backup, which must be `AVAILABLE`	A recovery point AWS Backup took itself
Where the copy lands	As an FSx for ONTAP backup in the destination Region	An AWS Backup backup vault
`BackupType`	`USER_INITIATED`	`AWS_BACKUP`
Scheduling	None. Called each time	Yes. Backup rules and copy rules
Cross-account	No	Yes, with AWS Organizations
Link back to the source	Keeps `SourceBackupId` and `SourceBackupRegion`	Followed through the recovery point ARN

Here is where the two get conflated. You cannot point CopyBackup at an AWS Backup backup vault, and an AWS Backup copy rule does not call fsx:CopyBackup on your behalf. Answering "I want CopyBackup on a schedule" with "use AWS Backup" is right not because the same call can be scheduled, but because the same goal is reachable through a different mechanism.

The restore constraint, on the other hand, is shared. Both paths ask you to pick an existing file system and SVM in the destination Region and both produce a new volume. The AWS Backup restore screen asks for a File system and a Storage virtual machine too (documented). So the benefit — no standby file system while nothing is wrong — holds for AWS Backup as well, and so does the cost: file system creation lands on the RTO either way.

On terminology: AWS Backup's own word is copy. How that differs from replication is the next section.

Replication and copy are different words for different things

Two words get used for "keep the data in another Region": replication and copy. They are different operations, and what sits at the destination differs. Without agreeing on that, a requirements conversation goes past itself.

	SnapMirror replication	AWS Backup / `CopyBackup` copy
What sits at the destination	A volume. A `DP` volume exists on the destination SVM	A backup (recovery point). No volume
When the source changes	It follows. Deltas are sent each schedule	It does not. A point-in-time image; later changes wait for the next backup
To use the destination	Break the relationship and promote it to become writable	A restore is required, and what you get is a new volume
Does the relationship persist	Yes. Having it in place is the normal state	No. Each copy is an independent artefact
What sets the RPO	The replication schedule (down to 5 minutes)	The backup interval (guidance: 60 minutes)
What the destination costs while idle	File system capacity and throughput	Backup storage only
Path back to production	`snapmirror resync` in the other direction	There is none

AWS's own wording splits the same way. The AWS Backup console and documentation call this operation copy throughout — the screens read Copy jobs, Copy rule, Copy type: Manual (screenshots above). The FSx for ONTAP API is named CopyBackup. Replication is the word for mechanisms where the destination exists and tracks the source, as in Amazon S3 Cross-Region Replication.

So "replicate cross-Region with AWS Backup" reads as though a volume at the destination is tracking the source. There isn't one. What is at the destination is a recovery point, and using it takes a restore. The difference lands directly on the RTO: replication promotes a volume that is already there, while a copy starts from creating a file system and an SVM and then restoring.

The reverse mistake is calling SnapMirror a backup. The destination tracks the source, so a file deleted on the source is gone from the destination's current state after the next transfer — what remains is whatever the destination's snapshots still hold. Retaining generations is the job of snapshots or SnapVault, not of the replication itself.

SnapVault is a third mechanism. ONTAP's SnapVault is a relationship that accumulates generations at the destination. It rides the same transport as SnapMirror with a different purpose. The backup copies in this post do not create an ONTAP SnapVault relationship.

Which one suits which requirement is in choosing between this and SnapMirror.

A short glossary

Only the terms this post uses. Skip if you already know them.

Term	What it is	Why it matters here
File system	The billing and performance unit. SSD capacity and throughput are provisioned here	This is what you no longer need at the destination
SVM (Storage Virtual Machine)	A logical server inside a file system, holding the NFS / SMB endpoints and auth config	Required to restore. A file system alone is not enough
Volume	Where data lives, and the unit of backup and restore	A restore produces a new volume
Junction path	Where a volume attaches in the SVM namespace (e.g. `/dr_restored`); the NFS mount path	Specified at restore
FlexVol / FlexGroup	Volume styles. FlexVol sits on one aggregate; FlexGroup spreads across several for capacity and parallelism	FlexGroup backup copies are unsupported
`RW` / `DP`	Volume type. `RW` is read-write, `DP` is read-only and used for SnapMirror destinations	`DP` cannot be backed up. A restoring volume transiently reads as `DP`
Tiering	Moving cold data from SSD to the cheaper capacity pool. `NONE` / `AUTO` / `SNAPSHOT_ONLY`	Affects read latency right after a restore
Snapshot vs backup	A snapshot lives inside the file system; a backup is an independent copy outside it	Only the backup can cross a Region boundary

If you run ONTAP on-premises

ONTAP concept	Relationship to this capability
SnapMirror	A different mechanism. Needs cluster peering and a standing destination file system
SnapVault	A different mechanism. Backup copies do not create a SnapVault relationship
Cluster peering	Not required for backup copies. Operationally this is the largest difference from SnapMirror
Driving it from ONTAP CLI / REST	Backup copies are an AWS-side feature with no corresponding ONTAP CLI command. Use the AWS CLI, API, or console
Trace left on the volume	A backup leaves a snapshot named `backup-<backup-id>` on the volume

Prerequisites and permissions

Prerequisites

Item	Condition
Source backup	must be `AVAILABLE`
Source volume type	`RW` (`DP`, LSM, and FlexCache destinations cannot be backed up at all)
Volume style	FlexVol. FlexGroup copies are unsupported
Partition	Between commercial Regions, between the two China Regions, and between the two GovCloud (US) Regions — but not across those sets
KMS	Encrypted with a key in the destination Region. No parameter needed if the default key is fine

A backup whose source volume has already been deleted can still be copied (measured). That is useful after an accidental deletion.

IAM

The requester must be able to reach the backup in the source Region. Following the shape in the AWS documentation:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CopyBackupAcrossRegions",
      "Effect": "Allow",
      "Action": "fsx:CopyBackup",
      "Resource": "arn:aws:fsx:*:<your-account-id>:backup/*"
    },
    {
      "Sid": "DescribeAndRestore",
      "Effect": "Allow",
      "Action": [
        "fsx:DescribeBackups",
        "fsx:DescribeFileSystems",
        "fsx:DescribeStorageVirtualMachines",
        "fsx:DescribeVolumes",
        "fsx:CreateVolumeFromBackup",
        "fsx:TagResource"
      ],
      "Resource": "*"
    }
  ]
}

The "Resource": "*" on the second statement is a simplification for readability. Narrow it in production. Some Describe* actions do not support resource-level scoping, but fsx:CreateVolumeFromBackup can be limited to the destination file system and SVM ARNs. To pin the destination Region, use the aws:RequestedRegion condition key:

{
  "Sid": "RestrictCopyDestination",
  "Effect": "Deny",
  "Action": "fsx:CopyBackup",
  "Resource": "*",
  "Condition": {
    "StringNotEquals": { "aws:RequestedRegion": ["ap-northeast-3"] }
  }
}

If you have a data residency requirement, put this guard in IAM or an SCP. "We have a rule that says not to" does not hold.

With a customer managed key (CMK), you will additionally need permissions such as kms:CreateGrant and kms:DescribeKey on the destination-Region key. I used the default key, so I have not verified this. If you are building on a CMK, prove it with a small volume first.

Values to replace with your own

Every command below works once you substitute these.

Placeholder	Meaning	How to get it
`<SRC_REGION>`	Source Region	e.g. `ap-northeast-1`
`<DST_REGION>`	Destination Region	e.g. `ap-northeast-3`
`<SRC_VOLUME_ID>`	The RW volume to back up	`aws fsx describe-volumes --region <SRC_REGION> --query 'Volumes[?OntapConfiguration.OntapVolumeType==`RW`].[VolumeId,Name]' --output table`
`<SRC_BACKUP_ID>`	The backup to copy	output of step 1
`<DST_SUBNET_ID>`	Subnet for the destination file system	`aws ec2 describe-subnets --region <DST_REGION> --query 'Subnets[].[SubnetId,AvailabilityZone,CidrBlock]' --output table`
`<DST_SG_ID>`	Security group for the destination file system	Must allow NFS / SMB. Sharing one SG with the client plus a self-referencing rule is enough
`<DST_FS_ID>`	Destination file system	output of step 4
`<DST_SVM_ID>`	Destination SVM	output of step 5
`<DST_VOLUME_SIZE_BYTES>`	Restored volume size, in bytes	Must be at least the data in the backup. Reusing the source `SizeInBytes` is safest. 1 GiB = `1073741824`

Two notes on sizing. The destination file system's SSD capacity must exceed the data being restored — restored data is written to SSD first. And a FlexVol cannot span more than one HA pair. For a large volume, settle the destination SSD capacity and generation (second-generation Single-AZ can add HA pairs) before you start.

Walkthrough — Tokyo to Osaka

1. Create a backup (skip if you already have one)

aws fsx create-backup \
  --region <SRC_REGION> \
  --volume-id <SRC_VOLUME_ID> \
  --tags Key=Name,Value=dr-source Key=Purpose,Value=cross-region-copy \
  --query 'Backup.{Id:BackupId,Life:Lifecycle}' --output table

Wait for AVAILABLE:

aws fsx describe-backups \
  --region <SRC_REGION> --backup-ids <SRC_BACKUP_ID> \
  --query 'Backups[0].{Life:Lifecycle,Pct:ProgressPercent}' --output table

2. Copy it

--region is the destination; --source-region is where the backup is now. This reads backwards at first — the copy is requested against the destination Region.

aws fsx copy-backup \
  --region <DST_REGION> \
  --source-backup-id <SRC_BACKUP_ID> \
  --source-region <SRC_REGION> \
  --copy-tags \
  --tags Key=Name,Value=dr-copy \
  --query 'Backup.{Id:BackupId,Life:Lifecycle,Src:SourceBackupId,SrcRegion:SourceBackupRegion}' \
  --output table

Add --kms-key-id <destination-region-key-arn> for a CMK. Incrementality is conditional on using the same KMS key, so changing keys mid-stream makes the next copy full.

Watch progress:

aws fsx describe-backups \
  --region <DST_REGION> --backup-ids <DST_BACKUP_ID> \
  --query 'Backups[0].{Life:Lifecycle,Pct:ProgressPercent}' --output table

COPYING → AVAILABLE means done. Mine took 7 m 15 s for a 9.4 MiB volume.

Day-to-day operation ends here. Steps 3 to 6 run only when you are recovering.

3. [Recovery] Check which generation the destination Region offers

Check the available deployment types before you need them. If first generation (SINGLE_AZ_1) is available, the minimum throughput is 128 MBps; where only second generation exists it is 384 MBps, which makes standby cost roughly 1.9× higher.

SINGLE_AZ_1 was accepted in Osaka (measured 2026-08-28). This varies by Region — confirm it for your destination before writing the runbook.

4. [Recovery] Create the destination file system

Billing starts here. Even at the minimum, Osaka costs roughly $0.37 per hour, and forgetting it runs about $270 a month. If you are doing this as a drill, open the teardown commands from step 8 first. During my own run I mistyped an option on delete-file-system and polled for 8 minutes while the command had already failed.

This is the 20 minutes that lands on RTO.

aws fsx create-file-system \
  --region <DST_REGION> \
  --file-system-type ONTAP \
  --storage-capacity 1024 \
  --storage-type SSD \
  --subnet-ids <DST_SUBNET_ID> \
  --security-group-ids <DST_SG_ID> \
  --ontap-configuration '{
    "DeploymentType": "SINGLE_AZ_1",
    "ThroughputCapacity": 128,
    "PreferredSubnetId": "<DST_SUBNET_ID>"
  }' \
  --tags Key=Name,Value=dr-restore-target \
  --query 'FileSystem.{Id:FileSystemId,Life:Lifecycle}' --output table

1,024 GiB and 128 MBps are the first-generation minimums. Raise them if your restore exceeds that. A restore pauses when SSD space runs out and resumes automatically once space is available.

5. [Recovery] Create the SVM

aws fsx create-storage-virtual-machine \
  --region <DST_REGION> \
  --file-system-id <DST_FS_ID> \
  --name dr_svm \
  --root-volume-security-style UNIX \
  --query 'StorageVirtualMachine.{Id:StorageVirtualMachineId,Life:Lifecycle}' --output table

For SMB you also need the Active Directory join here. If AD join is part of your recovery, verify separately that the destination Region can reach your domain controllers, and budget the time. My verification was NFS only.

6. [Recovery] Restore

A restore always produces a new volume. There is no in-place path.

aws fsx create-volume-from-backup \
  --region <DST_REGION> \
  --backup-id <DST_BACKUP_ID> \
  --name dr_restored \
  --ontap-configuration '{
    "StorageVirtualMachineId": "<DST_SVM_ID>",
    "JunctionPath": "/dr_restored",
    "SizeInBytes": <DST_VOLUME_SIZE_BYTES>,
    "OntapVolumeType": "RW",
    "TieringPolicy": { "Name": "NONE" }
  }' \
  --query 'Volume.{Id:VolumeId,Life:Lifecycle}' --output table

TieringPolicy: NONE follows the AWS recommendation for consistent read latency after a restore — it assumes you wait for the data to land on SSD before accessing it. Change it if you want the capacity pool.

Mine reached CREATED in 13 m 21 s for 9.4 MiB.

Three conditions drive restore and post-restore performance. Check these when it feels slow:

Condition	Detail
SSD capacity	Restored data is written to SSD first. The restore pauses when space runs out and resumes when it frees up
Generation	Second generation is readable during a restore (after metadata loads). First generation makes you wait for completion — mine was first generation
Background priority	Backup and restore are lower priority than client I/O and consume unused throughput capacity. They progress slowly while production I/O is saturated

The third one has an operational consequence: overlapping backups with a nightly batch window slows both. Separate the backup window from your peak.

7. Verify the content

Skip this and you cannot claim you restored anything. Order matters: take the hashes before you create the backup. Taking them afterwards mixes in every change since, and you can no longer separate that from a copy problem.

# On the source, BEFORE creating the backup
cd /mnt/src && find . -type f -exec sha256sum {} + | LC_ALL=C sort -k2 > /tmp/before.txt

# On the restored volume
cd /mnt/restored && find . -type f -exec sha256sum {} + | LC_ALL=C sort -k2 > /tmp/after.txt
diff /tmp/before.txt /tmp/after.txt && echo "MATCH"

Check permissions and symlinks too:

find . -mindepth 1 -printf "%y %m %s %p\n" | LC_ALL=C sort -k4

All five sha256 digests matched in my run, and mode 0640, the symlink target, and a UTF-8 (Japanese) filename all survived. I also used the ordering deliberately: a 2 MiB file added after the backup was absent from the restored volume, which is what makes it a point-in-time image rather than a mirror.

8. Delete what the test created

Forgetting the destination file system costs roughly $270 a month. The order matters.

# 1. volume (do not let it create a final backup)
aws fsx delete-volume --region <DST_REGION> --volume-id <DST_VOLUME_ID> \
  --ontap-configuration '{"SkipFinalBackup":true}'

# 2. SVM (after the volumes are gone)
aws fsx delete-storage-virtual-machine --region <DST_REGION> \
  --storage-virtual-machine-id <DST_SVM_ID>

# 3. file system (after the SVM is gone)
aws fsx delete-file-system --region <DST_REGION> --file-system-id <DST_FS_ID>

# 4. the copied backup, if you no longer need it
aws fsx delete-backup --region <DST_REGION> --backup-id <DST_BACKUP_ID>

Without SkipFinalBackup a final backup is created, and that backup then blocks the next deletion. I have been stuck on this before.

And confirm the deletion through the API. During this run I passed an unsupported option to delete-file-system and polled for 8 minutes while the command had already failed. Waiting without reading the response only advances the bill.

Measured through AWS Backup as well

Everything above is the CopyBackup path. The same route, Tokyo to Osaka, was also measured through AWS Backup. That is the path that carries scheduling and cross-account copies, so its behaviour is what matters for production.

Two ways round. An on-demand backup job followed by an on-demand copy job, and a backup plan with a copy rule left to fire on its schedule. Both used the same source volume (1 GiB, 9,458,747 bytes, 5 files).

Step	Elapsed
On-demand backup job	4 m 02 s
On-demand copy job, Tokyo to Osaka	8 m 35 s
Plan-triggered backup job	30 m 06 s, of which about 24 minutes was the start window
Copy rule copy job (automatic)	6 m 31 s
Restore from the AWS Backup console	16 m 16 s

The copy rule fired without being touched. The schedule came round, a job was created, the backup completed, and a copy job followed on its own to put a recovery point in the Osaka backup vault.

A plan-triggered job waits in `CREATED`

This changes an operational assumption. The on-demand job went to RUNNING immediately. The plan-triggered one stayed CREATED for about 24 minutes. When it starts inside the start window — 60 minutes in this configuration — is AWS Backup's decision.

So a runbook that reads "the scheduled time has passed, therefore the backup exists" is wrong by the length of that window. Confirm a generation by job state or by the recovery point, not by the clock.

The recovery point in Osaka points at the source

The Resource ID column names the Tokyo file system and volume. It sits in the Osaka vault and points at where it came from.

Open the recovery point and the copy is visible as such.

Source region reads Asia Pacific (Tokyo). That is the evidence this artefact came from a cross-Region copy. Copy type is Manual because this one was run on demand; the copy-rule one arrives as a separate recovery point.

The restore form makes you pick a destination file system and SVM

The "File system ID" at the top is the source. The destination is chosen separately, below. Read the two as the same thing and the restore target looks already decided. It is not.

Open the dropdown and only file systems in the destination Region are listed.

With no file system in Osaka this list is empty. Exactly the same constraint as the CopyBackup path: not holding a destination day to day is paid for here.

Storage efficiency is checked by default

Two things.

Volume size is pre-filled from the source (1024 MiB). The FSx for ONTAP console defaults the same field to 1 TiB (below), so AWS Backup is the more predictable of the two here.

"Enable storage efficiency" is checked by default. The source volume had it off, and the restored volume came back with StorageEfficiencyEnabled: true. The FSx for ONTAP console restore form pre-selects the source value, so the two consoles behave differently. Uncheck it to match the source.

Attribute	Source	After the AWS Backup restore
`OntapVolumeType`	`RW`	`RW`
`SizeInBytes`	1 GiB	1 GiB
`TieringPolicy`	`NONE`	`NONE`
`StorageEfficiencyEnabled`	`false`	`true` (the form's default)
`SecurityStyle`	`UNIX`	empty

SecurityStyle coming back empty had been seen once on the CopyBackup path. It reproduced through AWS Backup, so it is no longer a single-path observation.

The content matched

Mounted from an EC2 instance in Osaka and compared. All five sha256 values matched, and the symlink target, the 0640 mode, the UTF-8 filename, the nested directories and the mtimes were preserved.

The restored volume carries .snapshot/backup-<id>/. That <id> is the source FSx for ONTAP backup, not the Osaka recovery point ID. The destination volume's default snapshot policy starts running too, so hourly.* snapshots appear beside it.

What cannot be monitored, and what catches you on teardown

BackupSizeInBytes was 0 on both backup jobs. Not a size to alert on.

Restore progress read 0.00% for the whole 16 minutes, then 100%. Anything that estimates remaining time from the percentage will not work.

One thing caught me on teardown. A vault cannot be deleted until an EXPIRED recovery point is fully gone. After delete-recovery-point the entry stayed listed as EXPIRED for about six minutes, and delete-backup-vault refused with "contains recovery points" throughout. The underlying FSx for ONTAP backup was still AVAILABLE at that point. Teardown automation has to poll; taking the delete request's success as done leaves you stuck.

On the console language: the screens in this section are in English. That was the setting at capture time; the other sections show the FSx for ONTAP console in Japanese. Field names are quoted in English either way.

Cross-account copies were not measured. They require AWS Organizations, so this verification stayed inside one account.

For the cross-account half — an earlier post, and what this one adds

There is a post on the NetApp Community that goes further on cross-account copies than this one does.

Cross-Region and Cross-Account Backup for Amazon FSx for NetApp ONTAP (Jordan Roth, NetApp Community)

It walks the cross-account path I did not measure, having actually run it. It sets out the protection layers (Snapshot copies, Multi-AZ HA, SnapMirror, AWS Backup copies), four architecture patterns — fan-in into an isolated vault account, fan-out to several, crossing Region and account in one hop, and a clean room for incident work — and the organization, key and IAM prerequisites behind them. If cross-account is going to production, read that first. What follows here is only where it meets my own measurements.

What this post was missing — cross-account needs a customer managed key

This is the one that matters. A cross-account copy requires a customer managed key (CMK). The AWS Backup documentation states that for resource types not fully managed by AWS Backup, cross-account copy with AWS managed keys is not supported, because an AWS managed key's key policy is immutable and cannot be shared across accounts (Encryption for backups in AWS Backup). FSx for ONTAP sits on the "not fully managed" side.

My verification did not meet that condition. The destination vault was created with defaults, so its key was alias/aws/backup (KeyManager: AWS). The cross-Region copy still succeeded. That is looser than the documentation's general statement, and it is the reason for measuring rather than assuming.

	Key used	Result
Cross-Region copy (Tokyo to Osaka, one account)	Defaults on both the source file system and the vaults (`aws/fsx` / `aws/backup`)	Succeeded (measured)
Cross-account copy	AWS managed keys cannot be used	Not measured. Unsupported per the documentation

There is a design consequence. A file system's KMS key is chosen at creation, and update-file-system has no parameter for changing it (checked in the AWS CLI). So if a cross-account copy might ever be needed, the CMK decision happens before the file system exists. It is not the kind of thing to switch on later. The destination vault has its own constraint: the default vault cannot be used, because its key cannot be shared.

The Region exceptions are listed individually

Earlier I wrote "commercial Regions where both services are available". AWS Backup feature availability lists the exceptions one by one.

Exception	Scope
Cross-Region copy not supported	Middle East (Bahrain), Middle East (UAE) — for all four of FSx for ONTAP, Lustre, Windows File Server and OpenZFS
Cross-Region and cross-account copy not supported	Asia Pacific (New Zealand), China (Beijing), China (Ningxia) — FSx for ONTAP

The China Regions need reading carefully. The AWS Backup document history carries an entry dated the same 27 August 2026 saying cross-account backup copy and cross-account management are now supported in the China Regions. That is about AWS Backup generally; FSx for ONTAP is excluded there, per the table above. Merge the two same-day announcements and you reach the opposite conclusion.

On "what was possible before", that post and I read it differently

Its table of changes records cross-Region CopyBackup within one account as already available before the launch. This post has been saying a backup could not cross a Region boundary before. Rather than assert, here is what I checked.

What I checked	What it says
ONTAP User Guide, "Protecting your data with volume backups" (archived 2026-07-05)	The sub-page list has no "Copying backups" — only "Copying tags to backups". The body says a backup restores to a file system in the same Region where it is stored
`copy-backups.html` in the ONTAP User Guide	No snapshot at all in the Internet Archive
The FSx for ONTAP What's New	"Previously, you could create and restore backups in the same Region and account as your file system"

What I cannot test now is whether the CopyBackup API would have accepted an ONTAP backup before the launch. That API is not specific to FSx for ONTAP, and has supported cross-Region copies for FSx for Windows File Server and FSx for Lustre for years, so reading the API reference alone and concluding it was already possible is a reasonable reading. Documentation not describing something is not the same as an API refusing it, so I am leaving that unverified.

For a reader today the practical answer is the same either way. What is unambiguously new is the AWS Backup half: cross-Region copy through AWS Backup, and any cross-account copy, from 27 August 2026.

What this post adds

The parts that do not overlap.

Here	What
Measured durations	Backup, copy and restore on both paths, to the second — including a plan-triggered job waiting about 24 minutes in the start window
Two consoles behaving differently	The AWS Backup restore form checks storage efficiency by default; the FSx for ONTAP one pre-selects the source value
A single-path observation reproduced	`SecurityStyle` coming back empty
What cannot be monitored	`BackupSizeInBytes` reads 0; restore progress sits at `0.00%` for 16 minutes
Where teardown sticks	An `EXPIRED` recovery point blocks vault deletion for minutes
Cross-Region copy on default keys	It works with `aws/backup` (above)
Replication and copy as terms	Covered in its own section
Regional DR inside one country	Tokyo to Osaka, with the cost of keeping data in-country

The two things that caught me out

1. A restoring volume reads as `DP`, not `RW`

I read the attributes right after the restore reached CREATED and got OntapVolumeType: DP. DP volumes are read-only and cannot be backed up. I started writing "a restore produces a read-only volume".

Then I tried touch from the client. It succeeded. A re-read returned RW. To be sure, I ran a second restore passing OntapVolumeType: RW explicitly — and it also reported DP while CREATING.

State	`OntapVolumeType`
Restoring (`CREATING`)	`DP`
Restore complete (`CREATED`)	`RW`

It is transient. If monitoring or automation reads this field, it sees DP during a restore. Since DP volumes cannot be backed up, a naive "back up as soon as the restore finishes" step behaves differently depending on when it reads. Gate on Lifecycle reaching CREATED first.

Without the write probe I would have published the wrong claim.

2. FlexGroup backup creation failed asynchronously

The documented limitation is that copying backups of FlexGroup volumes is unsupported. It attaches to the copy and says nothing about creation. I misread it and built the test on the creation side.

CreateBackup against a FlexGroup volume was accepted and returned CREATING, then went FAILED about 30 seconds later. The message names no cause:

Backup failed. Please delete the backup and try again.

As a control, a FlexVol RW backup created in the same session with the same credentials succeeded — so this is not permissions.

But this is one observation on one volume. I am not claiming FlexGroup backups cannot be created. AWS documentation describes the restore behaviour of FlexGroup backups (adding constituents when the HA pair count differs), which does not read as creation being unsupported. Separately it states only that a SnapLock FlexGroup volume cannot be backed up. My target was a non-SnapLock FlexGroup, single constituent, one aggregate, 113 GiB provisioned.

The consequence: no AVAILABLE FlexGroup backup was produced, so the documented copy restriction was never exercised. If FlexGroup is in scope, verify it from the creation side in your own environment.

3. The attribute I called "not carried over" was a CLI default

Seeing StorageEfficiencyEnabled: true on the restored volume, I wrote that storage efficiency is not carried over by a restore. That was wrong too.

Running the same restore through the console showed the field pre-selected to the source value (disabled), and it restored as false. The initial true came from omitting the field in the CLI --ontap-configuration. It is an API default when omitted, not restore behaviour. State it explicitly if you script the restore.

That is twice in one exercise that I nearly published a behaviour inferred from a single observation. Running the same operation through a second interface is what exposed both.

Attribute	Source	Restored
`StorageEfficiencyEnabled`	`false`	`true` if omitted in the CLI, `false` when stated
`SecurityStyle`	`UNIX`	empty in the API response (both CLI and console; UNIX mode bits behaved correctly over NFS)

Check storage efficiency after a restore — omitting the field enables it even when the source had it off. The empty SecurityStyle reproduced across both paths, but against the same UNIX-style source; other security styles were not tested.

What the console shows that the CLI does not

Four behaviours only visible in the console.

A note on the screenshots: my console is set to Japanese, so that is what you will see. The field
positions and the behaviour are identical in any locale, and each caption names the field in English.
If you would rather follow along in English, switch your own console language under Unified
Settings → Language.

The copy form defaults to the same Region

Backups list. Select a backup, then **Actions → Copy backup* (バックアップをコピー), alongside restore
(バックアップを復元) and delete (バックアップを削除).*

Copy backup form. *送信先リージョン (destination Region) defaults to the current Region** — leaving it
alone produces an in-Region copy. タグをコピー is "copy tags".*

The destination Region list. Commercial Regions only; the China and GovCloud sets cannot be crossed.

With Asia Pacific (Osaka) selected, the **KMS キー ID (KMS key ID) changes to that Region's default
key. This is what "incrementality requires the same KMS key" means in practice: the key in the
destination Region.

While the copy runs

Destination Region's Backups list. ステータス reads コピー中 ("copying"), and the entry is labelled
*(コピーされたバックアップ) — "(copied backup)"*. Tag copying carried the name across.

The completed copy. *ソースバックアップ ID (source backup ID)** and ソースバックアップリージョン
(source backup Region) = ap-northeast-1 are the two fields that establish this is a copy. タグ shows
the copied tags.*

Restore needs more than a file system

The dialog is titled *バックアップからボリュームを作成 — "Create volume from backup"*. The title states
plainly that a restore produces a new volume. ファイルシステム (file system) starts empty.

The ファイルシステム dropdown lists **only file systems that already exist in the destination Region. If
you have not created one, there is nothing to select — which is where the "create the file system first"
step announces itself.

After picking the file system, *ストレージ仮想マシン (storage virtual machine) appears with a red 必須
("required")** and blocks submission. This is the concrete form of "the destination needs a file system
and an SVM". Note the SnapLock section at the bottom — irreversible, and not a field to set in
passing during a restore.*

The volume size default is 1 TiB

The same form filled in. *ボリュームサイズ (volume size) defaults to 1 TiB** even though the source
volume is 1 GiB — I changed the unit to GiB here. ボリューム名, ジャンクションパス, and ストレージ効率
are all pre-filled from the backup; size is not. On a 1,024 GiB destination the default consumes the
entire file system.*

The console shows `DP` too, and does not refresh

During the restore. *ライフサイクルの状態 (lifecycle) = 作成 ("creating")** and ONTAP ボリュームタイプ
(ONTAP volume type) = DP — with Read-Write (RW) selected in the form. ジャンクションパス and
ストレージ効率 are absent, and セキュリティスタイル (security style) is blank.*

The same page after a reload: *作成済み ("created")** and RW, with junction path, storage efficiency,
and snapshot policy now shown. The page does not auto-refresh — it kept showing DP after the API had
already returned RW, so judging completion by eye alone misses it. セキュリティスタイル stayed blank.*

Designing this for production

Everything above was one manual copy. Six things still need deciding.

1. Scheduling — `CopyBackup` has none

The AWS documentation describes the FSx for ONTAP path as "manually copy volume backups". It runs from the console, CLI, or API on demand and carries no scheduler. Two ways to make it periodic:

Path	Shape	Fits when
AWS Backup backup plan	Adding a copy rule to a plan's rule writes each backup to a backup vault in the chosen Region, and optionally the chosen account, as it is taken. On-demand copy jobs are available too	You want policy control, or the same protection applied across an organization. The only path if the copy has to land in another account
Your own automation	EventBridge Scheduler → Lambda → `create-backup` → wait → `copy-backup`	You want to keep using FSx for ONTAP backups directly and manage generations yourself

AWS Backup is a managed service that centralizes taking, copying, and retaining backups under policy. FSx for ONTAP joined its cross-Region and cross-account copy targets in August 2026. Applied across an organization it pairs with AWS Organizations, and the account boundary is defined by Organizations policies; the documentation describes fan-in (many production accounts into one isolated account) and fan-out (one production account into several isolated accounts).

The cross-Region copy and the restore were measured (Measured through AWS Backup as well). The copy rule fired without being touched and the restore went through. A plan-triggered job does wait inside the start window, though — about 24 minutes here — so the scheduled time is not when the generation exists.

AWS Backup's Region coverage here is "all commercial Regions where both services are available", but the feature availability page lists individual exceptions (below). That page also states that when a resource type has no checkmark in the copy columns, copy operations are unsupported in every scenario, including a same-Region same-account copy to a different vault.

If you build it yourself, calling copy-backup before create-backup completes fails — the source must be AVAILABLE. Poll, or use a Step Functions wait.

Whether AUTOMATIC backups can be copied is unverified. The What's New post says "new and existing backups" without qualifying the type. If automatic backups are your evacuation source, prove it with one generation first.

2. Make recovery declarative — CloudFormation can express the restore

Typing six steps during an incident is not realistic at any team size. This can be declarative.

AWS::FSx::Volume has a BackupId property ("the ID of the volume backup to use to create a new volume"). So the file system, the SVM, and the restore all fit in one template:

Parameters:
  CopiedBackupId:
    Type: String
    Description: ID of the backup in the destination Region

Resources:
  DrFileSystem:
    Type: AWS::FSx::FileSystem
    Properties:
      FileSystemType: ONTAP
      StorageCapacity: 1024
      StorageType: SSD
      SubnetIds: [!Ref DrSubnetId]
      SecurityGroupIds: [!Ref DrSecurityGroupId]
      OntapConfiguration:
        DeploymentType: SINGLE_AZ_1
        ThroughputCapacity: 128
        PreferredSubnetId: !Ref DrSubnetId

  DrSvm:
    Type: AWS::FSx::StorageVirtualMachine
    Properties:
      FileSystemId: !Ref DrFileSystem
      Name: dr_svm
      RootVolumeSecurityStyle: UNIX

  DrRestoredVolume:
    Type: AWS::FSx::Volume
    Properties:
      Name: dr_restored
      VolumeType: ONTAP
      BackupId: !Ref CopiedBackupId
      OntapConfiguration:
        StorageVirtualMachineId: !Ref DrSvm
        JunctionPath: /dr_restored
        SizeInBytes: 1073741824
        OntapVolumeType: RW
        TieringPolicy:
          Name: NONE

Keeping this template undeployed in the destination Region is the cold-standby-IaC shape. Zero cost while idle, one deploy at recovery, and no step to forget.

I have not deployed this template — I ran the same sequence through the CLI. Deploy it once and time it before it becomes your runbook. An undeployed recovery template is a draft, not a procedure.

3. RTO breakdown — "20 minutes" is one line of it

I measured file system creation (20 min) and the restore (13 min). Your RTO is not the sum of those two. Fill this in for your environment.

Step	My measurement	Yours
Decision to recover (detection to action)	not measured
Destination file system creation	~20 min
SVM creation	a few minutes
AD join (if using SMB)	not performed
Restore to `CREATED`	13 m 21 s (9.4 MiB; about 22 h at 10 TB on gen 1, calculated)
Export policy / share reconfiguration	not measured
Client remount, DNS cutover	not measured
Business-side validation	not measured

If you use SMB, the AD join is in there. Confirm destination-Region reachability to your domain controllers while nothing is wrong. Discovering that mid-incident is a dead end.

Restore time gets longer with capacity

The 13 m 21 s and 16 m 16 s I measured were on a 9.4 MiB volume. Treat those as reference points only. Restore time for FSx for ONTAP backup, through either path, scales with capacity.

AWS publishes the rates (Backup and restore performance).

Operation	Mostly large files	Mostly small files
Backup creation	750 MBps	100 MBps
Restore	250 MBps	100 MBps

Two conditions come attached to those numbers, and both push an estimate to the longer side.

They are values "across several concurrent restores" — that is AWS's wording. Nothing says a single restore runs at that rate.
Backup and restore are background work and use only unused throughput capacity. Throughput capacity is therefore the ceiling. A 128 MBps file system will not reach 250 MBps.

Applying both, the calculation is min(published rate, throughput capacity). These are floors, on the optimistic assumption that client I/O is zero.

Logical data	128 MBps / large files	256 MBps+ / large files	Mostly small files (any throughput)
1 TB	2 h 10 m	1 h 6 m	2 h 46 m
5 TB	10 h 51 m	5 h 33 m	13 h 53 m
10 TB	21 h 42 m	11 h 6 m	27 h 46 m
20 TB	43 h 24 m	22 h 13 m	55 h 33 m
50 TB	108 h 30 m	55 h 33 m	138 h 53 m

Restoring 10 TB on the minimum configuration (128 MBps) is 22 hours at the floor. Mostly small files is 28 hours, and that one does not shrink by adding throughput, because the published 100 MBps binds first. At 50 TB it passes four days.

I could not find a measured figure at 10 TB or above in any public source. I searched AWS blogs, re:Post, the Knowledge Center, and the documentation, and found no article that publishes a measured restore duration with the capacity stated. The table above is calculated from AWS published rates — it is not something anyone measured. If you need a meaningful number for your own environment, the only way is to measure once at production-equivalent capacity and file mix.

And this can overturn "generation 1 is enough"

The section below says gen 1 is enough for a DR standby. At scale that reverses.

	Gen 1	Gen 2
Monthly at minimum throughput	$115.97 (128 MBps)	$772.99 (384 MBps)
Reads while restoring	No. You wait for completion	Yes, once metadata has loaded
Time to service at 10 TB	21 h 42 m (floor)	6–46 min (after 1–7% of metadata loads; the rest loads in the background)

Metadata is documented as 1–7% of the backup data, depending on file size — the smaller the files, the larger the share. At 10 TB, loading just that 1–7% first lets clients mount and start reading. A first access to a block that has not loaded yet costs tens of milliseconds, and it is cached on SSD after that.

So $657 a month of difference can turn a 21-hour outage into something closer to 30 minutes. If you hold 10 TB or more and your RTO is constrained in hours, choosing gen 2 for the standby too is a defensible call. Gen 1 is enough when either the capacity is small or the RTO can absorb waiting for the restore to finish.

A few TB or less, or an RTO in days → gen 1
10 TB or more with an RTO in hours → gen 2's reads-while-restoring earn their keep
Both are a longer RTO than SnapMirror (neither reaches a shape where the destination volume already exists)

4. Application consistency — this does not replace a database backup

A backup is a point-in-time copy of a volume. It does not quiesce an application.

Workload	What you need
File servers, shared storage	A volume point-in-time is usually enough
Databases (Oracle, SQL Server, …)	A database-level backup, or quiescing before the backup
Anything with constantly open writes	Application-level consistency checks after restore

Reading "we copy volume backups to another Region, so DR is covered" for a database workload can end with a volume that restores and a database that will not start.

5. Concurrency limits at scale

The limits are on operations in flight, not on how much you can keep.

Limit	Value	Consequence
Copies in flight (per volume, per destination Region, per KMS key)	5	Parallelizing one volume's generations stops at 5
Copies in flight (per account)	1,000	Firing off hundreds or thousands of volumes at once exceeds it. Excess requests are rejected
Backups per volume	4,091	Work your generation plan backwards from here

In a multi-tenant design with one volume per tenant, tenant count becomes your concurrency. Batch the submissions and retry rejections with exponential backoff.

6. Monitoring and control

Copies are asynchronous. When copy-backup returns, nothing has finished yet.

Decide	Options
Where failures are detected	With AWS Backup you can follow job state. For `CopyBackup`, polling `describe-backups` is the dependable route. Whether CloudWatch or EventBridge can carry this is unverified
How success is evidenced	Store the `describe-backups` response (`BackupId`, `SourceBackupId`, `SourceBackupRegion`, `CreationTime`)
Who approves a restore	A restore creates a new volume and destroys nothing, but it consumes destination SSD and starts billing. Put it in an approval flow where that is expected
Which destinations are allowed	Enforce it in IAM or an SCP, not in a written rule

One note on tags. --copy-tags carries source tags to the destination. If your tags name systems or departments, that naming appears in the isolated account too. Depending on why you are isolating, re-tagging may be the better choice.

Cost

The $0.55 I spent was a test bill. What is useful in your environment is a monthly figure, so this builds one up from unit rates.

Unit rates — queried from the AWS Price List API

The rates below were queried from the AWS Price List API on 2026-08-29 (effective 2026-07-01). Tokyo and Osaka were identical on all four of the items that appear in both.

Charged item	Tokyo / Osaka rate	Nature of the charge
SSD storage (Single-AZ)	$0.150 / GB-month	provisioned
SSD storage (Multi-AZ)	$0.300 / GB-month	provisioned
Capacity pool Standard (Single-AZ)	$0.0238 / GB-month	consumed
Capacity pool Standard (Multi-AZ)	$0.0476 / GB-month	consumed
Backup storage	$0.050 / GB-month	consumed (incremental)
Throughput capacity (Single-AZ, gen 1)	$0.906 / MBps-month	provisioned
Throughput capacity (Single-AZ, gen 2)	$2.013 / MBps-month	provisioned
Capacity pool read requests	$0.00037 / 1,000 requests	consumed
Capacity pool write requests	$0.0047 / 1,000 requests	consumed

Going through AWS Backup does not change the backup storage rate. FSx for ONTAP is not a resource type that AWS Backup fully manages, so the storage charge appears on the FSx for ONTAP side rather than on AWS Backup (Metering, costs, and billing for AWS Backup). The rate is therefore $0.050 / GB-month on the native path and on the AWS Backup path alike. Only when you use a logically air-gapped vault (LAG vault) does all of the storage and transfer land on the AWS Backup side.

FSx for ONTAP backup is not the cheap option

$0.050 / GB-month is 2.1× the Single-AZ capacity pool ($0.0238). Put the other way round, the capacity pool is 47.6% of backup storage — less than half.

AWS makes the same comparison. AWS Prescriptive Guidance puts fully managed backup at $0.05 / GB-month, against $0.03221 / GB-month for SnapVault aimed at a capacity pool at an SSD-to-pool ratio of 1:10 (Choose the right SMB file storage, US Region example).

That 47.6% is Single-AZ only. A Multi-AZ capacity pool is $0.0476, which is 95.2% of backup storage, and the advantage all but disappears. Carrying "the capacity pool is less than half" into a Multi-AZ estimate will put your numbers wrong.

Running one continuously has a floor

Read the rate table straight and it seems to say "if the capacity pool is cheap, then keeping a destination file system running is cheap too". It does not work out that way. The minimums on the provisioned charges set the floor for anything kept running.

Constraint	Value	Source
Minimum SSD capacity	1,024 GiB per HA pair	Quotas
Minimum throughput capacity (gen 1)	128 MBps	same
Minimum throughput capacity (gen 2, 1 HA pair)	384 MBps	same

Monthly cost of a Single-AZ minimum configuration:

Generation	Components	Monthly at minimum
Gen 1 (128 MBps)	1,024 GiB × $0.150 + 128 MBps × $0.906	$269.57
Gen 2 (384 MBps)	1,024 GiB × $0.150 + 384 MBps × $2.013	$926.59

The same "minimum configuration" differs by 3.4× between generations, because gen 2's minimum throughput is 384 MBps and its per-MBps rate is 2.2× gen 1. For backup and a DR standby, gen 1 is the choice (below).

All-tiering does not remove this floor either. Metadata always stays on SSD, so the 1,024 GiB SSD minimum remains provisioned.

It inverts with scale

Assumptions: 30 daily backup generations, 2% daily change rate, no dedupe or compression benefit assumed, Single-AZ gen 1 at the 128 MBps minimum, the Osaka rates above. The all-SSD column provisions capacity to the recommended 80% SSD utilisation.

Logical data	Backup copies (no destination FS)	Standing + all-tiering	Standing + all SSD
1 TB	$80	$293	$303
2 TB	$160	$317	$491
5 TB	$400	$389	$1,053
10 TB	$800	$508	$1,991
20 TB	$1,600	$746	$3,866
50 TB	$4,000	$1,460	$9,491

The crossover is around 4.8 TB. That is the $269.57 floor divided by the per-GB difference: 1.6 generations of backup at $0.050 ($0.080) against the capacity pool at $0.0238, a gap of $0.0562.

This is what backs up "we keep one running for RTO and RPO". Past 4.8 TB a standing destination file system is lower monthly, and a minutes-level RPO, a single-digit-minute RTO, and a failback path come with it. Improving recoverability is not an increase in monthly cost at that size. At 1 TB it is the other way round, and the gap between $80 and $293 has to be explained as a difference in requirements.

Why a DR standby takes generation 1

For backup and a DR standby, gen 1 is enough. The reason is the throughput increments.

	Gen 1	Gen 2 (1 HA pair)
Selectable throughput	128 / 256 / 512 / 1,024 / 2,048 MBps (Tokyo and Osaka cap at 2,048)	384 / 768 / 1,536 / 3,072 / 6,144 MBps
Monthly at minimum throughput	128 MBps × $0.906 = $115.97	384 MBps × $2.013 = $772.99

A standby's throughput is sized by SnapMirror transfer volume and the reads right after a failover, not by production load. Gen 2 still floors at 384 MBps for that use, which is 6.7× on the throughput line alone. With gen 1 you start at 128 MBps and move up to 256, then 512, when you need it.

Deployment type cannot be changed after creation (documented). Going from gen 1 (SINGLE_AZ_1) to gen 2 (SINGLE_AZ_2) means a restore from backup, or a migration over SnapMirror or AWS DataSync. Decide the generation before you build the standby.

Total monthly by throughput

The 4.8 TB crossover above assumes the 128 MBps minimum. Raise throughput and the crossover moves up with it. Here are the combinations laid out.

Assumptions: Single-AZ gen 1, all-tiering, SSD at the 1,024 GiB minimum, Tokyo / Osaka rates. The backup-copy column is 30 daily generations at a 2% change rate. Capacity pool request charges are not included.

Logical data	128 MBps	256 MBps	512 MBps	1,024 MBps	2,048 MBps	Backup copies
1 TB	$293	$409	$641	$1,105	$2,033	$80
5 TB	$389	$505	$736	$1,200	$2,128	$400
10 TB	$508	$624	$855	$1,319	$2,247	$800
20 TB	$746	$862	$1,093	$1,557	$2,485	$1,600
50 TB	$1,460	$1,576	$1,807	$2,271	$3,199	$4,000

Bold is cheaper than holding backup copies. Restated as the highest throughput that still comes in under the backup-copy figure:

Logical data	Backup copies / month	Highest throughput under that
1–2 TB	$80–$160	None. Even 128 MBps standing costs more
5 TB	$400	128 MBps
10 TB	$800	256 MBps
20 TB	$1,600	1,024 MBps
50 TB	$4,000	2,048 MBps (the gen 1 ceiling in Tokyo and Osaka)

At 50 TB you can provision the largest throughput gen 1 offers in-Region and still come in cheaper than holding backup copies. At 1–2 TB backup copies are cheaper at every throughput, and there the difference is a requirements conversation.

What the "zero while idle" of backup copies buys

With no destination file system in normal operation, there is no throughput capacity charge and no SSD capacity charge. That is a clear advantage on the monthly figure. The work it defers, though, reappears at the moment of failover, as urgent work.

And the unit of a backup is the volume, so a restore always produces a new volume. There is no path that overwrites an existing volume in place, on the CopyBackup route or the AWS Backup route. That is what makes steps 4 to 6 below unavoidable rather than optional: a new volume means a new junction path if the original still exists, and export policies, SMB shares, quotas, and the snapshot policy do not travel with it. The volume size default is a separate trap — it comes up as 1 TiB rather than the source's size, covered further up.

In an incident that same constraint cuts the other way and becomes an advantage: you can preserve a compromised volume exactly as it stands and restore beside it, so the original is still there to investigate.

Here is what a cold start to restored service actually needed, in the range I measured.

#	Step	Measured / state
1	Create the file system in the destination Region	20–22 min (measured, `SINGLE_AZ_1`, 128 MBps)
2	Create the SVM	a few minutes
3	Restore the volume from backup	13 m 21 s (`CopyBackup` path) / 16 m 16 s (AWS Backup path), both at 9 MB
4	Recreate export policies and SMB shares	Only the volume is restored. Share configuration does not come with it
5	AD-join the SVM if you use SMB	It is a new SVM, so it joins again
6	Repoint clients	The new file system has a different DNS name and different IPs. Either move a CNAME or remount on the client

Steps 1 and 3 finish if you wait. Steps 4 to 6 are a person deciding and typing. And they are needed mid-incident, not on a quiet afternoon. What you saved monthly is traded against doing that work under pressure.

In a SnapMirror shape, 1 to 3 do not exist, and the destination SVM in 5 is already joined. What is left is 6 and the break-and-promote, so the number of steps is not comparable to begin with.

Steps 4 to 6 can be automated ahead of time. Declare the restore in CloudFormation, script vserver cifs create and the share creation, put a CNAME in front of DNS. But "can be" and "confirmed to work during an incident" are different claims. Run it through once while nothing is wrong.

The price difference is the requirement difference

Here is what sits inside the $80 and the $293 at 1 TB.

	Backup copies	Standing (SnapMirror shape)
RPO	60 min (the automatic backup minimum)	5 min
RTO	file system creation 20–22 min + restore 13–16 min (measured)	promote the volume
Failback	no path	yes
Destination file system in normal operation	not needed	needed

If the requirement is a minute-level RPO, backup copies are not a candidate. And at recovery time the backup-copy side creates a destination file system too, so the same floor starts applying from then on. The accurate reading is a design whose cost structure differs between normal operation and recovery.

Not included above

Capacity pool request charges: not in the tables above. The rates are known, but I have not measured the conversion from SnapMirror transferred bytes to request counts, so I cannot produce a number. With all-tiering every write goes to the capacity pool, so write requests ($0.0047 / 1,000) may not be negligible.
Inter-Region data transfer: treated differently depending on the path. Split out into the next section
Source-side backup storage (unchanged — you only added somewhere to put a copy)
KMS request charges if you use a CMK
SnapMirror inter-Region transfer (SnapMirror is not measured in this post)

Inter-Region data transfer is treated differently per path

AWS Backup copies have a transfer line item. Tokyo to Osaka is $0.09 / GB (Price List API, AWSBackup, APN1-APN3-CrossRegion-WarmBytes-FSx, retrieved 2026-08-29). The AWS Backup pricing page gives the same transfer rate for Resource Group 3, which includes Amazon FSx, on a standard vault and on a logically air-gapped vault alike.

For native CopyBackup, the reasonable thing is to estimate as though it applies. I started to write "probably does not apply", checked again, and withdrew it. Here is why.

The argument I withdrew: backups sit in AWS-managed S3 and never traverse the customer VPC, so they are not subject to inter-Region transfer charges the way EC2 is. That does not hold. EBS snapshots are equally in an AWS-managed area and equally do not traverse a VPC, and a cross-Region snapshot copy does incur AWS Data Transfer charges. The AWS Storage Blog walks through tracking that charge in Cost Explorer, and the usage type that appears there is USW2-USE1-AWS-Out-Byte, the service is "EC2 - Other", and the usage type group is Data Transfer - Region to Region (Out) (Effectively track AWS data transfer costs for cross-region Amazon EBS Snapshot Copy).

That also exposes a hole in how I first checked it. I based it on "there is no transfer item in the FSx for ONTAP price list", but inter-Region transfer is listed under AWSDataTransfer, not under the originating service. EBS is exactly that case: there is no "snapshot transfer" item in the EBS price list, and the charge happens anyway.

And AWSDataTransfer has a generic, service-agnostic inter-Region SKU.

Direction	Usage type	Rate
Tokyo → Osaka (out)	`APN1-APN3-AWS-Out-Bytes`	$0.09 / GB
Osaka → Tokyo (in)	`APN1-APN3-AWS-In-Bytes`	$0.00 / GB

(Price List API, AWSDataTransfer, effective 2026-06-01, retrieved 2026-08-29)

The Amazon FSx SKU on the AWS Backup side is also $0.09 / GB. The same transfer rate appearing in both places makes it natural to read them as the same underlying charge.

To summarise, this is as far as it can be taken right now.

Question	State
AWS Backup cross-Region copies incur transfer	Listed in the price list ($0.09 / GB, Tokyo → Osaka)
Native `CopyBackup` incurs transfer	Estimate as though it does. A generic $0.09 / GB SKU exists, and the same shape (EBS snapshot copy) is charged in practice. But I have not confirmed it on a bill — unverified
"It is not in the FSx for ONTAP price list, so it is free"	This inference is wrong. Transfer is listed under `AWSDataTransfer`

In an estimate, put in the first full copy's capacity × $0.09 / GB. After that it is incremental, so changed volume × $0.09 / GB. A 10 TB first full is $900 — not a monthly figure, but a one-off of real size. The shortest way to settle it is to copy one small volume and read the UsageType in Cost Explorer. I have not done that.

On which account is billed for transfer, two AWS statements disagree. FSx for ONTAP is not a resource type AWS Backup fully manages, so that discrepancy lands directly on this case.

Source	Statement
AWS Backup Developer Guide	For resource types that are not fully managed, transfer appears on the destination account
AWS Backup pricing page	Transfer is charged to the account sending the data (the copy source)

Which one is correct I have not been able to confirm. If cost attribution has to go into the design of a cross-account setup, copy once first and check which account it lands on in the actual bill.

The trade-off of all-tiering

Keeping a standby cheap with all-tiering means reads after a failover start from the capacity pool. All does not pull read blocks back onto SSD, so performance immediately after a cutover is not the same as production sized for SSD. It is a choice between the monthly cost while idle and the performance immediately after cutover.

Prioritise cost while idle → all
Need production-equivalent performance from the moment of cutover → provision SSD as in production (the "all SSD" column above)

When you build your own estimate, replace the change rate and the generation count with measurements from your environment. The 2% is an assumption, not something I measured. Rates are revised, too. The figures above were retrieved on 2026-08-29 and are effective 2026-07-01, so check the current values on the FSx for ONTAP pricing page and the AWS Backup pricing page before you decide.

As a ransomware layer

An isolated copy is a useful layer, but it lacks two properties people expect of it.

Expectation	Reality
Put it in an isolated account and it cannot be deleted	Copies can be deleted. The isolation holds only if the destination account's IAM or vault policy prevents deletion
A backup copy satisfies an immutability requirement	It does not. Preventing deletion during a retention period is the domain of WORM features (Object Lock, SnapLock, Tamperproof Snapshot, Vault Lock)

WORM features are irreversible. Once enabled, nothing is deletable until retention expires, and the blast radius is routinely wider than the resource you named. Get the retention value and the blast radius approved separately before enabling one. I made a file system undeletable for six months this way.

On the other hand, a restore always producing a new volume is an advantage during an incident. You can preserve the compromised volume for investigation and restore into a separate volume to resume operations, without overwriting the original.

Common errors and how to isolate them

Decide which layer stopped before you start debugging. Get the layer wrong and you spend the incident suspecting something that works.

Layer	Look at	What to check
1. Source	The backup	`Lifecycle` is `AVAILABLE`; volume type is `RW`; style is FlexVol
2. Permissions	IAM / KMS	`fsx:CopyBackup` present; source-Region backup reachable; destination key permissions for a CMK
3. Copy	`describe-backups` at the destination	`COPYING`, `AVAILABLE`, or `FAILED`; concurrency limits
4. Restore	The destination volume	`Lifecycle`; enough SSD (it pauses when short)
5. Access	The client	Export policy, security group, SVM NFS / SMB endpoint

Messages I actually saw:

Message	Layer	What to do
`Backup <id> is being copied, can't be deleted`	3	Wait for the copy. There is a short delay even after it completes
`Backup failed. Please delete the backup and try again.`	1	Names no cause. Suspect FlexGroup and take a FlexVol control
`Volume with type DP is not backupable.`	1	That is a SnapMirror destination. Back up at the source instead
Rejection from exceeding copies in flight	3	5 per volume per destination Region per KMS key, 1,000 per account. Wait and retry
`Unknown options: --ontap-configuration` (`delete-file-system`)	—	A CLI version difference. Read the response. This cost me 8 minutes of polling

Whether a failed copy leaves a partial backup at the destination is unverified — I did not induce a failure deliberately. If you automate this, detect FAILED and delete it (the backup created by my FlexGroup failure did remain as FAILED, and it deleted cleanly).

Where this sits alongside an existing backup product or SaaS

If you already run a data protection product or backup SaaS, this update is not a migration question. It is a question of which layer you give to what. There are three paths, and combining them is normal.

Path	Managed by	Another Region	Another account	Fits
A. FSx for ONTAP backups	FSx for ONTAP API / console	yes (new)	no	Volume-level evacuation with the fewest moving parts
B. AWS Backup	Backup plans	yes	yes (needs Organizations)	Policy control, one view across services
C. Existing data protection product / SaaS	That product's policy	Depends on the product	Depends on the product	File- or application-level restore, catalog search, fitting existing workflows

A concrete way to split them

File-level restores from the existing product; whole-volume evacuation from FSx for ONTAP backup copies. One split that often works, and the reason is restore granularity. A restore from an FSx for ONTAP backup produces one new volume, which is a lot of machinery to return a single file a user deleted. Conversely, pushing whole volumes to another Region through an existing product tends to grow the transfer volume and the catalog. Where your product already handles cross-Region evacuation efficiently, the opposite split is equally reasonable.

The decisive question for AWS Backup is whether you need another account. Evacuating to an isolated account is out of reach for the FSx for ONTAP API alone; it needs AWS Backup and AWS Organizations. If you only need another Region, the FSx for ONTAP API is fewer moving parts.

What to ask, regardless of product

Ask along these axes rather than by product name:

Axis	What to establish
Unit of protection	Whole volume, file / directory, or application-consistent
Restore granularity	Can it return one file, or only a whole volume
Cross-Region / cross-account path	The product's own feature, via AWS Backup, or your own copy
What the restore target requires	Does a file system need to exist first
Where retention policy lives	The product or AWS. Is it managed in two places
Monitoring and alerting	Where failures surface, and whether that reaches your existing observability
Where the cost lands	Product licensing, AWS usage, or both

"Managed in two places" is where the accidents are. If FSx for ONTAP automatic backups and the product's policy each hold generations independently, changing retention on one side leaves the other holding an unintended number. The ceiling is 4,091 backups per volume, so plan backwards from it.

Storage is also billed twice. Holding the same data in both the product's repository and FSx for ONTAP backups charges for both sets of generations. If you combine them, pick a primary and make the generation counts differ:

Shape	Primary	Secondary
File-level restore matters most	Existing product, long retention, many generations	FSx for ONTAP backups short and few (recent whole-volume evacuation)
Cross-Region evacuation matters most	FSx for ONTAP backup copies, many generations	Existing product limited to in-Region file restores

Neither is correct in the abstract. It comes down to which granularity 90% of your restore requests arrive in. Counting your past restore tickets is the most reliable way to find out.

About monitoring

My verification polled describe-backups. I have not verified a path for detecting copy completion through CloudWatch metrics or EventBridge events. If you plan to wire this into an existing observability stack, that is a gap you will need to close yourself.

Choosing between this and SnapMirror

Neither is better. The AWS documentation states RPO and RTO figures for both, and they protect different things.

For context: cross-Region disaster recovery for FSx for ONTAP has been buildable with SnapMirror for years — a destination file system, cluster peering, asynchronous replication. Plenty of production environments run exactly that, and backup copies do not displace it. What got added is the option of not holding the destination while nothing is wrong.

Dimension	Backup copies	SnapMirror
Suits	Offline copies for compliance or isolation	Availability in a second Region
RPO guidance	60 minutes	as low as 5 minutes
RTO guidance	minutes to hours (size dependent)	single-digit minutes
Needed while idle	No destination file system	A running destination file system
Shape of recovery	Restore as a new volume. The original does not come back	Break the relationship and promote the destination
Path back to production	None. Continue on the recovery side, or copy back in the other direction	Delete the relationship and `snapmirror resync` from the DR side
Trade-off	Destination file system creation lands on RTO (20 min measured); RPO is as coarse as the backup interval. The unit of backup and restore is the volume, and with AWS Backup / FSx backups a restore always lands in a newly created volume — there is no restore-in-place path, so junction paths, share configuration and where clients point all have to be rebuilt	You pay for destination capacity and throughput continuously; needs cluster peering and does not work through NAT

In operation, the asymmetry in "path back to production" is what bites. SnapMirror can be re-established in reverse to fail back. Backup copies only restore, and what you get is a new volume. If a documented failback procedure is a requirement, backup copies do not substitute for it.

One caveat on snapmirror resync: user-created snapshots are not replicated by a resync. The preserve parameter is available on XDP relationships.

Running both

This is a natural combination: SnapMirror for availability, backup copies for isolated retention. One catch — a SnapMirror destination is a DP volume and cannot be backed up, so "back up the replica and evacuate that" does not work. Take backups at the source and copy those.

Added (September 2026): read-only does not mean inert. Verified on ONTAP 9.18.1P5 in ap-northeast-1: a SnapMirror destination can be served read-only through an FSx for ONTAP S3 Access Point while replication is still running — no break, no clone. What gates the attachment is the junction path, not the volume type, and setting that junction path is an ONTAP-side operation: the Amazon FSx API refuses it on a DP volume, and UpdateVolume refuses it silently, returning HTTP 200 with the volume unchanged. After a transfer, new data was readable through the same access point in 15 seconds. Budget minutes to tens of minutes for the FSx control plane to report a newly mounted volume as attachable. Design considerations.

Regional DR within one country

Copying to Osaka keeps data inside Japan while placing it in a second Region.

Consideration	Detail
Where the data sits	Both Tokyo and Osaka are in-country
Idle cost	Backup storage only ($0.050/GB-month in Osaka, rate effective 2026-07-01)
Needed at recovery	A file system and SVM in Osaka. Creation time lands on RTO (20 min measured)
Generation availability	`SINGLE_AZ_1` was creatable in Osaka (measured 2026-08-28). This differs by Region
Trade-off	Coarser RPO and RTO than SnapMirror. A minute-level RPO requirement means keeping the destination running

How you satisfy a data residency requirement is a design question, not a legal or regulatory determination. Interpreting regulatory requirements belongs to your compliance function and the relevant authority. This post does not substitute for that.

A staged rollout

Do not start on a production volume. The order I would suggest:

Run the whole path on a small test volume (~1 GiB) — copy, destination file system, restore, content check, teardown. This is where the holes in your procedure show up
Time it including teardown. Destination file system creation lands on RTO, so this is the number you most want
Measure once at production data volume. My 9.4 MiB figures are dominated by fixed overhead and are not evidence for your environment
Check whether the second copy is incremental by comparing transfer volume and duration
Run periodic copies on a non-production volume and build the failure-detection path. This is where the real work is
Go to production, keeping the existing backup path running in parallel
Put restore drills into operations. Without them your RTO is an estimate

Do not go to production without step 5. Copies are asynchronous, and when one fails your command has already returned success.

A verification plan template

Useful to have on one page if a team is involved:

Item	What goes in it	Mine
Question being answered	"Does cross-Region evacuation work without a standing destination?"	same
Success criteria	Content matches after restore, and every line of the RTO breakdown is filled	Hash match achieved. RTO breakdown incomplete (AD and remount unmeasured)
Time required	Half a day for the small path, a day at production volume	~1.5 hours including teardown
Cost ceiling	Driven by destination file system runtime. Set it first	Capped at 3 hours / $2; actual $0.55
Stakeholders	Storage operations, the business (accepting the restored data), audit / control, networking (destination VPC, AD reachability)	—
Go criteria	Measured RTO under target / incrementality confirmed / failure detection working	Not met (incrementality and detection unverified)
No-Go criteria	RPO needed in minutes / failback required / FlexGroup in scope	—

By that standard I cannot call my own run a Go. Incrementality and failure detection are unverified, and the RTO breakdown does not include AD. What this post establishes is that the path works.

A recovery runbook skeleton

Naming the decision points keeps the day itself calm:

#	Decision or action	Branch
1	Is recovery needed (Region-level failure, or data loss)?	Data loss alone is an in-Region restore. You do not need the other Region
2	Which generation?	Check `CreationTime` on the copies. This is also where you see whether copying kept up
3	Create the destination file system	`deploy` the template, or steps 4–5 by hand
4	SMB in play?	Then AD join is required. Put the reachability check here
5	Restore	Wait for `Lifecycle: CREATED`. `OntapVolumeType` reads `DP` while restoring
6	Business validation	Data match and application startup. Databases need a consistency check
7	Return to the original Region?	Backup copies have no failback path. Either continue on the recovery side, or copy back in the other direction

Try it now

The same steps with only the values to substitute:

# Change these five lines for your environment
export SRC_REGION=ap-northeast-1
export DST_REGION=ap-northeast-3
export SRC_VOLUME_ID=fsvol-0123456789abcdef0
export SRC_BACKUP_ID=backup-0123456789abcdef0
export DST_BACKUP_ID=   # fill in after the copy

# 1) create a backup
aws fsx create-backup --region "$SRC_REGION" --volume-id "$SRC_VOLUME_ID" \
  --query 'Backup.BackupId' --output text

# 2) wait for AVAILABLE
aws fsx describe-backups --region "$SRC_REGION" --backup-ids "$SRC_BACKUP_ID" \
  --query 'Backups[0].Lifecycle' --output text

# 3) copy to another Region (--region is the destination, --source-region is the origin)
aws fsx copy-backup --region "$DST_REGION" \
  --source-backup-id "$SRC_BACKUP_ID" --source-region "$SRC_REGION" --copy-tags \
  --query 'Backup.BackupId' --output text

# 4) confirm the copy
aws fsx describe-backups --region "$DST_REGION" --backup-ids "$DST_BACKUP_ID" \
  --query 'Backups[0].{Life:Lifecycle,Src:SourceBackupId,SrcRegion:SourceBackupRegion}' --output table

If step 4 shows SourceBackupId and SourceBackupRegion, the copy is real. That is the whole day-to-day preparation.

Wrapping up

What this update removed is what you keep running day to day. Putting data in a second Region no longer requires a file system there.

You pay for that at recovery time instead — file system creation (20 minutes measured) plus an RPO as coarse as your backup interval. If minute-level RPO and a failback procedure are requirements, this does not replace SnapMirror. If isolated retention is the requirement, you now have an option that needs nothing standing by.

Three things I plan to do next:

Measure cross-account copies. The AWS Backup cross-Region copy and restore are measured now; cross-account needs AWS Organizations and remains documentation only
Measure incrementality at production data volume. At 9.4 MiB it disappears into fixed overhead
Establish a failure-detection path. Whether CloudWatch or EventBridge can carry copy state is unverified

The most useful thing I learned was smaller than the feature: I believed DP was a permanent property until I tried writing to the volume, and I believed storage efficiency was not carried over until I ran the same restore through the console. Two conclusions drawn from a single observation on a single interface, both wrong.

Thanks for reading.

Resources

AWS What's New: FSx for NetApp ONTAP now supports copying backups across AWS Regions and accounts
AWS What's New: AWS Backup adds cross-Region and cross-account backup support
AWS: Copying backups
AWS: Copying backups within the same AWS account — the backup-copy vs SnapMirror guidance and the RPO / RTO figures
AWS: Protecting your data with volume backups — restore target confined to the backup's Region, SSD-first restores, background priority
AWS: Restoring a backup to a new volume
AWS: CreateVolumeFromBackup
AWS CloudFormation: AWS::FSx::Volume — the BackupId property
AWS Backup: Creating backup copies across AWS Regions
AWS Backup: Creating backup copies across AWS accounts
AWS Backup: Encryption for backups in AWS Backup (why AWS managed keys cannot do a cross-account copy)
AWS Backup: Feature availability (the per-Region exceptions)
Cross-Region and Cross-Account Backup for Amazon FSx for NetApp ONTAP (NetApp Community: the cross-account walkthrough, four architecture patterns, clean room and restore testing)
AWS: Quotas (FSx for ONTAP)

Every resource used for this test has been deleted. The figures are measurements from one specific environment and configuration, and will differ with data volume and setup.

Using Databricks FILE type against data on NAS: what works today with FSx for ONTAP S3 Access Points

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 12 Aug 2026 15:07:29 +0000

A follow-up to Part 2: Databricks and FSx for ONTAP S3 Access Points in the "FSx for ONTAP S3 Access Points x Lakehouse Deep Dive" series. Some results changed since Part 2 — if you read that one, see "Why reads fail" below.

TL;DR

I tested whether images and PDFs already sitting on FSx for ONTAP can be referenced from
Databricks without copying them into the lakehouse. The result is: registration works, reads
do not.

If you only need the zero-copy verdict, read "Connecting to an FSx for ONTAP S3 Access Point"
and "Design tips" below. If you want the behaviour of the FILE type itself, start from the
top.

Databricks shipped FILE as a beta column type: a reference to an unstructured file held as a
single column in a Delta table. The value is a struct carrying uri / offset / size /
content_type / checksum. Because it holds the reference and the metadata as a struct, you
can pass it straight from a table into an AI function such as ai_parse_document. There are
two variants — FILE EXTERNAL, which holds only the reference, and FILE MANAGED, which
copies the bytes into Databricks.

I tested this against files on Amazon FSx for NetApp ONTAP, reached through an FSx for ONTAP S3
Access Point — the feature that exposes an ONTAP volume over an S3-compatible API.

Here is what I found:

Registration against an FSx for ONTAP S3 AP succeeds; reads do not. The cause sits in the session policy attached to the temporary credentials Unity Catalog vends, and there is no way to work around it from the user side
The FILE type itself works. But you have to choose between FILE EXTERNAL and FILE MANAGED before you ingest
If your design puts metadata in object tags, tag values are effectively ASCII. Most CJK strings are rejected while a minority are accepted. AWS Support has escalated that split to the service team as a potential defect
The operator surface is inconsistent. GROUP BY and DISTINCT are accepted; = and ORDER BY are rejected

Below: what is established for each of those four, and how to turn it into design decisions.
The templates and scripts are in the repository, so you can re-run the same tests in your own
account.

Repository: Yoshiki0705/fsxn-lakehouse-integrations

Test environment

Item	Value
Measured	2026-08-12
Storage	One FSx for ONTAP file system
Access Point	One, INTERNET origin, UNIX root identity
Workspace	One non-trial workspace, purpose-built, same account and region
Compute	Serverless SQL warehouse, and a classic DBR 18.2 cluster
Control	A S3 general bucket in the same account, running the same operations alongside
Method	SQL Statement Execution API, and boto3 from my workstation

Running the S3 general bucket control alongside matters. Looking only at the Access Point side, you
cannot separate a defect in your own environment from behaviour specific to the Access Point.
Except where stated otherwise, every conclusion below rests on the same operation succeeding
against S3 general bucket.

Note that the object tag results depend on the ONTAP tag validation implementation, so a
different ONTAP version may behave differently. The FSx console and describe-file-systems do
not expose the version; use the ONTAP REST API (GET /api/cluster?fields=version) to check
yours.

What the FILE type can and cannot do

First, the behaviour of the type itself. There is prior work on this feature from Databricks
Japan: パス文字列でもバイナリでもない。DatabricksのFILE型を試す
by taka_yayoi. Every operational caveat it lists reproduced in my environment, so I have
tabulated those first. (What follows is my paraphrase and my own measurements, not a
translation.)

Some terms first. FILE MANAGED copies bytes into a Databricks-side location called a
FileSpace. list_files enumerates files on a Volume and returns FILE values; create_file
constructs a FILE value explicitly. _object_metadata reads an object's tags and user
metadata.

Behaviour	Measured	Design implication
FileSpace on the same volume as the source	`CREATE TABLE` succeeds; `INSERT` fails with `Cannot get file metadata under managed storage`	Give the FileSpace its own volume
`checksum` on FILE values from `list_files`	Null on every row	Not usable for integrity checks
`checksum` under `FILE MANAGED`	Populated with `ETAG:"…"`, matching the `etag` from `_object_metadata` on S3 general bucket exactly as a string	Comes from the object store, so it is usable for reconciliation
Filenames under `FILE MANAGED`	Replaced with an opaque UUID; the extension does not survive either	Copy the name to a column before ingesting
Format of `uri`	Prefixed with `dbfs:`	Normalise before writing string comparisons
Serverless notebooks	Unsupported	Use the SQL Statement Execution API against a serverless SQL warehouse
Automatic garbage collection in beta	Does not run	The FileSpace grows on every re-run. Plan for manual deletion
The Previews page toggle	Used `FILE EXTERNAL` / `FILE MANAGED` without touching it (confirmed on 2 workspaces)	May not be required. Do not assume either way
`content_type` from `create_file`	`binary/octet-stream` for both Japanese and ASCII	Pass it explicitly if anything downstream branches on it

What this tells us

The difference between FILE EXTERNAL and FILE MANAGED is not only whether bytes are copied.
Whether checksum is populated and whether the filename survives both change. Reversing the
decision later means re-ingesting, so decide before you ingest.

On content_type, I got binary/octet-stream for both Japanese and ASCII input. For the same
files, list_files reported text/plain, so the two paths infer differently. If anything
downstream branches on content_type, pass it explicitly.

The state of operator support

Both the documentation and the prior article state that FILE columns cannot be used in grouping
expressions. Testing it, support splits more finely than that.

Operation	Result
`GROUP BY file`	accepted
`SELECT DISTINCT file`	accepted
`GROUP BY file.uri`	accepted (the documented approach)
`d.file = r.attachment`	rejected — The `=` does not support ordering on type "FILE"
`ORDER BY file`	rejected — The `sortorder` does not support ordering on type "FILE EXTERNAL"

What this tells us

GROUP BY and DISTINCT both need equality semantics to decide group membership. Both are
accepted while the equality operator itself is explicitly rejected. Since the error names the
type's operator surface, this does not look like row-count-dependent behaviour.

That said, this was confirmed on two rows. I cannot rule out that grouping degrades to identity
comparison rather than value equality. In practice, use GROUP BY file.uri and do not depend on
GROUP BY file being accepted. This is beta, so the behaviour may change.

Connecting to an FSx for ONTAP S3 Access Point

This is the main question. The answer is "registration works, reads do not", and the cause is
not a lack of Access Point support.

The four things registration needs

Item	Required form	Symptom when wrong
External ID in the trust policy	The Databricks account UUID	`403 Forbidden` from the storage provider. The metastore ID and the workspace ID do not work
Self-assume in the trust policy	Account root as the principal, with an `aws:PrincipalArn` condition naming the role ARN	Naming the role as its own principal fails at creation with `Invalid principal`, because IAM validates that the principal exists. Unity Catalog requires the role to be able to assume itself
Resource in the IAM permission policy	The access point ARN (`arn:aws:s3:<region>:<account>:accesspoint/<name>` and `.../object/*`)	`AccessDeniedException`. The alias-as-bucket-name form alone does not work
External location URL	The alias form (`s3://<alias>/`)	An ARN-style URL is rejected immediately with `url does not specify a valid bucket name`

What this tells us

There is an asymmetry worth committing to memory: the same Access Point must be written as an
ARN in the IAM policy and as an alias in the external location URL. The AWS CLI works with the
alias form of the ARN, so verifying connectivity with the CLI first and then writing the IAM
policy will trip you up.

The aws:PrincipalArn approach works because that key resolves to the role ARN rather than the
assumed-role session ARN. Rather than trust my memory of the condition key, I deployed the
template, assumed the role, and assumed it again with those credentials to confirm.

With all four satisfied, CREATE EXTERNAL LOCATION succeeds with skip_validation=False —
meaning Unity Catalog's own validation passed. The external volume on top can be created too.

Why reads fail

After registration, every read path returns 403 or an authorisation error. The same on
serverless SQL and on a classic DBR 18.2 cluster.

This is the test that isolated the cause. Unity Catalog will vend the temporary credentials it
uses if you ask for them. I requested them for both paths and used them from my workstation,
with no Databricks compute or network involved.

Vended for	`ListObjectsV2`	`HeadObject`
S3 general bucket control path	success	success
FSx for ONTAP S3 AP path	AccessDenied	403

Same role, same session, same network; the only variable is which path the credentials were
scoped to. The error states the cause explicitly:

is not authorized to perform: s3:ListBucket on resource:
"arn:aws:s3:<region>:<account>:accesspoint/<name>"
because no session policy allows the s3:ListBucket action

What this tells us

because no session policy allows is the decisive part. What is named is the session policy,
not the role's permissions. By session policy I mean the down-scoped policy passed to
AssumeRole to constrain the role's permissions on a per-request basis.

The policy body is generated by Unity Catalog and is not readable by the user. But since adding
an access-point-ARN allowance to the role changes nothing, the session policy appears not to
mention the access point ARN resource form at all. Only the overlap between the session policy
and the role policy is effective, so additions on the role side have no effect.

There is no workaround on the user side. Unity Catalog generates the policy, so the fix has
to happen there. Emitting the access point ARN form in the session policy should resolve it,
though I do not know what implementation constraints apply.

Note also that validation at registration time (skip_validation=False) passes while reads
return 403. What that validation checks is not published, so I cannot account for the
difference.

What changed since Part 2

Part 2 of this series (measured 2026-05) reported that listing top-level files and reading an
explicit file did work through a UC external location. In this run (2026-08-12), read_files,
list_files, to_file and dbutils.fs.ls all returned 403; no read path partially succeeded.

That environment is gone, so I cannot account for the difference conclusively. The likely
explanation is that the earlier partial successes were observed on an instance-profile path
rather than through UC-vended credentials — Part 2 lists instance-profile direct access as a
separate approach that works but sits outside UC governance, so the paths may not have been
cleanly separated.

This run had a S3 general bucket control alongside and includes the vended-credential test outside
Databricks entirely, so take these results as the current ones. I will annotate the Part 2
table accordingly.

This test has prerequisites. Enable External Data Access on the metastore and grant
EXTERNAL USE SCHEMA and EXTERNAL USE LOCATION. Both are disabled by default. They are the
controls that allow credentials to be used externally, so they are worth understanding
independently of this issue.

A table to work through when you see a 403

Arranged so that anyone hitting the same 403 can check top to bottom. None of these is the
cause.

What to suspect	Finding
Unity Catalog volume privileges	The failing volume had `READ VOLUME`; the working control volume had no privileges at all
IAM role permissions	Assumed the same role from my workstation and listed successfully. HEAD on a real object returned 200
HEAD on a missing key returning 403 (killing the `_delta_log` probe)	The Access Point returned 404 for every shape I tried — same as S3 general bucket
`x-amz-expected-bucket-owner` being rejected	Accepted by both the Access Point and the control bucket
A different endpoint	Both Databricks and the AWS CLI use `<alias>.s3.<region>.amazonaws.com`, which CNAMEs to `s3-r-w.<region>.amazonaws.com`. Same host
Insufficient compute role permissions	`sts:GetCallerIdentity` raised `NoCredentialsError` on the driver. The cluster has no default AWS credentials; the compute role is not in the path
The S3 gateway endpoint	No change after disassociating the route. The failure reproduces outside the VPC with vended credentials, so this path is not the problem

One note on the last row. A Databricks-managed VPC creates an S3 gateway endpoint at workspace
creation, and the private route tables point the S3 managed prefix list at it. That was not the
cause here, but for designs that reach an S3 AP from inside a VPC, check first whether the
alias resolves into the prefix list's range. I did not reconcile the resolved CIDRs against the
prefix list contents in this environment.

What object tags can actually hold

If you want metadata to live with the file, object tags are one option: tag at write time, read
from the table side later. That path has constraints.

The following was measured without Databricks in the picture — boto3 from my workstation
straight against the FSx for ONTAP S3 Access Point, with the same operations running against
the S3 general bucket control bucket.

Start with what works. PutObjectTagging, GetObjectTagging and DeleteObjectTagging all
function, as do x-amz-meta-* headers and x-amz-tagging on the same PutObject as the data.
The limits (10 tags per object, 128 characters for a key, 256 for a value) are the same values
S3 general bucket documents; I did not measure the boundaries.

Reading through a second access point on the same volume returned tags written through the
first. Tag retention is not scoped per access point. For existing NAS assets shared across
several paths, that property works in your favour. But I did not observe where the tags are
physically stored, so visibility from NFS or SMB, and retention across SnapMirror, FlexClone
and Snapshot restore, remain unverified.

Two constraints change how you design.

An object overwrite clears tags and user metadata. Nothing errors. A pipeline that rewrites
a file has to re-apply them in the same PutObject. My understanding is that S3 general bucket behaves
the same way, since PutObject replaces the object, so treat this as a general S3 design point
rather than something specific to FSx for ONTAP. I did not run a side-by-side control for this
particular case.

Tag values are effectively ASCII. Every printable Latin-1 character I tried
(U+00A1–U+00FF) was accepted. Above U+0100, most are rejected with InvalidTag — but not all.
A minority of CJK strings are accepted, deterministically, 6 runs out of 6.

To find the unit of validation, I tested single characters:

Input	Result
U+5206	`InvalidTag`
U+985E	`InvalidTag`
U+5206 U+985E	accepted
U+6771	`InvalidTag`
U+4EAC	`InvalidTag`
U+6771 U+4EAC	`InvalidTag`

What this tells us

Two two-character strings, every constituent character rejected on its own, and the two strings
disagree with each other. Validity is determined by the complete byte sequence, not as a
function of the characters in it — so this is not a per-character allowlist.

For the two-character pairs I tested, results were identical forwards and backwards, and
identical as a tag key and as a tag value. I did not test longer strings or other scripts.

I raised this with AWS Support. After checking it against the S3 and FSx for ONTAP
documentation, they gave the view that the pattern does not match any intentional validation
they could identify, and escalated it to the service team as a potential defect in the tag
validation layer.

Their reply also quoted the documented character set: letters, whitespace, and
+ - = . _ : / @. That does not predict the split under either reading. If "letters" means
Unicode letters, every string I tested qualifies yet half are rejected. If it means ASCII
letters, all of them should be rejected yet half are accepted.

The design guidance is simple: keep tags ASCII and keep localised text in a column. Today
the situation is "some strings pass and some do not", which is more awkward to handle than a
uniform restriction. I will update the repository and this article when a determination
arrives.

Design tips

The same material, ordered the way you actually make the decisions.

FILE type

Decide between FILE EXTERNAL and FILE MANAGED before ingesting. Checksum and filename handling differ, and changing course later means re-ingesting
Give the FileSpace its own volume. On the same volume, INSERT fails — and because the error surfaces at ingest rather than at configuration time, diagnosis is slower
Copy the filename to its own column before ingesting into FILE MANAGED. It is replaced with a UUID and the extension does not survive
Do not rely on content_type inference; pass it explicitly
Group by file.uri. Do not depend on GROUP BY file being accepted
Automatic garbage collection does not run in beta. Build manual FileSpace deletion into operations

Unity Catalog and FSx for ONTAP S3 Access Points

The external ID in the trust policy is the Databricks account UUID. The metastore ID does not work
The trust policy needs self-assume: account root as the principal with an aws:PrincipalArn condition naming the role ARN
The IAM permission policy needs the access point ARN, including .../object/*. The alias-form ARN alone does not work
The external location URL takes the alias form. ARN-style URLs are rejected
Reads do not work today. If zero-copy is a requirement, evaluate AWS-native engines that authorise in the caller's own IAM context alongside it (this repository has walkthroughs under integrations/athena, integrations/glue, integrations/emr-spark, integrations/redshift-spectrum and integrations/lake-formation)

Object tags

Keep tags ASCII and keep localised text in a column
Re-apply tags in the same PutObject when you rewrite a file. An overwrite clears them
Tag retention is not scoped per access point, so the same tags are readable through several access points

How to test

Run a S3 general bucket control alongside. Without it you cannot separate a defect in your environment from Access Point behaviour, and the first 403 looks like the answer
If the platform exposes credential vending, use those credentials from your own machine and compare. Isolating the problem is much faster without the vendor's compute in the path
Verify teardown by reconciling against a baseline. NAT Gateways do not disappear when you delete the workspace

Run it yourself

All of the above is in the repository as templates and scripts: one template, one script, one
runbook.

Artefact	Role
`integrations/databricks/uc-storage-credential-role.yaml`	The IAM role the Unity Catalog storage credential assumes, plus a S3 general bucket control bucket
`shared/scripts/probe_uc_external_location.py`	Seeds identical object tags on both sides, registers both external locations with validation on, reads `_object_metadata` through each, prints a verdict
`docs/en/databricks-verification-runbook.md`	Prerequisites, where each parameter comes from, the verdict branches, dependency-ordered teardown, measured cost

# 1. Baseline first. This is what makes "teardown looks clean" verifiable
python3 shared/scripts/audit_databricks_workspace_footprint.py \
  --region <region> --save /tmp/baseline.json

# 2. The IAM role and control bucket
aws cloudformation deploy --region <region> \
  --stack-name fsxn-databricks-uc-credential \
  --template-file integrations/databricks/uc-storage-credential-role.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides file://cfn-params/databricks-uc-storage-credential.json

# 3. Compare. --control-bucket is deliberately mandatory
.venv/bin/python shared/scripts/probe_uc_external_location.py \
  --profile <your-profile> --role-arn <from stack output> \
  --ap-alias <alias>-ext-s3alias --ap-name <name> \
  --control-bucket <from stack output> --region <region> --vend-check

# 4. Tear down and reconcile against the baseline
python3 shared/scripts/audit_databricks_workspace_footprint.py \
  --region <region> --compare /tmp/baseline.json

The script prints one of four verdicts. The first is the important one.

Inconclusive — the S3 general bucket control could not be read either, so this run says nothing about the Access Point. Fix the environment first
Registration succeeded, reads denied — the result in this article
Registration and reads both succeeded — the platform changed. Please tell me which release
Registration itself failed — almost certainly a missing Access Point ARN in the IAM policy

One note on the parameter file: a _comment key in a CloudFormation parameter file makes the
CLI fail with Unknown parameter in Parameters[n]. The description of each parameter lives in
cfn-params/README.md instead, so fill in values with that open alongside.

If you are evaluating a different platform

The specific bug is Databricks'. The shape is not, and the shape is the part worth taking away
if you are looking at a different product.

Managed lakehouse platforms generally reach into your cloud account by assuming a role you
create, then constrain that role at runtime with a down-scoped session policy so that
credentials handed to a query cannot roam. That is good design.

But the policy has to name your storage. If the storage is addressed through an access point
while the policy names it in bucket-form ARNs, the two do not match and authorisation fails —
however correct your own role policy is.

So the question to put to a platform you are evaluating, before you commit to an architecture,
can be narrow and answerable:

When you vend temporary credentials for an external location, which resource ARN forms does
the session policy contain? Specifically, does it include the S3 access point ARN form in
addition to the bucket form?

And if the platform exposes credential vending, you can verify it without asking anyone:
request the credentials, use them from your own machine, and compare a native bucket path
against an access point path. That is what --vend-check above does, and it was the only test
in this exercise that produced a conclusion with no room for interpretation.

Cost and cleanup

I created a test workspace and destroyed it the same day. The workspace mode that uses your own
AWS account creates a VPC with a NAT Gateway. A NAT Gateway runs roughly 45 USD a month per
gateway (hourly charge only, excluding data processing, at the August 2026 Tokyo region rate).

These resources are created directly rather than through a CloudFormation stack, so deleting
the workspace does not remove them. This is the easiest place to make a foolish mistake during
cleanup.

I recorded a baseline before starting and reconciled afterwards. Available NAT Gateways back to
zero; VPC, both IAM roles and the bucket gone, with no unassociated Elastic IPs.

I turned that check into a script (shared/scripts/audit_databricks_workspace_footprint.py). In
a shared account a plain listing is not a verdict, because most of what it shows belongs to
someone else. Run it once before creating anything and once after teardown; it exits non-zero if
anything exists that was not in the baseline. Running it confirmed this teardown was clean and
also surfaced leftover roles from an earlier workspace. The cost breakdown is in the runbook with
measured figures.

Current state and open items

Established

CREATE STORAGE CREDENTIAL / CREATE EXTERNAL LOCATION / CREATE EXTERNAL VOLUME succeed against an FSx for ONTAP S3 AP alias, with Unity Catalog validation left on
Reads through it return 403, on both serverless SQL and a classic DBR 18.2 cluster
The same failure reproduces outside Databricks, using vended credentials from a workstation
_object_metadata on S3 general bucket returns object tags and user metadata correctly
Object tag acceptance is determined by the complete byte sequence, and within the range tested is neither order- nor position-dependent

Not established

VPC-origin Access Points, and Access Points with WINDOWS identity
Whether _object_metadata would read tags through an Access Point if the session policy were fixed
ai_parse_document over files served from an Access Point
Throughput and latency of these paths
Whether GROUP BY on a FILE column is correct at scale, or merely accepted
Where object tags are physically stored; visibility from NFS or SMB; retention across SnapMirror, FlexClone, Snapshot restore and FabricPool tiering
The boundary values for tag limits (10 tags, 128-character keys, 256-character values)

Reporting status

The object tag behaviour has been escalated through AWS Support to the FSx for ONTAP service
team and is awaiting a determination. What I asked for is not a timeline but the distinction:
a defect that will be fixed, or an undocumented restriction that gets written down. Either
answer lets me state the constraint accurately.

The session policy issue and the operator inconsistency go to Databricks when this article
publishes.

This repository carried an entry from May 2026 saying Unity Catalog external locations do not
support S3 Access Points. Registration does work; the entry has been corrected. If you read the
earlier version, please note the change.

Closing

The FILE type is a well-built mechanism for bringing unstructured data under table governance.
For referencing files on an FSx for ONTAP S3 Access Point with zero copies, though, it does not
get there today. Registration works, so you can try it, but reads fail, and because the cause
sits in the session policy on the vending side, users cannot fix it.

If zero-copy is the requirement, AWS-native engines that authorise in the caller's own IAM
context are the more straightforward path for now — in exchange for assembling table-level
governance yourself with Lake Formation. If Unity Catalog governance is the requirement, then
for the time being data placement has to be part of the design.

The object tag character set is awaiting a determination. When it moves, I will update both the
article and the repository.

I hope this is useful to someone.

See you next time.

Previous: Part 2: Databricks and FSx for ONTAP S3 Access Points
Next: Part 8: OpenSharing and FSx for ONTAP
Series index: What works, what does not, and why

About this article

This series is based on verification in the fsxn-lakehouse-integrations repository
Author: Yoshiki Fujiwara
Affiliation: NetApp G.K. Japan
Role: Senior Cloud Solutions Architect - AWS
Topic: Amazon FSx for NetApp ONTAP / Hybrid Cloud / Generative AI Data Architecture
Level: 200-300

All test environments have been deleted. The behaviours described are measurements from
2026-08-12 in one specific configuration and may change with platform updates. Beta features are
involved, so please check against current documentation before making production decisions.

Client	Min	Median	Max	Spread
Rocky 9.7	842.70	1,040.26	1,092.50	29.6%
RHEL 9.7	1,105.75	1,112.65	1,129.30	2.1%

DEV Community: Yoshiki Fujiwara(藤原 善基)@AWS Community Builder

Why Amazon FSx for NetApp ONTAP block performance differed by about 2x on the same specified values, isolated by measurement

Why Amazon FSx for NetApp ONTAP block performance differed by about 2x on the same specified values, isolated by measurement

What was measured

The environment measured

The same configuration varied widely

The path candidate is gone

What was acting on it was when the data being read was last written

Not reads, and not elapsed time

What falls is not cache but the layout on disk

What breaks it is not writes but small random writes

The warm-side ceiling sits on the client side

At least it is not concurrency

Match the time of day when comparing configurations

On choosing between protocols

And that candidate was disproved too

Five things I hit on the instrument and the teardown

Three more hit during the rewrite

Summary

Notes on the AWS documentation

Primary record of the figures

Moving on-premises block storage to AWS? The "divide by 625 MBps" sizing approach comes out wrong

Moving on-premises block storage to AWS? The "divide by 625 MBps" sizing approach comes out wrong — measured session and queue counts

What was measured

Measurement environment

The divisor division doesn't hold

The requested queue count doesn't hold

Four failures hit before mounting

Summary

AWS documentation notes and support inquiries

Primary record of the figures

Amazon S3 Files vs. Amazon FSx for NetApp ONTAP: two designs for using S3 as a file (S3 Burst Part 2)

Amazon S3 Files vs. Amazon FSx for NetApp ONTAP: two designs for using S3 as a file, measured on the same host (S3 Burst Part 2)

Scope of this article

What it covers

What it does not cover

Measurement environment

The client-count scaling test and the sustained write used a separate file system

Six settings to change from default before measuring throughput (Part 5)

More on item 5 — the read cache's two-tier structure

Item 6 — SMB Multichannel disabled by default

Three ceilings that limit writes on the FSx for ONTAP side

Bottlenecks sitting in different places per path

FSx for ONTAP — a shared ceiling that wasn't one thing

How the ceiling splits as client count grows to 8

A hypothesis that turned out wrong

A single-client figure that was actually the client's own limit

The shared ceiling split two ways over SMB too

Amazon S3 — a total proportional to client count

Amazon S3 Files — the CPU of a proxy process on the client

Absence of decay over a 15-minute sustained write

Capacity growth from continued writing

15 minutes over SMB

Additional read capacity via FlexCache

Bottleneck shift with small objects

Per-tool visibility

AWS documentation notes and support inquiries

What hasn't been measured yet

Closing

Amazon FSx for NetApp ONTAP NVMe/TCP assumes RHEL 9.3: why multipathing fails on Amazon Linux 2023

Amazon FSx for NetApp ONTAP's NVMe/TCP procedure assumes RHEL 9.3: why multipathing is unavailable on Amazon Linux 2023, and how to check before you are billed

What was checked

The environment I measured

Behaviour on this AMI

iopolicy on a kernel with multipathing, and the value the procedure states

How to check whether multipathing is using the second path

Bytes per path can be read from two places

Counted properly, the second path was not running

Summary

AWS documentation notes and support inquiries

Primary record of the figures

AWS Transform Now Supports Block Storage Migration to FSx for ONTAP — Benefits and Pitfalls from a Hands-On EC2 Test

Introduction

Test Assumptions and Scope

The Shape of the Configuration

The Most Important Distinction: What Became GA Is the MGN Target

Environments This Configuration Suits

Environments Where Another Option Fits Better for Now

A Mini Glossary

The Procedure and the Measured Time for Each Step

DEV Community: Yoshiki Fujiwara(藤原善基)@AWS Community Builder

`iopolicy` on a kernel with multipathing, and the value the procedure states