DEV Community: Yoshiki Fujiwara(藤原善基)@AWS Community Builder

Turning a Step Functions workflow into AI agent tools through AgentCore Gateway

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 01 Nov 2026 12:35:21 +0000

Part of the FSx for ONTAP S3 Access Points serverless patterns series

The update this post is about

On October 29, 2026, AWS Step Functions workflows became usable as AI agent tools through Amazon Bedrock AgentCore Gateway. The agent decides when to run a workflow and what input to pass; Step Functions runs each step as you defined it. You can turn existing Standard / Express workflows into tools without rewriting them.

Announcement: AWS Step Functions workflows now available as AI agent tools (AWS What's New, 2026-10-29)
Documentation: AWS Step Functions as Connector Target

This post applies that update to an FSx for ONTAP S3 Access Points file-processing workflow and adds it to the repository as a sample pattern. The key points:

Standard workflow (discover → process → report) turned into tools via the connector
The agent decides when to run; the state machine owns the order of the steps
Gateway and connector target described in CloudFormation, so sam deploy alone completes it
Verified on a live ap-northeast-1 deploy (DemoMode, 3/3 files)
No performance or cost numbers (it was a smoke test, not a measurement)

What this post covers

What the Step Functions connector makes possible (within the official docs)
The shape of the pattern I added, and how it differs from the existing AgentCore integration
The design points that are easy to trip over

What this post does not cover

Performance and cost numbers (I ran it live, but that was a smoke test, not a measurement)
Express workflows and synchronous responses (StartSyncExecution) — this pattern stays on Standard
Cedar authorization on the input body — the official behavior is limited right now (see below)
The agent implementation itself (Strands, etc.) — this post is about the tool-provider side

Why I did this

This series has been building serverless patterns that process files over S3 Access Points. The processing itself is usually a Step Functions workflow: discover, process each file, report. That shape already runs in several patterns.

Then this update landed: you can turn a workflow into an agent tool. A workflow that used to be triggered by an EventBridge schedule or by hand now gets an entry point for an agent. You keep the business logic as-is and move the launch and status-check to natural language. Reusing existing assets without rewriting them looked worthwhile, so I tried it in the context of the series.

The post that got me started is below. It uses a purchase-order-and-approval scenario and goes as far as task tokens and Cedar, which made the behavior concrete.

[Update] You can now use AWS Step Functions workflows as AI agent tools via AgentCore Gateway (Classmethod, JA)

Where this fits

The subject of this series is a workflow that processes files on a file store (FSx for ONTAP) through S3 Access Points. The discover-process-report shape maps directly onto business document processing. AWS itself shows a concrete pairing of Step Functions with AgentCore in an architecture blog on automating medical coding, and intelligent document processing — classifying and extracting from contracts, invoices, applications — is a recurring subject in the AWS solution guidance as well. A few situations where this connector fits:

When you want to add a natural-language entry point to a file-processing batch. A workflow you used to start on a schedule or by hand can now be started from a conversation ("summarize the unprocessed files in that folder") and followed to its result. The workflow internals stay as they are.
When you want the steps fixed but the timing left to a person or an agent. The order of classify, extract, notify and the business rules stay pinned in the state machine; only "when to run it and what input to pass" is handed to the agent.
When you want a human approval before a sensitive action. With task tokens you can pause the workflow partway, wait for a person's decision, and resume on approval (human-in-the-loop). AWS shows a reference flow that wires event-driven signals to human approval.

The readers I have in mind are roughly these:

Platform / operations people who run existing Step Functions workflows and want to add an agent entry point without rewriting the business logic — shifting start and result-checking toward a conversation.
Application / line-of-business developers automating document processing who want business users to start and check runs themselves, while the fixed steps stay in the workflow.
Security-minded people who want to bound what the agent can reach. Which state machine and which operations become tools is declared, so the agent's reachable surface is something you can read as code.

This overlaps with building the agent integration yourself (writing the agent with something like Strands and authoring your own tools) and with visual workflow builders. I read the connector approach as the one with the least extra implementation when "there is already a Step Functions workflow" matches your situation. It is less about which is best and more about picking the entry point that fits the assets you already have.

What became possible

AgentCore Gateway gained a built-in Step Functions connector (connectorId: "states"). When you create a target, you pick the operations to expose and the state machine, and the Gateway generates the tool definitions. You write neither a Lambda nor an MCP server.

Per the official docs, the connector exposes ten operations. Four of them (StartExecution / StartSyncExecution / ListExecutions / DescribeStateMachine) pin a state machine ARN on the target. The pinned ARN never appears in the tool arguments, so the agent cannot run a different state machine. The other six take an execution ARN or a task token the agent supplies.

The division of labor: the agent decides when to run and what input to pass, while the ordered steps and the business rules stay on the state machine side.

The pattern I added

It lives at solutions/genai/stepfunctions-agentcore-tools/. The subject is the S3 AP file-processing workflow that is familiar from this series.

Figure: exposing the workflow as agent tools through the Step Functions connector (dark theme here)

Discovery: scans the S3 AP and builds the list of files to process
Processing: reads one file, summarizes/classifies with Bedrock, scores confidence, and attaches a human-review decision
Report: aggregates, adds a data-classification label, and sends an SNS summary

With a target named workflow, the agent sees tools like workflow___StartExecution. It starts the workflow, receives an executionArn, checks status and result with workflow___DescribeExecution, and reads how far it progressed with workflow___GetExecutionHistory.

How it differs from the existing AgentCore integration

This repository already has a Lambda-target AgentCore integration. This one uses the connector, so the target is a different thing.

Lambda target: the Lambda becomes the tool; you write the tool definition yourself
Connector: the state machine's operations become the tools; the Gateway generates the definitions; the Lambda is only invoked as a workflow Task and is not exposed to the Gateway

It completes in IaC

The Gateway and the connector target are available as CloudFormation resources (AWS::BedrockAgentCore::Gateway and AWS::BedrockAgentCore::GatewayTarget). So they go into the same template as the Lambdas, the state machine, and the role, and sam deploy alone gets you all the way to the gateway's tools/list — no separate command to create the gateway. The deploy script is a thin wrapper that runs sam build and sam deploy and checks tools/list at the end.

Design points that are easy to trip over

Points I noticed while building. Let me separate them by source. The connector configuration schema and the Gateway role IAM I confirmed myself, from the CLI and the CDK sample as above, and then on a live deploy (below). The same-region and pinning constraints are from the official docs. The Japanese mojibake and the Cedar item I did not exercise on this live run — I am citing another post's report.

The Gateway and the state machine are in the same region. The Gateway calls Step Functions in its own region. A state machine in another region cannot be wired up.
One pin per operation. To expose several workflows under the same operation, use one target per workflow.
Tool descriptions are generic. The shape of the input JSON is not in the tool. Tell the agent in the system prompt or a skill.
Standard and Express differ in which operations work. I chose Standard because it keeps an execution history, so ListExecutions and GetExecutionHistory work. If you need a synchronous answer, that is a separate look at Express and StartSyncExecution.
Treat task tokens as secrets. SendTaskSuccess and friends cannot be scoped by IAM; anyone holding a valid token can complete the task. This pattern does not expose them.
Input-body authorization is not expressible in Cedar right now. A policy conditioned on the input body (an amount, a reason) currently hits a type-evaluation error — the Classmethod post above reports the same observation. Keep body-level decisions on the state machine side. Per-tool allow/deny does work in Cedar.

That said, each of these sits well with the point of the pattern: don't hand everything to the agent; pin the steps you want enforced into the workflow. If you want to branch on the input body, that was a Choice state in the workflow all along.

On Japanese input

The Classmethod post reports that Japanese in the input passed through the Gateway can get mangled, with a workaround of escaping the Japanese in the JSON to \uXXXX before sending. I have not confirmed this on my own deployment yet, so the docs record it as a reported known behavior together with the workaround.

Running it on real infrastructure

I deployed it to the Tokyo region in DemoMode and exercised the agent-facing path end to end. The gateway's tools/list showed all five tools; StartExecution returned an executionArn, DescribeExecution gave me status and result, GetExecutionHistory showed the step transitions, and ListExecutions listed the runs. The workflow succeeded, all three files were processed, and the category and data-classification labels came through. The Gateway and connector target came up with the stack create and went away with the stack delete.

On completeness

This pattern is scoped to exposing a Standard workflow as tools. It does not include Express synchronous responses, task-token human-in-the-loop, or a Cedar build-out. I confirmed each is possible in the official docs, but cramming all of them into one sample blurs the point, so I drew the boundary.

There are no performance or cost numbers either. I did run it live, but that was a smoke test, not a measurement — running once and running under load are different things. When I measure, I will write it up with the environment (region, model, file count, date) attached.

Wrapping up

Being able to turn an existing Step Functions workflow into agent tools without rewriting it was a welcome update for me. The split — steps in the workflow, launch decision with the agent — holds with no extra implementation.

I ran it live end to end and confirmed I could start a workflow from the agent and follow its result. What is left is seeing the Japanese mojibake and the Cedar behavior with my own eyes, and measuring performance and cost under load. Once I do that, I will write the follow-up with numbers. The use cases are something I want to dig into from there too.

The implementation is on GitHub.

Repository: FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns
Previous (a published S3 AP post): the 28-patterns post

I hope this helps anyone who wants to reach an existing Step Functions workflow from an agent.

Bye now!!

Yoshiki Fujiwara

Modernizing an App on Amazon FSx for NetApp ONTAP: Four .NET / Java / Python Candidates, Plus Containers and AI — Six Paths Measured (Once per Case)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sat, 31 Oct 2026 02:08:18 +0000

Changing the runtime and the access path without moving the data: how authentication, authorization, and failures differ

A long-lived business application often keeps the data it reads and writes on a file share. For rebuilding such an application on a newer language or runtime, Microsoft Learn lists dependencies reaching end of support, new security or compliance requirements, and performance or scalability limits as the signals (Upgrade .NET apps overview). Its examples include moving a Windows-only .NET Framework app to cross-platform modern .NET, and beyond that, containerizing the app or replacing components with managed services.

When you rebuild an application, what usually stalls the effort is not the application itself but what to do with the data it has been reading and writing. Moving the data to a new store brings along downtime during the migration, redesigning permissions, re-pointing reference paths, and re-validating on the destination. This bundle is what stops modernization.

Not moving the data removes that whole bundle, because it decouples rebuilding the app from migrating the data. You get three things:

You avoid the migration work itself. No downtime, no permission redesign, no path re-pointing, no destination re-validation. You can focus only on changing the app's language or runtime.
The data stays a single source of truth that several runtimes share at once. No extra copies to keep in sync. (Keeping the data in one place so copies and synchronization are unnecessary is the single-source-of-truth, SSOT, idea.) This article measured six paths reaching the same single volume, with the data unchanged.
You can choose the language and runtime independently of where the data lives. The same data is reachable from .NET, Java, or Python, and from EC2, Lambda, or EKS.

This matters when the existing data is large and costly to move, when the permission design is complex and you would rather not rebuild it on the destination, when several users or services use the same data at once, or when you cannot take the downtime a migration needs.

Amazon FSx for NetApp ONTAP (FSx for ONTAP below; NetApp's ONTAP as an AWS-managed shared storage service) makes this possible, because it lets you use one volume through several doors: NFS, SMB, and the S3 API. AWS documents that S3 Access Points let you read and write file data with S3 object operations, without application changes and without moving the data out of file storage, and that you can keep using NFS and SMB alongside (Accessing your data via Amazon S3 access points). On the container side, the Amazon EKS User Guide describes using NetApp Trident so that FSx for ONTAP serves as the storage behind Kubernetes persistent volumes (Use high-performance app storage with FSx for NetApp ONTAP).

But when you add access paths without moving the data, who authenticates, where authorization happens, and how failures return change per path. Add a path without knowing this, and you find the difference afterward. So this article measures reaching the same data through six paths and how authentication, authorization, and the shape of failures swap per path.

Last time I took one .NET Framework app and, keeping its business logic unchanged while renewing only the language and runtime — the kind of migration AWS calls Refactor — rebuilt it step by step down to AWS Lambda without moving its data, and measured how SMB, NFS, and S3 Access Points differ when opening the same volume. This article continues from there by adding the runtimes and paths that reach the data. I measured SMB last time, so I did not measure it this time.

Even after the app migration, the access path changes with how you run the app in the cloud: whether the same data is mounted from EC2, read with the S3 API from Lambda, or used from a container on EKS. This time I measure that next step after the migration — choosing the access path — by adding runtimes.

I measured the following six paths. I call E-1 and E-2 together E.

Path	What uses the volume, and how
A	A .NET 10 app on a Linux EC2 instance mounts the volume over NFS and reads and writes it
B	A Java (Spring) app on a Linux EC2 instance mounts the volume over NFS and reads and writes it
C	A Python Lambda reads and writes the volume through S3 Access Points (the S3 API)
C-REST	A Python Lambda reads the volume through the ONTAP REST API
D	Containers on EKS write to persistent volumes that Trident provisions
E-1	Athena (an analytics service) reads Parquet files with SQL through S3 Access Points
E-2	Amazon Bedrock Knowledge Bases (a RAG service) ingests documents through S3 Access Points, and a filter inside a web app narrows the search results to the user's permissions

Four of them (A, B, C, C-REST) ran the same file-processing job, reimplemented in .NET, Java, and Python, against the same single volume. Because the job is the same, these four can be compared with the difference attributed to the path. I call each reimplementation a "candidate" in this article.
D and E are not the same job. I tested different workloads, in other repositories, in other disposable environments. I do not compare them with A through C-REST; I report each path's behavior on its own.

I did not run the same job over all six paths.

First, the premises. All six paths ran on the same single file system (Single-AZ, ap-northeast-1; an existing verification file system, separate from the one I measured in stages 0 to 3 of the previous article), using disposable SVMs, volumes, and other resources I created. I read the ONTAP version in the measurements for candidates A, B, C-REST, and path D (across several days): 9.18.1 (NetApp Release 9.18.1P6). The C and E measurements did not read it and carried the value over. A different version can change a result, so read the results for C and E as results whose version I did not confirm at the time. A "case" here is a check fixed beforehand for each path (that it runs as expected, that it fails as expected on a deny or an unreachable target, and so on), and I ran each case once. I did not repeat runs to confirm reproducibility. None of these values is a guarantee for production.

For A through C-REST, the job I measured is the core of DocIntake, the same sample app as last time. It is not a real business app or a production app. It has no screen (UI) and no web layer: a minimal batch job that reads files on the volume, processes them, and writes a result, reimplemented per language. The results do not carry over to an app with a UI.

You may be wondering about these three points.

When the same file-processing job runs in languages other than .NET (Java, Python) over NFS, S3 Access Points, and ONTAP REST API, what changes in authentication, authorization, and how failures return?
When containers on EKS, or analytics and RAG services, use the same volume, what could I confirm, and what is outside this article?
Which of the six paths should I choose, and on what basis?

I found three things. I answer the three questions above in this order.

The answer to the first question is that the authentication principal, the network boundary, the authorization principal, and how failures return all change per path. For example, when access was denied, NFS (A and B) returned an OS permission error, S3 Access Points (C) returned HTTP 403 (AccessDenied), and ONTAP REST API (C-REST) returned HTTP 403 with an ONTAP error code. The four candidates in .NET, Java, and Python ran their designed cases, without moving the data, against disposable volumes on the same FSx for ONTAP file system.
The answer to the second question has three parts. On D, I wrote data from a verification pod on EKS to iSCSI and NFS persistent volumes that I provisioned with Trident, and the same data was readable after a pod restart and after a reschedule to another node. On E-1, Athena read Parquet through S3 Access Points and the aggregation matched the values I computed from the source data. On E-2, a filter inside the web app removed documents the user was not permitted to see from the Bedrock Knowledge Bases search results, according to the user's permissions. All three are small-scale results for workloads other than A through C-REST.
The answer to the third question is that authorization happens in a different place per path, so choose by the kind of client you use and where you want authorization to happen. The evidence is the tables "The paths at a glance" and "How failures return per path, and what was confirmed for D and E". I do not rank them.

What you can take back to your own environment is this. Each time you add a runtime or a path to the same volume, re-measure on the new path who authenticates (the OS, IAM, an ONTAP credential, the app), where the network goes (an NFS mount, the public S3 endpoint, a management LIF inside the VPC), where authorization is decided (OS permissions, IAM, ONTAP, a filter in the app), and how a failure returns (a permission error, an HTTP status, an ONTAP error code, removal from the search results).

Intended readers, and what this article does not answer

The intended readers are SAs, application developers, storage administrators, and Partner/SI engineers who are modernizing applications on AWS, or considering it. Not only those already using FSx for ONTAP, but also those still choosing a destination. I picture the moment of being stuck on "I want to change the language or runtime, but what do I do with the data?", or of choosing between NFS, S3 Access Points, ONTAP REST API, and Trident.

This article does not answer the following, because I did not measure them or measured them under different conditions.

Performance (I ran no measurement with defined performance conditions).
Guarantees for production (every value is one observation per case).
Multi-AZ behavior (I built only Single-AZ).
Behavior of an app with a UI (the A through C-REST sample has no UI).
Large data or long-running operation (I used only small datasets).

Reading order

To get the conclusion only, "I found three things" in this section is enough.
To see the differences between paths at a glance, read "The paths at a glance" and "How failures return per path, and what was confirmed for D and E".
To decide which path to choose, read "How to choose a path".
Per-path detail is in the sections for candidates B, C, C-REST, path D, and path E.
The axes this article does not cover are collected in "What this article covers / does not cover".

What this article covers / does not cover

This article reports six paths, each measured once per case in its own environment. It is not an exhaustive comparison.

What I cover

How authentication, network, authorization, and failure behavior differ when six paths use the same volume. A, B, C, and C-REST ran the same job over different paths, so I can attribute the difference to the path. D and E are different workloads, so I report each path's behavior on its own.
How the failure cases (deny, unreachable, invalid identifier) returned on each path. What returns differs per path (a permission error, HTTP 403, an ONTAP error code, a connection timeout).
Each case was measured once. Where I got through by hand, and where I did not retrieve a value, I say so on the spot.

What I do not cover

Performance. The purpose was to measure behavior differences per path, and I ran no measurement with defined performance conditions. Performance and scale are tracked in #38.
Multi-AZ and other ONTAP versions. I built only a Single-AZ configuration, and the measurements that read the ONTAP version read 9.18.1P6. Path D's Multi-AZ switchover and same-file concurrent-write consistency are tracked in #17 in Container-Datastore.
Response time. I did not measure path E-2's response time or the natural expiry of the Lambda cache (waiting five minutes); they are tracked in #209. Path E-1's behavior when the domain controller is unreachable is #155 in Lakehouse.
SMB. I measured it in the previous article. I also do not cover containers on ECS or Windows containers that mount over SMB. For path D, I tested only one path, EKS with Trident.
Reconciling with invoices. Costs are estimates for approved disposable environments, pre-bill values.
Whether this applies to real data. E-1 and E-2 were measured with non-confidential synthetic data and placeholder SIDs. Data classification, retention policy, and any legal or compliance judgment before applying it to production business data or regulated data are outside this article.
Whether to use the RAG's generated output. E-2 measured only the permission-based narrowing of search results; whether a generated answer built from the retrieved documents may be used directly for a decision (clinical or otherwise) needs separate consideration.

Terms used in this article

Term	One-line meaning
Path	The route by which a runtime reaches a volume on FSx for ONTAP. In this article there are six: A, B, C, C-REST, D, and E
Job	The core processing of the sample app DocIntake: a minimal batch unit that reads a document from an intake folder, processes it, and writes a result report
Candidate	A different implementation of the same job. Used only for A, B, C, and C-REST. D and E are not the same job, so I do not call them candidates
The same job	Running the job above as the same processing even when the language or the path changes
NFS	The protocol Linux uses for file sharing. This article uses NFS 4.1 with AUTH_SYS (a simple authentication where the client claims a uid)
S3 Access Points	A door that reads and writes an FSx for ONTAP volume through the S3 API. AWS IAM, the access-point policy, and a file-system user authorize requests. This is an S3-compatible door, not an Amazon S3 bucket itself. The backing store is the same ONTAP volume as NFS/SMB, so reaching one volume through several protocols shares its throughput
ONTAP REST API	A REST door for managing and operating ONTAP. Authentication is an ONTAP credential and ONTAP authorizes
Management LIF	ONTAP's network interface for management. It is reachable only from inside the VPC
SVM	ONTAP's virtual storage server. In this article I created a disposable SVM per verification
Trident	NetApp's CSI driver, which provisions Kubernetes persistent volumes (PVs) as FSx for ONTAP volumes and LUNs
PV and PVC	A PV is a persistent volume Kubernetes manages; a PVC is a pod's claim for one
RAG	An approach in which a generative model answers using documents retrieved by search
SID	A value that identifies a Windows user or group. In the RAG verification, the SIDs that permit each document are matched
fail-closed	Erring toward the safe (closed) side when information is missing. In this article it refers to E-2's runtime behavior: a user with no permission information gets all search results denied

The delta from the previous article

The previous article (the previous article in this series) rebuilt one .NET Framework app from stage 0 to stage 3 and measured how file handling (case sensitivity, exclusive locking, write visibility, and so on) differs as the ways to open the same volume increased across SMB, NFS, and S3 Access Points. It had one implementation (.NET), and its axis was how file handling changes when the protocol changes.

This time three axes change. One is more languages (Java and Python in addition to .NET). Another is a new access path, the ONTAP REST API. With these two, what I look at moves from file handling to authentication, network, authorization, and how failures return. The third is workloads other than the job itself that use the same volume: containers on EKS (D) and analytics and RAG services (E), which I tested in other repositories.

Candidate A (.NET / NFS) is the same lineage as stages 2 and 3 last time, so I keep it to a summary and point to the previous article and the issue that records its runtime verification (#42). What I newly measured is B, C, C-REST, D, and E.

The paths at a glance

Here are the six paths in one table. Only the four with "Yes" in the "Same job?" column can be compared with each other.

Path	Same job?	Runtime	How it reaches the volume	Authentication and authorization	Repository where tested
A	Yes (DocIntake)	.NET 10 / Linux EC2	NFS 4.1 mount	The OS (uid) authenticates; OS permissions authorize	This repository
B	Yes (DocIntake)	Java / Spring worker / Linux EC2	NFS 4.1 mount	The OS (uid) authenticates; OS permissions authorize	This repository
C	Yes (DocIntake)	Python 3.14 Lambda (outside the VPC)	S3 Access Points (S3 API)	AWS IAM authenticates; IAM and the access-point policy authorize	This repository
C-REST	Yes (DocIntake)	Python 3.14 Lambda (inside the VPC)	ONTAP REST API (management LIF)	An ONTAP credential (basic auth); ONTAP authorizes	This repository
D	No (a verification pod using Trident PVs)	EKS 1.35 / EC2 nodes	PVs Trident provisions (iSCSI, NFS)	Trident authenticates to the management LIF with a vsadmin credential and provisions the volumes	Container-Datastore
E-1	No (an Athena SQL query)	Athena (managed)	S3 Access Points (S3 API)	AWS IAM authenticates; IAM and a file-system user authorize	Lakehouse
E-2	No (a Bedrock Knowledge Bases search)	Bedrock Knowledge Bases and a web app	S3 Access Points as the ingestion source	Users authenticate with Cognito; a SID match inside the app decides per-document access	Agentic-RAG

As a diagram: the top tier is the four candidates for the same job (A and B on Amazon EC2, C and C-REST on AWS Lambda), the bottom tier is the two paths for different workloads (D on Amazon Elastic Kubernetes Service, E on Amazon Athena and Amazon Bedrock), and both reach one Amazon FSx for NetApp ONTAP volume.

Figure 1: Six paths reaching one FSx for ONTAP volume. The top tier is the same job's candidates; the bottom tier is different workloads (dark theme version).

Verification environment and method

Configuration (A, B, C, C-REST)

Region ap-northeast-1, Single-AZ. The ONTAP version is 9.18.1 (NetApp Release 9.18.1P6), read in the measurements for candidates A, B, C-REST, and path D. For C and E I carried the value over from the same file system and did not read it again in those measurements.
I used the existing file system (an existing verification file system that I keep, separate from the one I measured in stages 0 to 3 of the previous article) as the host, and created a disposable SVM and volume per candidate. I did not touch the existing 5 SVMs and 31 volumes. After each verification I deleted the disposable resources and confirmed that the existing 5 SVMs and 31 volumes were still present.
Candidates A and B mounted over NFS 4.1 from a Linux Amazon EC2 instance (Graviton arm64, AL2023). Candidate C reached FSx for ONTAP S3 Access Points from a Lambda outside the VPC, over the public S3 endpoint. Candidate C-REST reached the ONTAP management-LIF REST API from a Lambda inside the VPC.

Configuration (D, E-1, E-2)

I measured D and E in other repositories, on other days. In each case I added disposable resources to the same file system as A through C-REST, and deleted them after verification.

Path	What was disposable	Confirmation that existing resources were untouched
D	A dedicated SVM (with its own vsadmin), an EKS cluster (Kubernetes v1.35), two m5.large nodes (Amazon Linux 2023), and the Trident add-on v26.6.2	Before and after teardown, this file system's 5 backups, 5 SVMs, 31 volumes, and 72 ENIs in the VPC were identical
E-1	One volume, one internet-origin S3 Access Point, a Glue database and table, and an Athena results bucket	After teardown, the existing file system, its 5 SVMs, and its 8 access points were unchanged
E-2	One volume, one internet-origin S3 Access Point, a Bedrock Knowledge Base, an S3 Vectors index, and a CDK stack including the web app	After teardown, the existing file system, SVMs, access points, and volumes were unchanged

I read the ONTAP version in the path D measurement (9.18.1P6) and did not read it in the E measurements. The SVM used for E-1 was joined to Active Directory, and the SVM used for E-2 was not.

Method

For each path I fixed beforehand, as the success criteria, both the cases that run as expected and the failure cases (that fail as expected on a deny, an unreachable target, or an invalid identifier), and recorded the real responses. The failure cases are there to confirm not only "it ran" but also "the failure returns in the expected shape." If you only watch the happy path, you miss a request that should be denied slipping through, or an unreachable target quietly returning empty. Because the shape of a failure swaps per path (a deny as an OS permission error vs. HTTP 403; an unreachable target as an exception vs. a timeout), I measured the failure cases per path as much as the normal ones. Completion was judged not by a build or a call succeeding, but by each normal and failure case returning the expected real response. The real responses for A through C-REST are in a private record. For D and E they are in the public records of each repository, linked in "Resources".

Each case was measured once, without repeating runs to confirm reproducibility. Where I got through by hand, and where I did not retrieve a value, I note it on the spot.

Candidate A (.NET / NFS): the previous article in summary

Candidate A is a .NET 10 BackgroundService that processes files on a volume mounted over NFS 4.1. It is the same lineage as stages 2 and 3 last time; I confirmed four cases.

Two findings, briefly. One is the systemd unit type: Type=notify stalls startup because the worker sends no sd_notify, and the correct type is Type=simple. The other is that the local promotion gate (NfsPromotionGate) validated source_revision with a 64-hex (SHA-256) regex, rejecting a 40-hex Git SHA-1 commit id. Details are in the previous article and #42.

Candidate B (Java / NFS): FileLock behavior is invisible without two processes

Candidate B runs a Java/Spring background worker on Linux, processing a volume mounted over NFS 4.1. I confirmed file processing, lock acquire/release, cross-process lock contention, same-JVM lock overlap, a platform I/O error, and wait/retry, using two hosts.

File processing processed a document on the NFS mount and wrote a report with the expected length and first line.
File lock acquire/release recorded acquired_released, report-write and claim-release for each item, with a clean Spring start, item success, normal stop, and exit 0.
Cross-process lock contention ran two JVMs on the NFS mount. While the holder kept the lock, the contender got tryLock=null and was classified lock_contention.
Same-JVM lock overlap threw OverlappingFileLockException when opened twice within one JVM, classified same_jvm_contention.
A platform I/O error was recorded as platform_io_error with operation, platform, and exception.
Wait/retry: a bounded wait was measured on two hosts over NFS (below). Past the limit, a null is still lock_contention, and nothing is retried silently.

Wait/retry was measured once each on two hosts (ONTAP 9.18.1P6, NFS 4.1, Amazon Linux 2023 arm64; the SVM's lease is 30 s, grace 60 s). The waiter took the claim 198 ms after the holder released it (250 ms retry interval); a 10 s limit ended as lock_contention after 41 attempts; with no wait it was lock_contention after one attempt. After kill -9 of the holder JVM the claim was taken 162 ms later; after a forced stop of the holder's host it was taken about 75 s after the stop request (a single observation that does not separate stop latency from lease expiry). During the measurement, a start-signal file placed on the NFS volume was not seen by the second host for about 30 s; the cause is unconfirmed, and the two-host start signal was switched to the process's standard output.

A guide to setting the wait limit in production. When the holder releases normally, the claim was taken in a few hundred milliseconds, so a short limit is enough. But if you wait for recovery from a holder host failure, the lock only clears after the lease expires, so the limit must be longer than the lease (30 s) plus grace (60 s), about 90 s, or you give up before recovery. Decide by whether you prioritize responsiveness or not dropping work on a failure.

One behavior stood out. A null java.nio.channels.FileLock (cross-process lock contention) is observable only with two processes. Within one JVM the platform throws OverlappingFileLockException (same_jvm_contention). The NFS mount enforced cross-process exclusion as expected.

There was also a build-environment trap: Maven's default JDK on AL2023 did not accept release 21, and JAVA_HOME had to point explicitly at Corretto 21, otherwise mvn failed with "release version 21 not supported."

Candidate C (Python Lambda / FSx for ONTAP S3 Access Points): IAM authorizes

Candidate C is a Python 3.14 Lambda that reads and writes volume objects through FSx for ONTAP S3 Access Points. On a disposable UNIX SVM and volume not joined to Active Directory, I created an internet-origin access point and reached it from a Lambda outside the VPC over the public S3 endpoint. I confirmed five cases; file locking was out of scope.

PutObject wrote an object.
GetObject returned a read-back hash equal to the write input hash (roundtrip hash match).
ListObjectsV2 returned a listing.
Policy-condition deny: with an explicit Deny variant, a GetObject through the access point returned AccessDenied / HTTP 403. A before/after control (writable before the Deny, denied after) isolated the policy effect.
Invalid parser input classified invalid bytes locally. It is not a managed-Lambda result.
File locking was out of scope because this operation set does not express file locking.

The finding is that FSx for ONTAP S3 Access Point object operations (Get/Put/List) ran from a Lambda outside the VPC over an internet-origin access point on a UNIX SVM not joined to Active Directory, with no dependency on reachability to an AD domain controller. A deny returned through the access point as the exact AccessDenied / HTTP 403 class, and the before/after control isolated the policy effect.

Candidate C-REST (Python Lambda / ONTAP REST API): ONTAP authorizes, plus the self-signed certificate and multipart traps

Candidate C-REST is a Python 3.14 Lambda that reads files through the ONTAP REST API. It is the same Python Lambda as candidate C, but the access path is entirely different. Candidate C uses the public S3 endpoint and AWS IAM authorizes. C-REST has a Lambda inside the VPC reach the management-LIF REST API, authenticate with an ONTAP credential (basic auth), and let ONTAP authorize. I confirmed a read, recording the result, a write, a multi-call read, and four failure cases (ONTAP 9.18.1P6).

File listing read: a GET on the files resource returned a listing, HTTP 200, Accept application/json.
Recording the read result: the read result was recorded to a separate bucket, and its hash was independently confirmed to match the read's output hash.
Write: 2.5 MB written in three calls. The ONTAP REST API creates a file with a multipart POST and then writes with PATCH at a byte offset (a JSON POST to create first returned HTTP 405). The read-back hash matched the write hash — but the read-back uses the same REST API, so it is not independent of the write path.
Multi-call read: the size came from the parent directory listing, then four ranged reads reassembled the whole file and the hash matched.
Single-call size limit: 1,048,576 bytes returned HTTP 200 and 1,048,577 returned HTTP 413 (no body), confirming the documented 1 MB per-request limit.
Auth failure: a wrong credential returned HTTP 401, ONTAP error code 6691623, and no bytes were served.
Authorization deny: with a read-only user, GET was allowed and a write returned HTTP 403, ONTAP error code 6, "not authorized for that command."
Network unreachable: toward an unused address, a connection timeout was classified endpoint-unreachable, and no bytes were served.
Invalid identifier: a valid volume identifier with a non-existent path returned HTTP 404. A malformed volume identifier instead returned HTTP 200 with an empty listing, so I measured the invalid identifier on the path.

There are four findings about the shape of the API when you call the ONTAP REST API from a client. A file is created with a multipart POST and then written with PATCH at a byte offset (a JSON POST returns HTTP 405). A read response returns two entries (bytes_read and the data), so only the data is hashed. A JSON listing of a file path returns HTTP 400 "Not a directory", so the size comes from the parent directory listing. Exceeding the single-call read limit returns HTTP 413, which maps to its own class. These are recorded in #73.

There is also a TLS finding, and it is not about S3 Access Points (Candidate C). When you reach FSx for ONTAP through S3 Access Points, an AWS-managed endpoint terminates TLS, so the client does not deal with a certificate (putting a certificate on a custom domain is AWS Certificate Manager territory, separate from any ONTAP-side certificate). The ONTAP REST API (Candidate C-REST), by contrast, calls the ONTAP management endpoint (the cluster-management LIF), where ONTAP itself — not an AWS-managed TLS endpoint — presents the certificate. ONTAP presents a self-signed certificate by default (ONTAP management protocols and the TLS handshake, generating a self-signed certificate for FSx for ONTAP), so by default a certificate-verifying call fails. This failure must be classified as a TLS-trust failure, not "unreachable." Python's urllib wraps the SSL error inside the URLError reason, so you unwrap the reason before classifying. The option to disable certificate validation is used only as an explicitly recorded diagnostic, never as a silent bypass. In production, install a certificate signed by a trusted CA on ONTAP and have the client trust that CA.

The Accept header needs care too. An ONTAP REST API listing does not accept multipart/form-data (HTTP 406, ONTAP error code 262269). A directory listing must use Accept application/json, and a byte read branches to multipart with a length.

There is also a finding on FSx for ONTAP role scope. At the file-system level the roles are fixed (fsxadmin, fsxadmin-readonly); custom roles are definable only at the SVM level. An SVM-scoped user gets HTTP 401 on the management LIF (there is no SVM REST management LIF). The read-only role passes all GETs, and a write returns HTTP 403 / ONTAP error code 6.

Path D (EKS and Trident): using FSx for ONTAP as a container's persistent volume

Path D is not the same job. I tested, in the Container-Datastore repository, a path in which containers on EKS use FSx for ONTAP volumes as persistent volumes through Trident. Unlike A through C-REST, it does not read the data of an existing volume: Trident provisions new volumes and LUNs for the containers.

What I confirmed is whether data written by a container remains after a pod restart and after a reschedule to another node. I created two StorageClasses. ontap-san is iSCSI, with RWO, which only one pod writes to. ontap-nas is NFS, with RWX, which several pods share. I provisioned one 1 GiB PVC from each.

Type	Check	Result
iSCSI / RWO	PVC provisioning	`Bound` in about 10 seconds
iSCSI / RWO	After writing a marker, after a pod restart (same node), and after cordoning the node and deleting the pod (other node, other AZ)	The marker matched all three times. The new pod started only after the old pod had finished deleting. Both nodes mounted the same multipath device (ext4)
NFS / RWX	PVC provisioning	`Bound` in about 10 seconds
NFS / RWX	Pod B on another node reads what pod A wrote	Matched (NFS v4.2)
NFS / RWX	After restarting the Deployment so every pod was new	Matched. An append made after the restart was readable from another pod

With the corrected template, no manual step was needed (the Trident node pods restarted 0 times and both nodes could use iSCSI) and the persistence results in the table held. I also checked importing an existing volume: I wrote a file to a volume created with the FSx API, then imported it with an annotated PVC. It bound in about 10 seconds and a pod read the pre-existing file. ONTAP renamed the volume to trident_pvc_…, keeping its UUID and junction path. With reclaim policy Retain, deleting the PVC and the PV left the ONTAP volume in place (though Trident kept its own record, so backend deletion waited until tridentctl delete volume removed it). As a limit, for RWX I confirmed shared visibility, not consistency when several pods write the same file at once.

I also measured backend failure shapes. A backend with a wrong credential turned Failed (HTTP 401) and one pointed at an unreachable management LIF turned Failed (no route to host); both left the PVC Pending with a "no available backends" event. Changing a working backend's credential to a wrong value kept it running on the old credential until it was reloaded: existing pods' reads and writes did not stop, only new PVC provisioning did. This failure test locked the disposable SVM's vsadmin, which also stopped another backend using the same user, so I unlocked it as the file system administrator. Failure tests are safer with a separate SVM user, or run after the other checks.

During verification I found three defects in the CloudFormation template I used. I recorded them in an issue (#26) and fixed them in #27.

The default Kubernetes version was 1.31, which was in extended support in ap-northeast-1 on the day I tested. I removed the default so the user chooses a version in standard support.
The extra security group the template created attached to the control-plane network interfaces, not to the managed nodes (both nodes, one observation). I changed the source of the FSx-side ingress rules to the cluster security group, which the nodes carry.
The Amazon Linux 2023 EKS AMI did not include the iSCSI tools. I made a launch template install them, and with the changed template I confirmed that first-boot nodes had the tools and that Trident's node pods detected iSCSI without a restart (two nodes, one observation).

Teardown showed one more behavior. The FSx API (describe-volumes) reflects the volumes Trident creates on the ONTAP side with a delay. It did not list them at 23 seconds and at 11 minutes after creation, and listed them at 26 minutes. After I deleted the volumes on ONTAP, it still listed them at 41 minutes and no longer did at 44 minutes (one observation). While the FSx API showed a non-root volume, deleting the SVM was rejected. After teardown, check for leftovers with the ONTAP REST API as well as the FSx API. Deleting the PVCs removed the LUN and the volumes from ONTAP, and the file system's backup count was the same before and after (environments with automatic backups enabled are unchecked).

The results and the procedure are in the Container-Datastore record (section 4.3 of the storage-verification memo).

Path E (services that read through S3 Access Points): analytics and RAG

Path E is AWS services that read FSx for ONTAP data through the S3 API. It goes through the same S3 Access Points as candidate C, but the caller is not a Lambda I wrote; it is a service such as Athena or Bedrock Knowledge Bases. I tested analytics (E-1) in the Lakehouse repository and RAG (E-2) in the Agentic-RAG repository.

E-1 Analytics: Athena reads Parquet

I put a 10,000-row Parquet file (Snappy compression, about 150 KB, microsecond timestamps) on a disposable volume, and read a table registered in the Glue Data Catalog with Athena (engine v3) through an internet-origin access point (UNIX user).

Check	Result
PutObject, ListObjectsV2, and HeadObject on the access point	All succeeded. The listing showed storage class `FSX_ONTAP` and size 150,902 bytes
`SELECT COUNT(*)`	10,000 rows, matching the source row count
Conditional aggregation (`WHERE`, `GROUP BY`, `ORDER BY`)	The counts (1,667, 1,667, 1,666) and totals for the three regions matched the values computed from the source data exactly
Run time (small data, cache-resident)	`COUNT` took 821 ms in total and scanned 0 bytes. The aggregation took 942 ms in total and scanned 12,384 bytes

The run times are dominated by planning on small data. They are not production-scale values.

Two premises apply. Athena needed an internet-origin access point; a VPC-origin one did not work. The SVM I used was joined to Active Directory. The test succeeded, but I did not measure behavior when the domain controller is unreachable. Query results are written to a separate S3 bucket, not to FSx. I did not retrieve the ONTAP version. Other engines (Glue, EMR Serverless, Redshift Spectrum, and so on) are in other records in the Lakehouse repository, and I did not re-measure them here.

The record is the Lakehouse evidence-record.yaml.

E-2 RAG: narrowing search results by permission

I took documents on a disposable volume (on an SVM not joined to Active Directory), ingested them from an internet-origin access point into a Bedrock Knowledge Base (S3 Vectors as the vector store), and checked on a deployed environment a path in which a filter inside the web app matches SIDs that permit each document and narrows the search results. I measured four cases.

Case	Principal and search	Retrieved	Allowed	Denied	Method
1	An administrator (holding the SIDs of the administrator user, Domain Admins, and Everyone) searches "company sales"	6	6	0	SID match
2	A general user (the individual user's SID and Everyone only) runs the same query as case 1	6	0	6	SID match
3	A user with no registered SID runs the same query as case 1	6	0	6	Deny all (fail-closed)
4	The administrator searches "product overview" (over Everyone documents)	8	8	0	SID match
4	The general user runs the same query	8	3	5	SID match

In cases 2 and 3, the same query as case 1 retrieved 6 documents, and all 6 were denied. That shows the filter removed the documents after retrieval, not that the user's forbidden documents were never found. For the user with no registered SID, all 6 retrieved documents were denied, so the path was fail-closed (deny everything when there is no permission information).

The four cases above were measured on the inline path (filter inside the web app), which is the production default. That path has no cache and reads DynamoDB each time, so adding the confidential group's SID to a general user made the same query ALLOW immediately, and removing it returned it to DENY. As an alternative, with the filter moved to a Lambda (usePermissionFilterLambda=true) I measured the same four cases plus group add/remove, confirmed filterLog.source reads lambda, and got the same verdicts. The Lambda path has a five-minute cache: after removing the group, a cached entry returned the old verdict until it was cleared.

A CDK deploy plus one setup script get ingestion (11 documents) and the four cases through with no manual step, except one: creating the Cognito users, whose password does not meet the pool policy. Code defects found during the deploy are recorded in #228 and fixed. The set was 11 documents, and each case is one observation. I did not measure an SVM joined to Active Directory.

The record is section 3.3 of the Agentic-RAG evaluation.md.

How failures return per path, and what was confirmed for D and E

For A, B, C, and C-REST, which ran the same job, authentication, network, authorization, and the way failures return changed as follows.

Aspect	A (.NET / NFS)	B (Java / NFS)	C (Python / S3 Access Points)	C-REST (Python / ONTAP REST API)
Auth principal	OS (uid, AUTH_SYS)	OS (uid, AUTH_SYS)	AWS IAM	ONTAP credential (basic)
Authorization principal	OS (NFS permissions)	OS (NFS permissions)	AWS IAM + access-point policy	ONTAP
Network	NFS mount (in-VPC)	NFS mount (in-VPC)	public S3 endpoint (outside the VPC)	management LIF (in-VPC)
Auth failure returns	at the NFS mount stage	at the NFS mount stage	(credential is IAM-side)	HTTP 401 / ONTAP 6691623
Authorization deny returns	NFS permission error	NFS permission error	AccessDenied / HTTP 403	HTTP 403 / ONTAP code 6
Unreachable returns	mount failure	mount failure	(public endpoint)	connection timeout → unreachable
Invalid identifier returns	file not found	file not found	(per key)	HTTP 404 / ONTAP 6684674
TLS consideration	none (NFS)	none (NFS)	none (AWS terminates TLS; no client-side certificate)	ONTAP self-signed certificate → must classify as trust failure
Dependency on AD DC reachability	none (measured on an SVM not joined to AD)	none	none	none

The same "deny" returns as an OS permission error over NFS, as AccessDenied / HTTP 403 over S3 Access Points, and as HTTP 403 with an ONTAP error code over ONTAP REST API. When you implement failure classification, you have to fold in this per-path difference.

D and E are different workloads, and I did not measure their failure cases in the same form. I separate what I confirmed from what I did not into another table.

Path	What I confirmed	What I did not confirm
D (Trident PVs)	iSCSI and NFS volumes were provisioned in about 10 seconds, and the data remained after a pod restart and a reschedule to another node. Deleting the PVCs removed the LUN and the volumes. An existing volume was imported as a PV and its pre-existing data was readable. A backend with a wrong credential or an unreachable management LIF returned `Failed`, leaving the PVC `Pending`	Multi-AZ switchover. Performance. Same-file concurrent-write consistency
E-1 (Athena)	Reading and aggregating Parquet through S3 Access Points matched the source data. It succeeded on an SVM joined to AD	Behavior when the domain controller is unreachable. Production-scale data. Write-back
E-2 (Bedrock Knowledge Bases)	Documents outside the user's permissions were removed from the search results, and a user with no permission information was denied everything. A deny is not an error; it returns as removal from the results. Group add/remove was reflected in the results. The inline and Lambda paths gave the same verdicts. CDK and one script nearly completed (only a password step by hand)	An SVM joined to Active Directory. Response time. Natural expiry of the Lambda cache

How to choose a path

Not by winner, but by the client you use and where you want authorization to happen. I state trade-offs symmetrically, including for the option I lean toward.

Choice	When	Trade-off / consideration
NFS mount (A, B)	When a Linux server process needs POSIX file semantics (such as exclusive locking) directly	Auth is uid-based AUTH_SYS and depends on OS permissions. It assumes an in-VPC mount and is hard to use directly from serverless. File-lock behavior is only observable with two processes.
Your own code calling S3 Access Points (C)	When an S3 API client such as Lambda should be authorized by AWS IAM	Authorization is IAM plus the access-point policy; it holds no ONTAP credential. It suits jobs that fit the S3 API operation set (it does not express POSIX semantics such as exclusive locking). The network origin is fixed at creation. It uses the public S3 endpoint, so an in-VPC Lambda is not required.
ONTAP REST API (C-REST)	When you want fine control with an ONTAP credential and ONTAP-side authorization for management or file operations	The management LIF is reachable only from inside the VPC, so you need an in-VPC Lambda (mind cold starts and the ENI). You must handle the self-signed certificate (provide a trusted CA in production; disable verification only as an explicitly recorded diagnostic). File-system-level roles are fixed; custom roles are SVM-level only. Large files hit the 1 MB-per-call limit, so split them across calls.
Trident persistent volumes (D)	When stateful containers on EKS should use FSx for ONTAP volumes as PVs; also when you want to import existing NFS/SMB data as a PV	You need EC2 worker nodes (it does not run on Fargate), and iSCSI needs the tools and a multipath configuration on the nodes. Node preparation has no guaranteed start order, so restart the DaemonSet if the order breaks. The source of the FSx SG ingress is the cluster SG, not the node SG. Trident's objects are applied with kubectl, outside CloudFormation. The EKS cluster is billed. I confirmed both provisioning new volumes and importing an existing one.
An AWS service that takes S3 as input (E)	When a service such as Athena or Bedrock Knowledge Bases should use data on FSx for ONTAP	You are bound to the access-point settings (Athena needs an internet-origin one). Choose internet-origin when the service comes from outside the VPC, VPC-origin when it stays inside. Per-document permissions must be designed outside the service (for example, a filter in the app). It does not express all of S3 (for example, conditional writes are unavailable).

All of them are paths to the same FSx for ONTAP data. The basis for choosing is what the client is (a Linux process, an S3 API client, something holding an ONTAP credential, a Kubernetes container, or an AWS managed service) and where you want authorization to happen (the OS, IAM, ONTAP, or the app).

Reaching one volume through several paths at once (NFS, SMB, S3 Access Points) shares that volume's throughput. When you design for several paths at the same time, size the combined load against the volume's throughput.

This table does not decide which path is better. It lays out a fact: authentication, authorization, and the shape of failures swap per path. Which path fits your system, and whether you can keep operating it, is for someone who knows the app and the data to judge from the facts laid out here.

Summary and the path options

The result is the three findings from the start.

The same file-processing job ran as four implementations (.NET, Java, and Python) against disposable volumes on the same FSx for ONTAP file system, without moving the data. Changing the path swapped the authentication principal, the network boundary, the authorization principal, and how failures return.
Containers on EKS read the data in volumes Trident provisioned after a pod restart and after a reschedule to another node (SAN and NAS, once each). Athena read Parquet through S3 Access Points and the aggregation matched the source data (10,000 rows). In Bedrock Knowledge Bases searches, documents the user was not permitted to see were removed from the results (four cases, 11 documents).
The basis for choosing is the client you use and where you want authorization to happen.

All costs are estimates before billing, for disposable environments that lived minutes to a few hours, not production always-on costs. Each A through C-REST measurement was under $1 in an approved disposable environment. Path D's stack lived about 36 to 70 minutes; from the unit prices (ap-northeast-1, Pricing API, 2026-10) of an EKS cluster at $0.10/hour plus two nodes at $0.124/hour each, that is about $0.2–0.4, excluding NAT data processing. Path E-2 is a WAF Web ACL ($5/month, $1/month per rule, prorated) plus small usage-based charges for CloudFront, Lambda, and Bedrock, on the order of under $3 for a disposable few hours. For E-1 and E-2 I did not compute a cost for the actual time the resources lived, and I did not check the billed amounts.

Each case was measured once, and I said on the spot where I got through by hand and where I did not retrieve a value. Path E-2 is nearly completed by CDK and one script (only the password step is by hand), and the SAN and NAS data persistence on path D was observed on the corrected template with no manual step.

Repository:
https://github.com/Yoshiki0705/FSx-for-ONTAP-App-Modernization-Patterns

The previous article is "Keeping the Data Where It Is, Modernizing Only the App From .NET Framework to Lambda — What Changes and What Does Not (A 1-Environment Test)."

To try this in your own environment, start with the premises in reproduce.md.

Modernization does not have to replace everything at once. With the data in one place, you can add access paths a little at a time — first an EC2 instance over NFS, then Lambda, and later EKS or analytics and AI. You choose each path to fit the runtimes your team can operate and where you want authorization to happen. Choosing FSx for ONTAP as the destination lets you take this "keep the data in place, add access paths in stages" approach in combination with the various AWS services. This article measured those six paths actually reaching the same data, and how authentication, authorization, and the shape of failures change per path.

Whether you are about to modernize on AWS or already use FSx for ONTAP, I hope this gives you something to decide with as you go step by step.

See you next time.

Bye now!!

Yoshiki Fujiwara

Resources

Candidate A runtime verification issue (#42)
Candidate B spec / runtime issue (#46)
Candidate C spec / runtime issue (#47)
Candidate C-REST spec / runtime issue (#48)
Runtime verification tracking (#38). A, B, C, and C-REST are complete; their deliverables are in the issues and records above. #38 now tracks the remaining deep-dives such as performance and scale. The letters A to C in #38 differ from this article's B, C, and C-REST; the mapping is in #45
Path D record (Container-Datastore, section 4.3 of the storage-verification memo), and the issue for the template defects (#26)
Path E-1 record (Lakehouse, evidence-record.yaml)
Path E-2 record (Agentic-RAG, section 3.3 of evaluation.md)
FSx for ONTAP S3 Access Points user guide
Amazon EKS User Guide: use FSx for NetApp ONTAP
Microsoft Learn: Upgrade .NET apps overview
ONTAP REST API reference

Keeping the Data Where It Is, Modernizing Only the App From .NET Framework to Lambda — What Changes and What Does Not (A 1-Environment Test)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Thu, 22 Oct 2026 07:48:47 +0000

Keeping the data you already have, modernizing only the app — what changes across SMB, NFS, and S3 Access Points

Business apps written for .NET Framework are still running in many Windows environments. .NET Framework runs only on Windows, and its feature set is effectively frozen. Current .NET (modern .NET) is cross-platform: it runs on Linux, in containers, and on serverless such as AWS Lambda. Moving an app to modern .NET frees it from the OS dependency and the support-window constraint, and lets it run without an always-on server, paying only for what it uses. Microsoft itself frames the move from .NET Framework to modern .NET as a good opportunity to containerize and go cloud-native (Microsoft Learn: Upgrade .NET apps overview).

But work usually stalls on the same question: the app can move, but what about the data the app has been reading and writing? Do I rebuild that data or relocate it, and what happens to its permissions and protocols?

No new announcement from AWS or any other vendor prompted this work. I have been working through modernization on AWS as a series of themes — migration, multiprotocol access, performance, S3 Access Points, a serverless file portal, a data lake, observability — and the step I had not yet taken was migrating the application code itself. Then AWS's AI-Driven Modernization published its AI Modernization Flow (a workflow that has an AI agent carry out a disciplined migration), and I used it to take exactly that step.

This post is a measurement record of what changed and what did not when I moved only the app, from .NET Framework 4.8 to current .NET and AWS Lambda in four stages, with the data left on a single volume. Across the stages I added paths for opening that same data: SMB from a Windows Amazon EC2 instance, NFS and cifs (a way to use SMB from Linux) from a Linux Amazon EC2 instance, and S3 Access Points (a way to open the volume through the S3 API) from AWS Lambda. The data lives on a single volume of Amazon FSx for NetApp ONTAP (FSx for ONTAP from here on; NetApp's ONTAP offered as a managed AWS storage service), and I did not change that volume from start to finish.

First, the caveats. I measured one app in one environment (ONTAP 9.19.1P3, Single-AZ, ap-northeast-1), and each measured value in this post comes from one to a few runs. A value I saw only once is labeled "one observation". No value is a guarantee that production will behave the same way. Costs are the values AWS Cost Explorer returned before billing is final (Estimated), not invoice amounts. When a later chapter says "as stated up front", it refers to this paragraph: one app, one environment, one to a few runs, and Estimated costs.

The one app I measured is DocIntake, a sample app I wrote for this verification. It is not a customer app and not a production app.

Role: the Worker (the app proper, one of the two console apps) reads the documents in an inbox folder on the volume (seed/inbox/), processes them, and writes a report of the results (reports) and an index, index.json. It is a batch-style app
Structure: two console apps (the Worker, and the Probe that measures the five behaviors). It has no screen (UI), no Web layer, no database, and no authentication
How it reaches storage: all file reads and writes go through IFileStore, which has 5 methods. In the .NET Framework source, a single Store key in appSettings selects the storage implementation (SMB in stage 0, the first of the four stages). As the stages advanced, I added NFS and S3 Access Points implementations
Why it was built this way, and its size: so that the difference between paths can be observed behavior by behavior, I placed one habit that does not show up on Windows + SMB but can show up on another path in the source code for each of the five behaviors (the five behaviors are listed in the Probe row under "Terms used in this post"). It is .NET Framework 4.8, C#, 0 NuGet dependencies, 10 .cs files, and 935 lines

Because this is the result of a single self-written console app, it does not necessarily apply to real business apps, or to apps that have a screen.

You are probably wondering about these three points:

Can I move only the app to Lambda without moving the data?
When I change the path that opens the data from SMB to NFS to S3 Access Points, what changes in how the same files behave?
Which should I use to convert the code: atx (AWS's code analysis and conversion tool) or AIMF (the AI Modernization Flow mentioned above)?

Here are the three findings, answering the three questions above in order:

Question 1 (can I move only the app to Lambda without moving the data?): yes. I converted the code of DocIntake and ran it on Lambda, while the data stayed in one FSx for ONTAP volume, appdata. In this post, "move" means converting the code and changing where the code runs and which path it uses to open the volume. It does not include copying or moving the data.
- What did not change (where the data lives): the data stayed in the volume appdata from stage 0 through stage 3. I did not copy, move, or recreate that volume. The UUID of appdata, its security style (the method it uses to decide permissions; ntfs to the end, which decides permissions with NTFS ACLs), and the sha256 of the seed (the 6 files I placed at the start of the verification) did not change; I recorded and compared them at each stage boundary (these records are from the second of the two builds of the verification environment). Those 3 items are what I confirmed did not change. The output that the app writes did accumulate on the volume, and in stage 1 I changed the export policy and name mapping settings for NFS
- What did change (the code, and how it runs): three things changed: the app's code, where the code runs, and the path used to open the volume. I converted the code from .NET Framework 4.8 to .NET 10 (with atx, a code analysis and conversion tool, and AIMF, a kiro-cli migration workflow; the third finding covers how they differ). I changed where the code runs from a Windows EC2 instance to a Linux EC2 instance to Lambda. For the path, I added NFS and cifs to SMB, and finally added S3 Access Points
- What I checked: in stage 3, the reports and index.json that the app itself (the Worker) wrote on Lambda through S3 Access Points were visible from the Linux client that opens the same volume over NFS (one observation)
- The code that ran at each stage: the converted code ran in stage 2 and stage 3. Stage 0 is the starting point with the unconverted code, and stage 1 adds a path without changing the code, so neither has converted code. The table below shows each stage

Stage	Code that ran	Where it ran	Path to the volume	Evidence that it ran
0	The unconverted Probe (.NET Framework 4.8). The counterpart on the Linux side is a Python version of the Probe (not DocIntake's code, but a separate script that measures the same 5 behaviors). The unconverted Worker ran once, later, as the stage 0 baseline	A Windows EC2 instance and a Linux EC2 instance	SMB from Windows; cifs from Linux	The Probe records show every side and every pair as Success. The Worker exited with 0 and wrote reports and `index.json`
1	The same Probe as in stage 0 (no record of a code change). A Python version of the Probe on the Linux side. No record that the Worker ran	A Windows EC2 instance and a Linux EC2 instance	SMB from Windows; cifs and NFS from Linux	The Probe records show every side and every pair as Success
2	The converted Worker and Probe (.NET 10). The counterpart on the Windows side is the unconverted Probe	A Linux EC2 instance	NFS for the Worker; NFS and cifs for the Probe	The Worker exited with 0 and wrote reports and `index.json`. Every Probe record is measured
3	The converted Worker and Probe (.NET 10; two modes of one Lambda function)	Lambda (inside the VPC)	S3 Access Points	The Worker exited with 0, and the output it wrote was visible from the NFS side (names and sizes only; I did not read the contents; one observation)

Question 2 (when I change the path that opens the data, what changes in how the same files behave?): some of how the same files behave changed with the path that opens them.
- Case sensitivity: for each path, I measured whether a reference that points at an existing file with different letter case resolves to that file on the same volume, appdata. Both SMB paths (Windows, and a Linux cifs mount, which uses SMB from Linux) were case-insensitive (the reference resolved), while NFS and S3 Access Points were case-sensitive (it did not)
- Write visibility (the time until a file written by one client becomes visible to the other client): I measured both directions between Windows and Linux. When Windows wrote and Linux read, it took under 100 ms (35 ms when Linux read over NFS and 66 ms over cifs; the reading client tries every 100 ms to read the content the writer wrote, so these two values cannot be told apart). In the opposite direction, when Linux wrote and Windows read, it took about 8 seconds (7,968 to 8,342 ms: 7,968 ms when cifs wrote and 8,342 ms when NFS wrote). The asymmetry, fast when Windows writes and slow when Linux writes, had the same shape in three measurements (record identifiers s0, s1 and s2m). The values 35, 66, 7,968 and 8,342 ms above come from one of them, the single s2m measurement
- Other behaviors: the handling of names that contain \, whether an exclusive lock is possible (a way of opening a file so that no one else can open it), and the writability check (whether writability is judged by a pre-evaluation of the ACL or by actually writing) also differ by path (see "Behavior by access path")
Question 3 (which should I use, atx or AIMF?): atx and AIMF covered different scopes. I do not rank them.
- What I compared: atx and AIMF are two different tools. I set side by side what each produced from processing the same app, DocIntake, once each
- atx (AWS Transform custom, AWS's code analysis and conversion tool): analyzed the code and converted it to .NET 10, and the net10.0 build of the converted code (on macOS) succeeded. But the five legacy-code habits that I had placed in the sample app, one for each of its five behaviors, and that assume Windows and SMB, remained in the converted code. The NFS and S3 Access Points implementations were not in atx's output
- AIMF (a workflow run with kiro-cli): took the atx analysis as input, produced ADRs (architecture decision records) and a migration ledger (a list of where each legacy-code habit was handled in the new code), and went on to implement NFS and S3 Access Points
- Limits of the comparison: one app, processed once with each tool, and the cost units differ (agent minutes for atx, Kiro credits for AIMF), so I did not compare which took less time or cost less

What you should take back to your own environment: each time you add a path that opens the data (SMB, NFS, S3 Access Points, and so on), measure the five behaviors again on the added path, together with how root is treated. The five behaviors are case sensitivity, handling of names that contain \, exclusive locking, the writability check, and write visibility (listed in the Probe row under "Terms used in this post"). For root, an example is that on NFS the root user could not enter /mnt/appdata (the cause is an inference from the export policy setting; see the stages 0 and 1 chapter and the stage 3 chapter).

On cost: the total I could identify for the AWS resources used in this verification is $35.91. The breakdown is $27.49 that I could narrow down in Cost Explorer by tag; $4.80 for AWS Directory Service (Active Directory) in the second of the two builds of the verification environment (the first on 2026-10-07, the second on 2026-10-11 to 2026-10-12), which I could not narrow down and estimated by subtracting the steady usage of days without the verification from the usage of the verification days; and $3.62 for AWS Transform (atx) usage. All three are Estimated, not invoice amounts. The AWS Transform usage carries a Credit of the same amount, −$3.62, and the $35.91 does not deduct that Credit.

When this post fits and when it does not

This post fits SAs, application developers, storage administrators, and Partner/SI engineers who are considering moving a .NET Framework app, built around a Windows shared folder (SMB), to Linux or Lambda without moving its data. It reports one such app, opened over SMB, NFS, and S3 Access Points against one FSx for ONTAP volume (ntfs, Single-AZ), so it is useful for judging cases with the same conditions (an SMB-based .NET Framework app, an ntfs volume, Single-AZ).

This post does not fit if you need any of the following, because I either did not measure it or measured it under different conditions:

Performance (this verification put it out of scope, so I did not measure it; "What this post covers and what it does not" points to results that other projects measured)
Behavior on a volume whose security style is unix or mixed (I measured only ntfs)
Migrating the screens or Web layer of an app that has a UI (the app I measured is a console app with no screen and no Web layer)
Behavior on Multi-AZ (I built only Single-AZ)
A guarantee for production (every value is from one to a few runs in one environment)

Reading by purpose

You only want the conclusions: the three findings in this chapter are enough.
You want to know whether changing the path causes trouble: "Behavior by access path" and "Summary and choosing an access path"
You are choosing between atx and AIMF: "Choosing between atx and AIMF"
You want to know how to hand out a development environment without copying the data: the FlexClone subsection of "Staged design and the single-volume constraint" (it reports only what I read in documentation; I did not create a FlexClone in this verification)
You want to know the cost: "Cost estimates and observations"
You want the conditions and limits of the measurement, or want to reproduce it: "Test environment and method" and "What has not been measured yet"
You want to read in order: the staged design → stages 0 and 1 → stage 2 → stage 3 → behavior by access path

What this post covers and what it does not

This post reports five behaviors of one app, measured in one environment. It is not an exhaustive comparison.

What this post covers:

It covers what changed and what stayed the same on the volume across four stages (SMB, adding NFS, new .NET, Lambda). The data did not move and only the path changed, so differences can be traced to the path
It covers how I checked the measurements, and failures written up with numbers. Measuring without a positive control (a control that looks for something that must be found) invalidated the case-sensitivity values in 6 measurement runs. The same kind of error could occur in other measurements
It covers how to choose between atx and AIMF. I have one pair of results where both processed the same app, so the difference in scope can be compared concretely

What this post does not cover:

It does not cover volumes whose security style is unix or mixed. I decided to keep the security style at ntfs, and I did not create volumes with the other styles
It does not cover the container migration branch. The documents in this verification's repository hold only an explanation of the branch point and a link to another repository, and I did not go down the branch. The other repository that covers the branch (Container-Datastore Patterns, Japanese) is also, so far, at literature review and static validation (cfn-lint), and lists deployment to a real environment as the next stage
It does not cover apps that have a UI (a Web layer such as ASP.NET, or a desktop app), or tests that operate a screen. DocIntake, the sample app in this verification, is a console app with no screen and no Web layer, and I ran no screen-operation tests
It does not cover creating a FlexClone in a real environment, or the five behaviors on the clones. I did not create a single FlexClone in this verification, and the creation script went only as far as a dry run. The "Staged design and the single-volume constraint" chapter reports what I read in the documentation about FlexClone (documented)
It does not cover other ONTAP versions or Multi-AZ. The only configuration I built was Single-AZ on ONTAP 9.19.1P3
It does not cover reconciliation with an invoice. I have not checked an invoice, and the costs in this post are only the Cost Explorer values before billing is final (Estimated)
It does not cover performance. This verification measured how behavior differs per path (the five behaviors). I did not run measurements with performance conditions decided in advance (large files, many files, long runs), and ADR-2, AIMF's record of design decisions, also put performance out of scope. Another project measured the performance of the S3 Access Points path in two cases. Both are carried-over (values measured in another verification of mine), and neither is a value measured in this post's environment. Neither document records the ONTAP version
- From an Amazon EC2 instance (c5n.9xlarge) using Python, the median latency of a PUT to S3 Access Points at concurrency 1 was 363.7 ms for 1 MiB and 331.5 ms for 8 MiB. A fixed cost per request (about 330 ms) dominated over object size (2026-09-01, ap-northeast-1, Single-AZ, 128 MBps, NetworkOrigin VPC, UNIX root identity; measurement of the S3 Access Points path performance, Japanese)
- From a Lambda function outside the VPC (1,769 MB), the median latency of a 1 MB GetObject against S3 Access Points at concurrency 10 was 73.2 ms (2026-05-25, ap-northeast-1, Single-AZ, 128 MBps, NetworkOrigin Internet; S3 Access Points benchmark results, Japanese). That benchmark results document also lists the path this post uses, from a Lambda function in the VPC to an S3 Access Point with VPC origin, as not yet measured

Terms used in this post

Term	One-line meaning
ntfs / security style	The security style is the method a volume uses to decide permissions, and ntfs decides with NTFS ACLs. This post uses ntfs to the end
SMB	The protocol Windows uses for file sharing
NFS	The protocol Linux uses for file sharing. This post uses NFS 4.1 with `sec=sys` (a simple authentication where the client claims its uid)
cifs mount	A way to mount an SMB share from Linux. The path is SMB and the client is Linux
S3 Access Points	An endpoint that reads and writes an FSx for ONTAP volume through the S3 API. An S3 API client such as Lambda reaches the same volume without copying the data
NetworkOrigin	The setting for where an S3 Access Point can be reached from (a VPC in this post). You can specify it only when the access point is created, and changing it means recreating the access point
atx (AWS Transform custom)	AWS's code analysis and conversion tool. I use two of its transformations: analysis (comprehensive-codebase-analysis) and conversion (dotnet-modernization)
AIMF	Short for AI Modernization Flow. A workflow that runs through Phase 0a to 4 with kiro-cli and leaves ADRs and a ledger
ADR / migration ledger	An ADR is a record of a design decision and its reason. The ledger (AP-1 to 14) lists where each legacy-code habit was handled in the new code
Probe	A console app that measures five behaviors (case sensitivity, names containing `\`, exclusive locking, the writability check, write visibility) and records them as JSON. An exclusive lock is a way of opening a file so that no one else can open it
Positive control	A control that looks for something that must be found, through the same path, to confirm the measurement works
write-visibility	The label the Probe records use for write visibility: the time until a file written by one client becomes visible to the other
8.3 short name	A short Windows-compatible alias given to a long file name (for example `PROBES~1.TXT`)
name mapping	An ONTAP setting that maps a Windows user to a UNIX user
FlexClone	An ONTAP feature that creates a writable point-in-time copy, as a separate volume, from a volume's Snapshot (its state at one moment). I did not create one in this post

The modernization work so far, and the new step this time

This post continues an ongoing body of work on modernizing on AWS. So far I have worked through block storage migration with AWS Transform, multiprotocol access on FSx for ONTAP, hands-on throughput and block-performance verification, using S3 Access Points, a serverless file portal, data lake integration, and observability — each published as a GitHub repository and a blog post. This time the new, concrete step was migrating the application code itself.

What prompted this work is AWS's "AI-Driven Modernization (AIDM)" and its AI Modernization Flow (AIMF). I ran this migration with AIMF. AIMF is a workflow that has an AI agent carry out a disciplined migration of an app on an older technology stack; designing and building the cloud environment is out of its scope. That boundary is why the two fit together: AIMF handles the app side, and my earlier FSx for ONTAP work covers the data and storage side.

Differences from the previous article

The previous article is "AWS Transform Now Supports Block Storage Migration to FSx for ONTAP — Benefits and Pitfalls from a Hands-On EC2 Test". This post differs in two ways:

Scope: the previous article covered getting data onto FSx for ONTAP (AWS Transform migrating block storage). This post leaves the data where it is and moves only the app's code
What was measured: the previous article measured a migration job. This post measures how five behaviors differ when the same volume is opened over SMB, NFS, and S3 Access Points

Staged design and the single-volume constraint

I fixed the data on one volume so that failures in the app migration and failures in the data migration would not get mixed together. With the volume fixed, an observed difference can be read as a difference between paths. That choice to fix the volume is a design decision, not a measurement result. Fixing the volume does not mean there can be no working copies. FlexClone, a way to add a working volume while keeping appdata fixed, is described in the second half of this section.

Stage	Runtime	Path	Where the code comes from	Volume
0	Windows Amazon EC2 instance, .NET Framework 4.8 (a Linux Amazon EC2 instance opens SMB over cifs as the Probe's second measuring client)	SMB	The original app	appdata (ntfs)
1	The stage 0 setup; the Linux Amazon EC2 instance also opens NFS 4.1	NFS 4.1 added	The original app, plus a Python version of the Probe on the Linux side	appdata (still ntfs)
2	Linux Amazon EC2 instance, .NET 10	NFS 4.1 and cifs	AIMF (main line), atx (comparison)	appdata (still ntfs)
3	AWS Lambda (in the VPC), .NET 10	S3 Access Points	From AIMF; from atx (first run; run id s3)	appdata (still ntfs)

Source of this table: the stage table in docs/en/reproduce.md (commit-pinned; I reorganized the columns)

Figure 1: The flow from stage 0 to stage 3. The volume in the bottom row stays the same. The container migration is a branch and was not measured. (dark theme version)

The point of the figure: the volume in the bottom row stays one volume as the stages advance.

AIMF is the main line of stage 2. AIMF's first step (Phase 0a) imported the atx analysis results. I ran the atx dotnet-modernization transformation separately, for comparison.

Handing out working volumes with FlexClone (not done in this verification)

For app development and testing, you often need a separate place to read and write production-like data (in this post, the contents of appdata) without modifying the data itself. ONTAP FlexClone is one way to add that place without copying all of the data. FlexClone can be used alongside the design that fixes appdata as a single volume.

A FlexClone is a feature that creates a writable point-in-time copy by referencing the metadata of a Snapshot (the state of a volume at one moment). I call the copy a clone, and I call the volume it is made from the source volume (the NetApp documentation calls it the parent volume; in this post the source volume is appdata). The clone shares data blocks with the source volume, and until you write to the clone it uses capacity only for metadata (ONTAP: FlexClone volumes, files, and LUNs concepts). The free space a newly created clone uses is described as about 0.5% of the clone's nominal size. New data written to either the source volume or the clone is not shared between the two volumes, and the clone's usage grows by the amount written to the clone (ONTAP: space used by a FlexClone volume). The NetApp documentation contrasts this with traditional copies, which can take minutes to hours, and says that because FlexClone shares blocks, a copy of a large dataset can be made in a short time (this post gives no creation-time number).

How this project used it

Designed and implemented: the verification script integration-clone.sh creates and deletes, over the ONTAP REST API, a FlexClone named appdata_it_<n> from a Snapshot of appdata. The script never creates, changes, or deletes appdata itself. Its purpose is to create and remove the clones that the AIMF Phase 3 integration test (the integration gate) uses
Dry run only: dryrun_shell_tests.sh checks the requests that were assembled and the name validation, without calling ONTAP. No clone is created
A check at teardown only: in steps 4 and 5 of the environment teardown, I confirmed that no FlexClone and no it_* Snapshot remained and that the recovery queue was empty (docs/en/environment.md). This confirms absence; it is not evidence that one was created

So I have confirmed the design and the command assembly, but I have not created a clone in a real environment or measured behavior on one.

Use cases the documentation describes

Use case	What the documentation says (paraphrased)	Source	Tier
Copies for development and testing	Described as temporary, writable copies for trying an app against a production dataset	ONTAP concepts, ONTAP: clones for testing	documented
Several copies, per user or per server	Several copies of the same dataset (the documented example is virtual desktop deployment). An example that makes several clones from the same Snapshot for several development and test servers. A use that gives additional users access to the data without giving them access to production data. I found no document that says "one per developer"	The two in the first row, AWS: SAP HANA replicate and clone	documented (per-developer use not stated)
Rehearsal before a change	An example of trying a database operation on a clone before running it against production. Checks before an application or Kubernetes cluster upgrade. Validation of a system upgrade or migration	AWS: refreshing a development environment, AWS: applications on Amazon EKS, the SAP HANA post in the second row	documented
Checking recovery and incident response	A use for disaster recovery (DR) testing. Mounting a clone of a Snapshot taken before data corruption on another server to attempt a logical repair	The SAP HANA post in the second row	documented
Provisioning test data for CI	The documents I could fetch contain no statement that names use in CI. This use is only my extension of "several copies" above	None	hypothesis
Analytics and reporting on a point-in-time copy	The documents I could fetch contain no statement that names analytics or reporting	None	hypothesis

The examples in the sources are Oracle Database (Amazon EC2, NFS), MySQL (Amazon EKS, iSCSI), and SAP HANA. The documents I fetched have no example of a Windows app opening an SMB NTFS share.

Relationship to the single fixed volume

Through stages 0 to 3, appdata stays one volume with the same UUID. A FlexClone is a separate volume from appdata, and you can specify a separate junction path for it (the path that attaches a volume to the SVM namespace) (ONTAP CLI: volume clone create). So a clone does not contradict the fixed design. It adds a working volume, per developer or per stage, next to the fixed appdata. On the other hand, every clone adds a new junction path. The five behaviors this post measured (case sensitivity, names that contain \, exclusive locking, the writability check, and write visibility) and the treatment of root are properties of the access path. So you measure them again on each path that opens a clone, the same as the takeaway "measure again each time you add a path".

Aspect	One fixed volume (this verification, verified)	FlexClone working volume (documented, not done)
What you get	You can attribute differences to the path. The UUID, the security style, and the seed sha256 did not change across stages 0 to 3	A write target per user or per purpose that does not modify appdata
Behavior per path	The five behaviors are measured (one environment)	Needs to be measured again on each path that opens the clone. Opening a clone of an NTFS volume over SMB, NFS, and S3 Access Points is not measured. I found no statement in the fetched documents on how NTFS ACLs or SMB shares carry over to a clone
Capacity	No additional capacity, because no volume besides appdata is added	Metadata right after creation, then growth by the amount written. After a split (detaching the clone from its source volume to make an independent, normal volume), both the source volume and the clone need the full capacity set by the volume's guarantee (ONTAP: split)
Dependency on the source volume	None	You cannot delete the source volume while the clone exists (ONTAP: creating a FlexClone). A split cannot be undone, and no new Snapshot of that clone can be created during the split
What remains after deletion	Not applicable	In a separate verification of mine (2026-08-26, ONTAP 9.18.1P3D1, ap-northeast-1, a 1 GiB volume with the UNIX security style, one observation), a clone deleted in ONTAP stayed in the recovery queue, the deletion of the source volume it was made from failed, and a purge cleared it (measurement of ONTAP features over S3 Access Points, Japanese, carried-over)
Number of volumes	1	The volume limit per FSx for ONTAP file system is 500 (first-generation, and second-generation with one HA pair) or 1,000 (second-generation with two or more HA pairs), and 491 and 975 respectively when you use S3 Access Points (975 is for two HA pairs and decreases as HA pairs are added) (AWS: quotas). That page does not say whether a clone counts toward the number
How it appears on the AWS side	Not applicable	A change made on the ONTAP side can take up to several minutes to appear in the AWS console, CLI, and API (AWS: managing with ONTAP applications). In the separate verification described in the "What remains after deletion" row, a clone took 1,177 seconds (observed every 20 seconds, one run) to appear in the AWS-side list, and in another run it had not appeared at 1,258 seconds (carried-over)
Where you create it	Not applicable	The ONTAP CLI and REST API (ONTAP REST: creating a volume). The AWS API's ONTAP volume-creation settings list no field for specifying a clone (AWS API: CreateOntapVolumeConfiguration)
Relationship to backups	Not applicable	In an AWS Transform VMware migration example (a different feature from the atx in this post), locked Snapshots that FSx for ONTAP backups create can block a split (AWS: VMware migration). A clone that is in a data protection relationship cannot be split

You can also clone a FlexClone (ONTAP: efficient copies). In the table above, I did not confirm the behavior of a clone of an NTFS volume or the relationship to backups.

How to choose

Where it fits: you need working copies, per user or per purpose, that you can write to while keeping the contents of appdata, and the time and capacity of making full copies is a burden
Where to consider other means: if you need only a few working copies for a short time, an ordinary copy is enough. If you keep a clone for a long time and writes grow, the clone's usage grows and the period during which the source volume cannot be deleted gets longer. Also consider other means when the volume count is near the limit, or when you cannot build the ONTAP CLI and REST API into your operations. Restoring from a backup (AWS: restoring backups) is also a way to create a separate volume. The documents I fetched do not say whether a restore shares blocks

What you can decide: whether to take the approach of adding working volumes with FlexClone while keeping appdata fixed as one volume, and, if you do, whether you can accept the operational costs in the table above (deleting the source volume, splitting, the volume count, and the delay before changes appear on the AWS side).

The value of S3 Access Points and its four prerequisites

The value of using S3 Access Points is that an S3 API client such as Lambda reaches the same volume without copying the data. In this environment, the 5 reports and index.json written by the Lambda Worker were visible from the NFS side (stage 3, one observation). There are four prerequisites:

#	Prerequisite	Content	Evidence tier
1	Fixed NetworkOrigin	Specified only at creation. To change it, delete the access point and create it again	documented (AWS documentation; the source is in the stage 3 chapter)
2	The user (identity) that handles S3 Access Points requests on the volume	In this environment, the WINDOWS user `appsvc`	verified (stage 3)
3	Reachability to an AD domain controller (DC)	In this environment I used the access point with the DC reachable. Whether every data operation on an AD-joined SVM requires DC reachability, I have not confirmed. A successful HeadBucket does not show that file operations work	open (I could not find a record a reader can reproduce, or a public primary source; details are in "What has not been measured yet")
4	No exclusive locking and no event notifications	file-locking is skipped in the Probe run through S3 Access Points; the trigger that starts the Worker is polling from EventBridge Scheduler	The file-locking skip is verified. The unavailability of event notifications is documented (my reading of the AWS API support table, which lists GetBucketNotificationConfiguration as not supported; see the stage 3 chapter)

What you can decide: whether to adopt the approach of changing only the paths (SMB, then NFS, then S3 Access Points) without moving the data, and whether you can confirm the four prerequisites above in your own environment before you use S3 Access Points.

Test environment and method

I measured three paths (Windows, Linux, Lambda) against the same volume, appdata, in the same SVM. The setup is below (as in the caveats stated at the start).

Configuration

Unless noted otherwise, the tier is verified (versions are values read from ONTAP REST and similar).

Item	Value
Region and configuration	ap-northeast-1, SINGLE_AZ_1, 1 HA pair, 1,024 GiB SSD, 128 MBps
ONTAP	9.19.1P3 (the first build was 9.19.1P2; I did not look into why it was updated)
SVM and volume	1 SVM, 1 volume appdata, security style ntfs
Active Directory	AWS Managed Microsoft AD (Standard, 2 DCs). The directory needs 2 AZs, so I added a second subnet to the VPC
Windows client	Amazon EC2 t3.large. I did not record the Windows Server version
Linux client	Amazon EC2 t3.medium, Amazon Linux 2023 x86_64. NFS 4.1 `sec=sys` (uid 10001), cifs `vers=3.0`. I did not record the kernel or cifs-utils versions
Around the VPC	6 interface endpoints, an S3 gateway endpoint, 4 Secrets Manager secrets, an Amazon S3 artifacts bucket
Tool versions	AIMF v0.11.0 (pinned tag), kiro-cli 2.29.1, atx 3.18.0, .NET SDK 10.0.400 (runtime 10.0.11; self-contained on Linux), AWSSDK.S3 4.0.104.2 (pinned)
The app, DocIntake	.NET Framework 4.8, C#, 0 NuGet dependencies, 10 .cs files, 935 lines (sum of `wc -l`; atx reported 15 files and 1,089 lines). Two console apps (a Worker that processes files and writes reports and `index.json`, and the Probe) and one class library. `IFileStore` has 5 methods

Figure 2: The test environment. Arrow labels name the path or purpose (SMB, NFS 4.1, S3 API, fetch credentials, domain join). (dark theme version)

The point of the figure: all three paths reach the same volume in the same SVM.

Method

Probe: a console app that measures five behaviors and records them as JSON
Positive control for case-sensitivity: in addition to a lookup that refers to an existing file with different letter case, a lookup that refers to it exactly as it exists, to confirm the measurement works
Standalone and pair: standalone runs the Probe on one machine; a pair links two machines by a shared sync ID (locking and visibility need two machines)
Stage boundaries b0 to b3 and check-invariant: records of the volume's state taken after each stage (b0 after stage 0, b1 to b3 after stages 1 to 3). Each record holds the volume UUID, security style, sha256 of the 6 seed files, export policy, snapshots, and snapshot locking. check-invariant compares a later record with b0 and checks three items: the UUID, the seed sha256, and the top-level paths

Evidence labels and run ids

The labels in the tier column are as follows. verified: a value I measured in a named environment (with count and date). field-observation: a value I saw once and did not reproduce. documented: a value stated in a document whose source I read in full. carried-over: a value measured in another verification of mine that I did not measure this time. hypothesis: a guess. open: an item for which I could not find a record a reader can reproduce or a public primary source. not measured: an item I did not measure.

The main runs are below (a run id identifies one measurement).

run id	Date	What it measured
s0-20261011T065443Z	2026-10-11	All 5 behaviors in stage 0
s1-20261011T075058Z	2026-10-11	All 5 behaviors in stage 1 (after adding NFS)
s4c-20261012T034403Z	2026-10-12	case-sensitivity (SMB, cifs, NFS)
s3b-20261012T022900Z	2026-10-12	case-sensitivity (S3 Access Points)
s0w-20261012T102314Z	2026-10-12	Output of the old Windows Worker
s2m-20261012T090934Z	2026-10-12	8 pairs (4 lock, 4 visibility) of the Probe converted to .NET 10 in stage 2 (Linux side; the Windows side was the earlier build)
s2n-20261012T104314Z	2026-10-12	A run that re-took the shape of the writability check (acl-evaluation) record
s3m-20261012T104800Z	2026-10-12	The Probe and Worker converted to .NET 10, on Lambda

The source of this table is the run id naming in docs/en/reproduce.md, the same document as the stage table (I reorganized the columns).

The runs whose case-sensitivity values I invalidated are s0, s1, s2, s2f, s2g, and the first stage 3 run, s3. They had no case-sensitivity positive control, so I use none of those case-sensitivity values in any conclusion. The values of the other behaviors in those runs are valid. Source: record validity in docs/en/reproduce.md

The measurement window is 2026-10-11 to 2026-10-12. Times in the tables mix UTC and JST, and each row says which. For external unit prices and availability, I give the date I read them.

What you can decide: if your environment differs from the configuration table (versions, Multi-AZ, security style), that is a range this post did not measure. Look up the matching item in "When this post fits and when it does not" and "What this post covers and what it does not", and treat it as an item to measure again in your own environment.

Stages 0 and 1: unchanged UUID and seed, changed root handling

Adding NFS to the SMB volume did not change the volume UUID, the security style, or the sha256 of the seed. What changed was how root is treated over NFS and how to read the mode bits in ls -l (2026-10-11, one observation each).

Unchanged	Changed
The seed sha256, the volume UUID, and the security style (still ntfs before and after)	As root over NFS, `cd /mnt/appdata` gave Permission denied, while uid 10001 could read and write
`check-invariant` passed for b0 against b1 (UUID, seed, and top-level paths matched)	Over NFS, `ls -la` showed the top level as `d---------` (owned by root), yet uid 10001 could read and write. Files created over SMB showed as `-rwx------`, owner `nobody`

Stage 0 (SMB only): I created the AD users appsvc and appreader and placed the 6 seed files over SMB. The sha256 values read back matched the originals, and the Probe finished with the 2 standalone runs (Windows and the Linux cifs mount) and all 4 pairs (2 lock, 2 visibility) at Success
Stage 1: I changed the export policy to the NFS 4 family, sec=sys, and superuser none (a setting that replaces NFS root with an anonymous user). I created the UNIX users appsvc (10001) and appreader (10002) and put 4 name mappings in place (2 win_unix, 2 unix_win). The Probe finished with the 3 standalone runs (Windows, the Linux cifs mount, and Linux NFS) and all 8 pairs (4 lock, 4 visibility) at Success
Reading ls -l: on ntfs, the mode bits are a mapping of the ACL and do not tell you whether a write is allowed (this reading is a hypothesis). ADR-3 decides the Linux writability check by whether a real I/O attempt succeeds, not by the mode bits (a design decision; see the stage 2 chapter)

Security note
I read the denial of NFS root as an effect of the export policy's superuser none, but that is an inference from the setting (hypothesis). I did not try NFS without name mappings.

This chapter gives no case-sensitivity numbers. The stage 0 and 1 records have no positive control, so they support no conclusion (the re-measurement is in "Behavior by access path").

Sources for this chapter: the stage 0 re-measurement, the two-client behaviors in stage 1

What you can decide: when you add NFS to an existing SMB setup, this environment needed no data migration. Put the procedures that use root, and the procedure of checking permissions with ls -l, on your review list.

Stage 2: atx up to net10.0, AIMF up to ADRs and a ledger

atx got as far as a net10.0 build, but five legacy-code habits (ways of writing that assume Windows and SMB) remained after conversion. AIMF produced ADR-1 to 5 and ledger AP-1 to 14 and went on to implement NFS and S3 Access Points. The scopes differ.

Step	Run date and duration	Unit and cost	Output and result check	Who made the fix
atx analysis (comprehensive-codebase-analysis)	2026-10-11 09:48 to 10:03 UTC (about 15 min), exit 0	92.46 / 120.00 agent minutes (atx's billing unit), about $3.24 (estimate)	28 `ATXDocumentation` files, no code changes. The gitleaks scan before upload found no leaks	Analysis only
atx dotnet-modernization	2026-10-11 11:05 to 11:25 UTC (about 20 min), exit 0	139.00 / 300 agent minutes (atx's record). In the hourly Cost Explorer data, 11.025 minutes were billable (about $0.39, Estimated); the other 127.97 minutes do not appear as billable (breakdown below the table in "Costs that could not be attributed, and why")	11 files, +55/−162. net10.0, SDK-style csproj, settings moved to `IConfiguration` and `appsettings.json`. The parity gate (a check that the old and converted code agree on Config, Boot, Endpoints, and Auth) passed on all 4 items. Build succeeded on macOS with 0 errors and 7 warnings	atx did the conversion. The Linux fixes were my separate work (not an atx feature)
AIMF (Phase 0a to 4)	2026-10-12 13:50 to 17:23 JST (3 h 33 min; see ※)	284.26 Kiro credits (AIMF's billing unit; limit 5,000, notification at 4,000). Not converted to dollars	ADR-1 to 5, ledger AP-1 to 14. Build warnings 0, 8 tests passed (later 54), a self-contained Linux publish output of 196 files (81 MB); the sha256 of all 196 files matched after deployment	AIMF did the Path.Combine change and the real-I/O writability check. AIMF ported my Probe fix that adds the positive control

※ The 3 h 33 min is the span from the first to the last commit in the records repository. It is not the session length. The log starts with an approval at 13:25:54, and the span includes waiting for my decisions.

Sources for this table: the atx and AIMF comparison in stage 2, cost estimates and observed costs in stage 2 (I reorganized the columns for this post)

The five legacy-code habits that remained after the atx conversion:

The backslash join in StorePaths.Combine
The Docs\Index.json reference
ACL: File.GetAccessControl was only replaced by FileInfo.GetAccessControl, which does not work on Linux. No NfsFileStore was added either
The FileShare.None exclusive lock
The assumption that a file can be read back right after it is written

When I ran the code that atx converted on Linux (2026-10-11, self-contained, one observation), over NFS, the path-separator, file-locking, and write-visibility behaviors failed with UnauthorizedAccessException, and acl-evaluation failed with PlatformNotSupportedException. On the Linux cifs mount side, Combine created names joined with \ directly under /mnt (a local xfs), outside the mount, and nothing was written to the share.

The AIMF ledger, verification, and decisions:

Ledger: of AP-1 to AP-14, 12 resolved, 1 replaced by an alternative implementation, 1 not needed, 0 open
Negative tests of the detectors for AP-1 to AP-13 (I injected a temporary violating file and checked that it was detected): 13 pass, 0 fail
Independent verification: one run by a separate agent with a separate context. It found no missing functionality and raised one scope point: OpenExclusive cannot be used on S3, and no one had approved that limitation. AP-14 and ADR-5 closed the point
ADRs: ADR-2 puts performance out of scope. ADR-3 makes the Linux writability check a real I/O attempt instead of a pre-check. ADR-4 makes index.json compact System.Text.Json output (ADR-4 was my decision)
FileShare.None: AIMF also kept this exclusive lock on purpose (AP-4)

Conditions of the comparison: the scopes differ (atx goes up to the .NET 10 conversion and the parity gate; AIMF goes up to implementing NFS and S3 Access Points), and running AIMF depends on the atx analysis. I measured one app, once per tool, and the cost units are not converted, as the table shows.

What you can decide: if you only need to raise the .NET version and check agreement with the old code, atx's scope is enough. If you also need a per-path implementation and a record of decisions, that is AIMF's scope (how to choose is in the later chapter "Choosing between atx and AIMF").

Stage 3: Lambda reaching the same volume through S3 Access Points

A Lambda Worker (.NET 10, in the VPC) wrote 5 reports and index.json through S3 Access Points, and they were visible from the NFS side (one observation). Three conditions held when the Worker reached the volume: NetworkOrigin set to the VPC, the WINDOWS user appsvc as the identity, and an AD domain controller (DC) that was reachable. I have not confirmed whether the DC must be reachable.

Attachment: AWS::FSx::S3AccessPointAttachment reached CREATE_COMPLETE, and Lifecycle was AVAILABLE
NetworkOrigin: VPC. Changing it means deleting and recreating the access point (Creating access points restricted to a virtual private cloud, Configuring network access for Amazon S3 access points)
Stack creation, invocation, and deletion: 3 times (s3, s3b, s3m). The wait for delete-stack took more than 20 minutes (one recorded case), and no attachment remained after deletion
Lambda settings: 512 MB, 120-second timeout. The EventBridge Scheduler rule was created as DISABLED

Results of the runs:

atx-derived code (first run): the Worker exited 0 and wrote 5 reports and index.json to out/s3/. The run took 2,505 ms (Init 361 ms) with peak memory of 115 MB. The Probe took 821 ms with peak memory of 120 MB
AIMF-derived code (s3m): index.json was 181 bytes (confirmed with ls; I did not read the contents). The Probe gave 4 measured and 1 skipped (file-locking). Lambda ran only as the writer, so there is no write-visibility value

A name mapping that appeared only while the access point was attached (the per-path behavior is in the table of the next chapter, "Behavior by access path"):

Name mapping s3_win (replacement set to the AD domain's appsvc): it appeared in ONTAP only while the access point was attached; ONTAP held 5 name mappings at b3 while attached and 4 after deletion and at b2. I have not confirmed in documentation whether the attach process creates s3_win (one observation)

I did not check this time whether the AD domain controller must be reachable (the first item in "What has not been measured yet"). Because S3 Access Points cannot use event notifications, I used polling from EventBridge Scheduler as the trigger. I read the table as meaning that event notifications are unavailable, because the AWS API support table lists GetBucketNotificationConfiguration as not supported (Access point compatibility (FSx for ONTAP User Guide), checked on 2026-10-21, documented). That table lists only some of the APIs, and it has no row that names event notifications as unsupported.

Request charges in Tokyo are $0.00108 per 1,000 PUT, COPY, POST, and LIST requests, and $0.000029 per 1,000 GET and other requests. I read them on 2026-10-12 in the "S3 Access Points for FSx" tab of the English Amazon FSx for NetApp ONTAP pricing page, and on 2026-10-21 I opened the same tab in a browser and saw the same values. On the same day I also read them again in the Price List API under the AmazonS3 usage types APN1-Requests-FSXONTAP-Tier1 and Tier2; the values were the same. 1,000,000 PUT or LIST requests come to $1.08, which is unit price × count arithmetic (the billed amount is unconfirmed).

What stayed unchanged at the stage boundaries: the UUID, security style, seed, snapshots, and snapshot locking did not change from b0 to b3. The export policy attached to appdata changed once, from default at b0 to appmod_nfs (set in stage 1) at b1, and did not change from b1 to b3. check-invariant raised one finding at b2 (s2m) and at b3 (s3m), and passed at b3 after the deletion (the details are in failure 5).

Sources for this chapter: the table of the five behaviors in stage 3, the effect of a \ key on the SMB side, and the name mapping that appears only while attached

What you can decide: whether Lambda can use the same volume without copying the data depends on whether you can meet the four prerequisites in "Staged design and the single-volume constraint" in your own environment.

Behavior by access path

Even on one volume, case sensitivity, the handling of names containing \, whether an exclusive lock was possible, and the writability check differed by path, and write visibility differed by direction. What stayed the same across the paths was the volume they reach, the seed sha256, and the Worker output as JSON values.

A caution on reading: the implementation changed together with the path. The Windows column is the old build (.NET Framework 4.8) and the others are the modernized build (.NET 10), so some differences cannot be attributed to the path alone. Unless noted otherwise, the tier of each cell is verified.

Behavior	Windows (earlier .NET Framework 4.8 build, SMB)	Linux cifs (.NET 10)	Linux NFS 4.1 (.NET 10, uid 10001)	Lambda (.NET 10, S3 Access Points)	Source runs
Case sensitivity	Case-insensitive (a reference with different letter case resolved: true)	Case-insensitive (resolved: true)	Case-sensitive (did not resolve: false)	Case-sensitive (did not resolve: false)	s4c, s3b. s2m, s2n, and s3m show the same values as s4c and s3b
Names containing `\`	Treats `\` as a separator (1 run)	May reject a name with `\` (errno 22). Not re-observed with the post-AIMF Probe	An ordinary character (Python version). The .NET Probe got "not found" for the same name (1 run each)	The key `probe/sep\check-s3.txt` could be written and read back. The first design, a key made only of `\`, was created directly under the volume root and appeared in the SMB listing as `PROBES~1.TXT` (field-observation, 1 run)	s1, s2f, s2g, s2m, s3m, s3
Exclusive locking	With the holder (the side that opens first) and the contender (the side that opens afterwards) swapped between Windows and Linux, the contender was refused in both directions and succeeded after release (1 run per pair)	Same result as the Windows column, in pairs with Windows	Same result as the Windows column, in pairs with Windows	skipped (no operation to open exclusively). not measured	s2m (Python version: s0, s1)
Writability check	Pre-evaluates the NTFS ACL: writable	Pre-check skipped; real I/O, run as root, showed it writable	Pre-check skipped; real I/O showed it writable	Pre-check skipped; the PUT was allowed	s2m, s2n, s3m
Write visibility	Takes part as one half of the pair with Linux (the values are in the two Linux columns, by direction)	66 ms when Windows writes and cifs reads; 7,968 ms when cifs writes and Windows reads	35 ms when Windows writes and NFS reads; 8,342 ms when NFS writes and Windows reads	not measured	s2m (the same shape, fast when Windows writes and slow in the other direction, in s1 and s0)

Source of this table: the source material is spread across the per-stage documents. I combined it into one table for this post, and the public repository has no such table (commit-pinned).

Case sensitivity (SMB, cifs, NFS): the stage 1 table
Exclusive locking and write visibility: the stage 2 measurements of the modernized app (7,968 / 8,342 ms; Worker output 201 / 181 bytes)
The S3 Access Points column, names containing \, and name mapping: the links at the end of the stage 3 chapter (two tables in stage 3)
The original tables for exclusive locking and visibility with two clients: the links at the end of the stages 0 and 1 chapter (stage 1)

Two points are not in the table: identity (which user a request is treated as) and Worker output. For identity, the treatment of root is in the stages 0 and 1 chapter and the stage 3 chapter. For Worker output (index.json), the old Worker wrote 201 bytes and the modernized Worker (Linux) that AIMF built wrote 181 bytes. The two index.json files are equal as JSON values (JSON-equal), and the 20-byte difference is whitespace (one run each). The Lambda Worker exited 0 and wrote 181 bytes (confirmed with ls; I did not read the contents). The source runs are s0w, s2m, and s3m.

In the write-visibility measurement, the reading client tries every 100 ms (polling interval 100 ms) to read the content the writer wrote, so differences below 100 ms (35 and 66) cannot be told apart. The asymmetry, fast when Windows writes and slow when Linux writes, had the same shape in three runs (s0, s1, s2m). I did not isolate why only the direction where Linux writes and Windows reads takes about 8 seconds. The Linux writer was Python (with fsync) in s0 and s1 and .NET (without fsync) in s2m, so the implementation differed too. In a separate verification, the reader's NFS mount options alone changed the time until a deletion made with S3 DeleteObject became visible over NFS, from 7 ms (actimeo=0) to 2,171 ms (default mount) (2026-08-09, ap-northeast-1, NFSv3, UNIX security style, concurrency 1, ONTAP version not identified, carried-over; measurement of S3 and NFS visibility on the same volume, Japanese). I have not confirmed whether that is the same cause as the 8 seconds in this post. Details of the exclusive-locking pairs, such as the result of the read-only open, are in the original stage 1 table.

What you can decide: when you add a path, you can make these five behaviors an item to measure again for each added path. On case sensitivity, Windows and cifs were insensitive while NFS and S3 Access Points were sensitive, so do not apply the SMB result to NFS or S3 Access Points.

Eight failures and the checks to put in place before measuring

There were 8 failures. The largest was the case-sensitivity measurement with no positive control, which made the case-sensitivity result unusable as a conclusion in 6 runs. The other 7 concern record shapes, environment creation, and side effects of scripts.

The eight failures

Case sensitivity with no positive control: invalid in 6 runs (s0, s1, s2, s2f, s2g, and the first stage 3 run, s3)
- Fact: I looked for the reference Docs\Index.json at the volume root, but the actual file was seed/docs/index.json. Even on case-insensitive Windows SMB the lookup resolved to false, and taken at face value this would have meant "SMB is case-sensitive"
- Fix: added the positive control and measured again as s4c and s3b
A measured on the cifs mount side (SMB used from Linux; measured is the state in which the Probe recorded that it could measure the behavior) that was not written to the share: s2 left 3 files directly under the local /mnt
- Fact: the cifs-mount-side measured did not mean a successful write to the share. It meant that 3 files had been created directly under the local /mnt
- Fix: deleted the 3 files that had been created
topology: cross-host, which does not show two machines coordinating: the Probe converted to .NET 10 emitted it even when run standalone
- Fact: the record's topology held the fixed value cross-host, which is supposed to show that two machines coordinated. The value was not derived from a sync check, so it is not evidence
- Fix: changed to a step that judges a pair only when the two records carry the same sync ID (probe_merge.py). In s2m, all 8 pairs had empty mismatch reasons and were judged as pairs
A writability record that disagreed with ADR-3 (s2m), and the missing comparison baseline out/s0/ for the Worker output
- Fact: the s2m record held only the pre-evaluation value, which did not match the shape in ADR-3 (pre-check skipped, decided by real I/O). The baseline out/s0/ did not exist either, so the s2m JSON-equality check was a comparison against reference values
- Fix: fixed the Probe and measured again as s2n to get the ADR-3 shape. I ran the old Windows Worker once and made the baseline as s0w, and the reference values (the index.json written by the Worker that atx transformed) and the old Worker's index.json were identical down to the bytes (201 bytes)
check-invariant failing because of PROBES~1.TXT: two boundaries, b2 and b3
- Fact: the first S3 Probe design wrote a key made only of \ with no slash. My reading that the key made only of \ was mapped to the 8.3 short name PROBES~1.TXT is an inference (PROBES~1.TXT appeared before the NFS deletion and was gone after it, which supports the inference)
- Fix: after I moved the key under probe/, the new key did not appear at the root. I deleted the file that had been created at the root with one NFS rm, with my approval. The first attempt at the deletion script was aborted before deleting anything, because the root user could not see the volume's top-level directory (nothing changed)
AIMF's Phase 3 Step 1 (the step that creates the implementation tree) failing because the base branch (base-ref) master could not be resolved: once
- Fact: the base-ref master could not be resolved in the implementation tree (a working clone), and the step failed
- Fix: reran with origin/master and it succeeded
Two kinds of failure while creating the base environment: the template size limit and a missing secret
- Fact: base.yaml was 52,466 bytes and exceeded the 51,200-byte limit for --template-body (no resources were created). On another attempt, the Secrets Manager secrets did not exist, so the Directory resource reached CREATE_FAILED and the stack rolled back (no resources remained)
- Fix: deploy.sh now puts the template in the deployment Amazon S3 bucket and passes it from there. I now create the secrets with create-secrets.sh first and then run the base creation again
A script that ignored --help and ran: it created 4 Secrets Manager secrets (01:24 JST on 2026-10-13)
- Fact: I called create-secrets.sh, a script that does not parse arguments, with --help to check its usage. It ignored --help, ran its main process, and created 4 Secrets Manager secrets. About 7 hours after they were created, I deleted those 4 secrets. The cost of those 4 secrets is an estimate: $0.40 per secret per month, prorated by hours. 4 secrets × $0.40 × 7 hours ÷ 730 hours (taking a month as 730 hours) comes to about $0.015 (1.5 cents)
- Fix: added to the build a gate, check_help_is_safe, that calls every script with --help and fails if any script does anything

Five preventive checks

Put the positive control in place before measuring (failure 1). Without a file to find, even SMB returned false
Put the NFS mount in a script (the first Probe in stage 1 failed because NFS was not mounted; that failure is not counted among the 8 above)
Check that the secrets exist before building the base environment (failure 7)
Put S3 keys under probe/ (failure 5)
Make "--help does nothing" a build gate: the build checks that --help executes nothing (failure 8)

Source for this chapter: why a measurement without a positive control is invalid (record validity is in the link in "Test environment and method")

What you can decide: before your own measurements, you can put these five points on a checklist. The positive control is needed in every measurement whose result can end as "not found".

Choosing between atx and AIMF

atx and AIMF cover different scopes, so the table does not say which is better. It lets you choose by the scope you need (the result of processing the same app once with each).

Aspect	atx	AIMF
What it is	A tool for code analysis and transformation	A workflow that runs Phase 0a to 4 through kiro-cli
Input and output in this verification	Input: the source. Output: 28 analysis documents and an 11-file transformation. The `index.json` (201 bytes) written by the Worker that atx transformed matched, byte for byte, the `index.json` written by the old Worker (.NET Framework 4.8) before the transformation	The atx analysis in; ADRs, a ledger, an implementation, and tests out. The `index.json` (181 bytes) written by the modernized Worker (Linux) that AIMF built is JSON-equal to the old Worker's `index.json`; ADR-4 made the output compact, so the bytes are 20 whitespace bytes smaller
Duration	About 15 min for analysis, about 20 min for the transformation (run time)	3 h 33 min from the first to the last commit (not the session length; includes waits for my decisions)
Cost unit	agent minutes	Kiro credits (not converted to dollars)
Human involvement	I ran it with `run-atx.sh` (I had put in a guard that forbids running it directly from an agent)	6 decisions and 13 manual approvals from me; the AIMF run was held 35 times, waiting for my decision, because the approver script did not approve automatically (reasons: a command not on the allowlist of operations it approves automatically, a match with a forbidden pattern, a cd outside the working directory, and so on; the count is the STOP lines in the execution log)
Trade-offs	The output contains no NFS or S3 implementation and no decision records such as ADRs or a ledger. Five legacy-code habits remained	It needs setup (a records repository, hooks, running only with `--agent migration`, and `install.sh` writing `.kiro/` and `CLAUDE.md`). Credits cannot be converted to dollars, and the shape of a Probe record did not match its ADR (failure 4)

Duration and cost are measured differently and in different units, so I do not compare atx and AIMF as shorter or longer, or cheaper or more expensive. Not measured for either tool: run-to-run variation, larger apps, other languages, and maintainability.

Source of this table: the same comparison table as in the stage 2 chapter (I reorganized the columns).

How to choose, in three points:

If moving to .NET 10 and checking agreement with the old code is enough: atx. Plan for the legacy-code habits that remain after conversion
If you need to add paths or keep a record of decisions: AIMF. Plan for the setup, the 3 h 33 min of this verification (the span between commits), and my 6 decisions
If you already have an atx analysis: import it in AIMF's Phase 0a. In every case, re-measuring each path comes next

flowchart TD
    Q1{"Adding access paths?"} -->|Yes| Q3{"Already have an atx analysis?"}
    Q1 -->|No| Q2{"Need decision records?"}
    Q2 -->|No| A["atx: move to .NET 10 and check parity"]
    Q2 -->|Yes| Q3
    Q3 -->|Yes| C["AIMF, importing the atx analysis"]
    Q3 -->|No| B["AIMF"]
    A --> R["Re-measure each path"]
    B --> R
    C --> R

The diagram above summarizes the table and the list.

What you can decide: two questions, whether you are adding paths and whether you need decision records, tell you whether to start with atx or with AIMF. The differences in time and cost are single observations, so do not use them as the basis for choosing.

Cost estimates and observations

Per row, the hourly-rate estimate and the value I attributed from Cost Explorer differed by −1% to −3% (largest −3.06%). This difference checks whether the hourly rates in the estimate match the observed values; it is not a reconciliation with an invoice. The cost identified for the whole verification is $35.91. That figure does not deduct the AWS Transform Credit line (−$3.62); if the Credit is deducted, the AWS Transform portion nets to $0.

The values in this chapter are either estimates (unit price times hours) or single observations from one environment. Cost Explorer values are before billing is final (Estimated: true). They are not invoice amounts, and they do not guarantee that production would cost the same. I read them on 2026-10-13 JST with UnblendedCost, Credit and Refund excluded, for 2026-10-11 and 2026-10-12 (UTC dates). I read them again on 2026-10-21 JST under the same conditions: the main rows were unchanged to three decimals, and the values were still marked Estimated: true. The account is shared with other work, so I did not use service-level totals. I used only the values filtered by the CloudFormation tag aws:cloudformation:stack-name (appmod-base, appmod-stage3) and the values by resource ID.

Item	Estimate	Estimated duration	Actual uptime and observed value
Base environment (second build)	$57.71	72 hours	About 33.6 hours (stack created 2026-10-11T02:08Z; deletion of `appmod-base` finished about 11:44Z on 2026-10-12). The tag-attributed observed value is $21.18, excluding Directory Service
Base environment, Directory Service	Hourly rate $0.146 × 33.6 hours ≈ $4.91	33.6 hours	Not separable by resource. The difference method (estimating this verification's share by subtracting the steady usage of days without the verification from the usage of the verification days) gives about $4.80
Base environment (first build)	$19.23	24 hours	Tag-attributed $6.31 (2026-10-07). The uptime is unknown, so I did not compare it with the estimate
atx analysis and dotnet-modernization	Analysis about $3.24 (92.46 min × $0.035; upper bound $4.20). dotnet $0 inside the free allowance, about $4.87 outside (upper bound $10.50)	About 15 min for analysis, about 20 min for dotnet	Cost Explorer shows AWS Transform on 2026-10-11 as Usage $3.6221 (103.49 minutes: 92.465 for analysis, $3.24, and 11.025 for dotnet-modernization, $0.39) and a Credit of −$3.6221 for the same amount, so the net is $0. 2026-10-12 is $0
Lambda (each stage 3 run)	Upper bounds of $0.04 (s3), $0.01 (s3b), $0.01 (s3m)	Not applicable	$0 on both days when filtered by tag. I did not measure the actual request counts
AIMF	284.26 Kiro credits (limit 5,000)	3 h 33 min from the first to the last commit	As in the estimate column. Not converted to dollars

Source of this table: the estimate and actual uptime in docs/en/reproduce.md (commit-pinned; I reorganized the columns)

The $57.71 estimate for 72 hours is not the cost of the roughly 33.6 hours the environment was up. The whole estimate prorated to 33.6 hours is about $26.9 (the breakdown is in the private estimate made at approval time).

Hourly-rate estimates against observed values

I prorated the estimate's hourly rates to the same 33.6 hours of uptime and compared them.

Item	Estimated hourly rate × 33.6 hours (USD)	Observed (USD)	Difference
FSx for ONTAP (SSD 1,024 GiB + 128 MBps throughput)	12.41	12.03	−3%
EC2 (Windows + Linux)	6.41	6.36	−1%
VPC interface endpoints	2.82	2.77	−2%
AWS Managed Microsoft AD	4.91	4.80 (difference method, inference)	−2%
Total (excluding AD; the observed value includes $0.02 of S3)	21.64	21.18	−2%

Source of this table: the tag-attribution table and the estimate-against-actual comparison in docs/en/environment.md (I renamed the column "Observed (tag)" to "Observed" and changed some row names)

The observed value 21.18 is the sum of FSx for ONTAP 12.03, EC2 6.36, VPC 2.77, and S3 0.02. Each value is the two-day sum of the unrounded figures Cost Explorer returned, rounded afterwards. If you round each daily row first and add them up, some days are off by 0.01 (for 2026-10-11, the unrounded total is 13.8632 and the sum of the rounded rows is 13.87). The AD figure of 4.80 is an inference, so it is not in this total.

Identified cost of the whole verification

Item	Day (UTC) and basis	Amount (USD)
Tag-attributed: first build	2026-10-07	6.31
Tag-attributed: second build	2026-10-11, 2026-10-12	21.18
Tag-attributed subtotal	—	27.49
Directory Service, second build	Difference-method inference	4.80
AWS Transform usage	Cost Explorer Usage line (Credit line excluded)	3.62
Identified cost	—	35.91

Source of this table: the AD difference method, the identified total, and the AWS Transform line in docs/en/environment.md (I merged two tables into one)

Not in the $35.91:

The first build's Directory Service: an estimate only, about $1.6
Secrets Manager, CloudWatch, and Lambda: not separable; estimated at $0.1 to $0.2 in total
The four secrets I created by accident (failure 8): expected to be about $0.015 (1.5 cents)
The AWS Transform Credit line: with it included, the net is $0

Costs that could not be attributed, and the teardown check

This section answers two questions. The first is which costs of this verification I could not attribute, and why: AWS Directory Service, which Cost Explorer cannot filter by resource ID or by tag, and AWS Transform, whose billed minutes do not match the minutes in the atx records. The second is what I checked to confirm that everything created for the verification was destroyed. The cost values are Cost Explorer values before billing is final (Estimated), or inferences from them, and they are not invoice amounts.

Costs that could not be attributed, and why

Item	Amount (USD; Cost Explorer values are Estimated)	Attributed?	Why	Source and tier
AWS Directory Service (AWS Managed Microsoft AD, Standard, 2 DCs), second build	About 4.80	No. Inferred with the difference method	Cost Explorer shows no resource ID (`NoResourceId`), and the resource carries no CloudFormation tag	Inference by the difference method (assumes the baseline is constant). Calculation in the next table
AWS Directory Service, first build (2026-10-07)	About 1.6	No. The difference method cannot be used	The usage on the day before, 2026-10-06, was 48.52 DC-hours, half of the baseline, so the baseline cannot be called constant	Estimate only. Not in the $35.91
AWS Transform (2026-10-11)	Usage line 3.6221; Credit line of the same day −3.6221	The amount, yes (the service-level Usage line). The minutes, no match	The Usage line shows 103.49 minutes. The atx records show 92.46 minutes for analysis and 139.00 for dotnet-modernization, 231.46 in total, so the gap is 127.97 minutes. All of the gap is in dotnet-modernization. A free allowance may be the cause, but I have not confirmed it (details below the table)	Cost Explorer Usage and Credit lines (Estimated); atx records

The identified cost of the whole verification, $35.91, includes the roughly $4.80 in the first row and the AWS Transform Usage line of $3.62. It does not include the roughly $1.6 in the second row. It does not deduct the AWS Transform Credit line (−$3.62); if the Credit is deducted, the AWS Transform portion nets to $0.

I could explain part of the AWS Transform minute gap (2026-10-11). The match between the hourly Cost Explorer data (UTC) and the atx log (debug.log) is verified (once each). Whether a free allowance is the cause is a hypothesis.

How the minutes are counted: the atx figures of 92.46 and 139.00 minutes are sums over the main agent and the sub-agents, not wall-clock time (verified). Wall-clock time was about 15 minutes for the analysis and about 20 minutes for dotnet-modernization. The pricing page says that agent minutes exceed wall-clock time when several agents run in parallel (documented). The analysis alone has 78.95 minutes on the main agent, which is already more than its wall-clock time of about 15 minutes, so I have not confirmed that parallelism by itself explains the difference.
The analysis matches: the hourly Cost Explorer data (UTC) shows 63.318 minutes at 09:00 and 29.147 at 10:00, 92.465 in total. That equals the atx record for the same hours (63.318 and 29.147).
Where the gap is: the Usage for the 11:00 UTC hour, when dotnet-modernization ran, is 11.025 minutes. The other 127.97 of the 139.00 minutes in the atx record do not appear as billable.
Dates and unit: both atx runs (the analysis and dotnet-modernization) fall inside UTC 2026-10-11, so a date boundary is not the cause. The total from 1 September to 21 October is also 103.49 minutes. The unit is minutes (AgentMinute).
A possible free allowance: AWS announced that the AWS-managed .NET modernization transformation includes 50,000 free agent minutes per month (AWS: .NET modernization via CLI is generally available; the allowance's existence is documented). Most of the dotnet-modernization minutes may have fallen inside it, but Cost Explorer shows no line for the allowance and I could not see how much of it was used.
Still unexplained: why exactly 11.025 minutes were billed. The seven PhaseValidator sub-agent runs (validation steps) in the same execution total 11.027 minutes, which is close to 11.025 minutes, but they differ by 0.002 minutes and chance cannot be ruled out. I also did not confirm why CloudWatch reports 144.75 minutes for dotnet-modernization against 139.00 in the atx record (a 5.75-minute difference).

The difference method is an inference that subtracts the steady usage of days without the verification from the usage of the verification days, to find this verification's share. The calculation for the second build's Directory Service is as follows.

Item	Value	Calculation or basis
Baseline (mean of days without the verification)	95.72 DC-hours per day	Mean of 2026-10-08 to 2026-10-10
Difference on 2026-10-11	41.40 DC-hours	137.12 − 95.72
Difference on 2026-10-12	24.36 DC-hours	120.08 − 95.72
Total difference	65.76 DC-hours	2.1% off the expected 67.2 DC-hours
Unit price	0.073 USD per DC-hour	7.012 USD ÷ 96.05 DC-hours on 2026-10-08
Estimated amount	About 4.80 USD	65.76 × 0.073

The 0.073 USD per DC-hour matches the Price List unit price for AWSDirectoryService, APN1-Std-MicrosoftAD-DC-Usage (Tokyo, Standard); I read it on 2026-10-21.

I read the atx unit price of $0.035 per agent minute again on 2026-10-21 in the Price List (AWSTransform, APN1-AgentMinute, Tokyo). It had not changed.

Teardown check results

I tore the base environment down on 2026-10-12. I confirmed the teardown through API listings and the teardown steps. The results are as follows.

Target	How it was checked	Result	Date
Stacks starting with `appmod`	API listing	0	2026-10-13 JST
S3 buckets starting with `appmod`	API listing	0	2026-10-13 JST
Secrets starting with `appmod`	API listing	0	2026-10-13 JST
Lambda functions starting with `appmod`	API listing	0	2026-10-13 JST
S3 Access Points attachments starting with `appmod`	API listing	0	2026-10-13 JST
Scheduler schedules starting with `appmod`	API listing	0	2026-10-13 JST
Log groups starting with `appmod`	API listing	0	2026-10-13 JST
File system, directory, EC2 instances, VPC	API listing	`NotFound` or `terminated`	2026-10-13 JST
Leftover FlexClones	Environment teardown steps 4 and 5	None	2026-10-07 (first build), 2026-10-12 (second build)
`it_*` Snapshots	Environment teardown steps 4 and 5	None	2026-10-07 (first build), 2026-10-12 (second build)
Recovery queue	Environment teardown steps 4 and 5	Empty	2026-10-07 (first build), 2026-10-12 (second build)
Daily cost for 2026-10-13 (filtered by the tags `appmod-base` and `appmod-stage3`)	Check in Cost Explorer (daily, UnblendedCost)	0 (`Estimated: true`)	2026-10-21 JST

What you can decide: for a configuration where you know the uptime, you can use the unit-price-times-hours estimate as material for a budget draft, as a single environment's data point. Before you use it as the basis of a budget, check an invoice. The items to check are the three whose limits are written in this chapter:

I have not yet reconciled the figures with an invoice
For the first build's Directory Service I have only an inference, not an actual amount
The 127.97-minute gap in AWS Transform minutes is entirely in dotnet-modernization. A free allowance may be the cause, but I have not confirmed it

What has not been measured yet

This chapter lists only the items that you need when deciding whether to apply this approach to your own app migration, and that I found measured nowhere in the documents of my other repositories that I checked. There are three. For each item I write what I did not measure, what another verification measured, when not knowing it hurts, and how to measure it. The results of other verifications are labeled carried-over. They are not values measured in this post's environment.

These items are not in this chapter. Security styles unix and mixed, other ONTAP versions and Multi-AZ, reconciliation with an invoice, and performance are in "When this post fits and when it does not" and "What this post covers and what it does not". The limits of the cost figures are in "Cost estimates and observations". I left out NFS without name mapping, the case where the writability check finds a file not writable, NFS Kerberos and mount options, and pairs of the same OS. They are variants of this post's question, and I judged that they add nothing to your decision about migrating your own app.

S3 Access Points data operations while the AD domain controller is unreachable
- What I did not measure: the results of S3 Access Points data operations (ListObjectsV2, GetObject, PutObject) on an AD-joined SVM when the domain controller (DC) is unreachable. In this verification I used the access point with the DC reachable
- Status of public information (open): whether every data operation on an AD-joined SVM requires DC reachability is something I could not confirm, either from a record a reader can reproduce or from a public primary source (What an endpoint needs to reach S3 Access Points, Japanese)
- What another verification measured (carried-over): on an SVM that was joined to a domain but found no DC at all, HeadBucket, ListObjectsV2, GetObject, PutObject, and DeleteObject all succeeded when the identity resolved to a local SMB user on the SVM (2026-08-26, ap-northeast-1, ONTAP 9.18.1P3D1, 128 MBps, an NTFS volume; file portal identity and visibility measurements). With a domain account as the identity, that verification could not tell whether a DC is required, because its environment had no reachable DC. That document says a re-measurement in an environment with a reachable DC is needed. This post's identity is the AD domain account appsvc and its environment had a reachable DC, so the conditions are close to that re-measurement. But I did not cut off the DC in this post either
- When not knowing it hurts: you use S3 Access Points on an AD-joined SVM and AD goes down
- How to measure: cut the route to the DC with a positive control and take the result of each operation. Do not judge by a successful HeadBucket alone
When a file written by Lambda through S3 Access Points overlaps with a file a Windows app handles over SMB
- What I did not measure: the results of two things. A write from S3 Access Points to a file while SMB holds an exclusive lock on it. The time until Windows can read a file written through S3 Access Points, and the time until S3 Access Points can read a file Windows wrote over SMB. In this verification Lambda ran only as the writer and there was no reader. The S3 API has no operation to open exclusively, and file-locking was skipped
- What another verification measured (carried-over): the median time until a 64 B object written through S3 Access Points could be read directly from the volume over NFS (NFSv3, actimeo=0) was 3 ms, and until it could be read from the FlexCache side (FlexCache is an ONTAP volume that caches the data of a source volume) over a Linux cifs mount (SMB 3.0, cache=none) it was 7 ms (concurrency 1, 30 runs, 2026-08-09 and 2026-08-10, ap-northeast-1, ONTAP 9.18.1P3D1, 128 MBps, UNIX security style, UNIX root identity; comparison of visibility in all directions, Japanese). These are not values read by a Windows client, and not values for an NTFS volume
- When not knowing it hurts: a Lambda function and a Windows app handle the same file at the same time. Also when you must build into the design the wait until Windows can read a file Lambda wrote
- How to measure: design a pair experiment that links Lambda and Windows by a sync ID, and measure both directions
The five behaviors on a clone of an NTFS volume made with FlexClone
- What I did not measure: the five behaviors of a clone of an NTFS volume made with FlexClone, opened over SMB, NFS, and S3 Access Points. I did not create a single FlexClone in this verification
- What another verification measured (carried-over): when I attached S3 Access Points to a clone of a 1 GiB volume with the UNIX security style, LIST, GET, and PUT worked. The sha256 of a 256 MiB file matched the source volume, and an object PUT to the clone did not appear in the source volume (2026-08-26, ap-northeast-1, ONTAP 9.18.1P3D1, 128 MBps, UNIX root identity; measurement of ONTAP features over S3 Access Points, Japanese). That document has no measurement of a clone of an NTFS volume
- When not knowing it hurts: you hand out a working volume per developer or per stage from a Snapshot of appdata and apply the results measured on appdata to the clone as they are
- How to measure: create a clone of an NTFS volume and run the same Probe over each path that opens the clone

Governance note
This article describes design considerations. It is not a legal, compliance, or regulatory judgment. The ADR option summaries and the independent verification result are AI output and serve as supporting signals. I made the decisions, including the decision to keep the security style at ntfs.

What you can decide: look at which of the three items above your environment falls under, and decide whether to add that item's "How to measure" to your own measurement items.

Summary and choosing an access path

The results are the three findings at the start of this post (the converted app ran on Lambda without moving the data, what changed with the path, and the difference in scope between atx and AIMF). There were 8 failures, and the measurement with no positive control invalidated the case-sensitivity values in 6 runs, more than any other failure.

The three paths coexisted on the same volume up to stage 3. The criterion for choosing is which client you use.

Choice	When	Constraints (from this environment's measurements)
Stay on SMB	You keep the existing Windows app unchanged and want to keep the case-insensitive assumption	Windows and cifs resolved a reference with different letter case. NFS and S3 Access Points did not. Write visibility (the time until a file one client wrote becomes visible to the other) was 66 ms with Windows writing and cifs reading, and 7,968 ms in the reverse direction (s2m, one observation)
Add NFS	You want a Linux app to run on the same data	On NFS, a reference that differs only in letter case does not resolve. the NFS root user could not enter `/mnt/appdata` (the cause is an inference from the export policy setting), while the uid 10001 user could read and write. I did not measure the case with no name mapping
Serve through S3 Access Points	An S3 API client such as Lambda needs to use the data	On S3 Access Points, a reference that differs only in letter case does not resolve. The S3 API has no operation to open exclusively, and a key made only of `\` was created directly under the volume root and appeared in the SMB listing as an 8.3 short name (one observation). The fixed NetworkOrigin and the AD domain controller reachability (open) are as in the table in the design chapter

The repository is here:
https://github.com/Yoshiki0705/FSx-for-ONTAP-App-Modernization-Patterns

The previous article is "AWS Transform Now Supports Block Storage Migration to FSx for ONTAP — Benefits and Pitfalls from a Hands-On EC2 Test".
Related articles (two):

I thank the people who publish AIMF and atx, and the people who support my day-to-day work.

I hope this is useful to anyone considering moving a .NET Framework app to a newer .NET without moving its data.

See you next time.
Bye now!!
Yoshiki Fujiwara

Resources

AIMF (pinned to v0.11.0)
AIMF introduction blog (Japanese)
NetApp explanation of security styles
Where AI Modernization Flow fits (FSx for ONTAP Adoption Playbook) (Japanese) — AIMF owns the app side, the Adoption Playbook owns the data side; the linked document cites this article's field results
FSx for ONTAP auxiliary pack (Japanese) — a drop-in pack for an AIMF migration project, surfacing storage-origin silent failures as Phase 0b–3 analysis points

Can the QoS burst defaults on FSx for ONTAP be changed from ONTAP 9.19.1, the same as on-premises? I checked on a live file system

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Tue, 20 Oct 2026 11:04:52 +0000

Conclusion first: on Amazon FSx for NetApp ONTAP you can change the QoS burst defaults. Not only is the configuration accepted — a brief overshoot above the ceiling (the burst) happened exactly as configured. The catch: you cannot change this setting from the AWS Management Console or the AWS CLI (the AWS API). FSx for ONTAP has two ways in — the AWS-side consoles and APIs, and a separate path that drives the storage engine (ONTAP) directly through its own REST API / CLI. The QoS burst can be set only from that second path.

What this post covers, and what it does not

A colleague at NetApp asked me, "since ONTAP 9.19.1 you can change the QoS burst defaults (burst-percent / burst-duration / burst-iops), but is that possible on FSx for ONTAP too?" — so I tried it.

This post records what I measured on a verification FSx for ONTAP file system in ap-northeast-1 (ONTAP 9.19.1P2): setting the options, and confirming that a burst above the ceiling occurs as configured.

Who benefits from this setting

When several workloads or tenants share one file system, one workload can eat the bandwidth and slow the others — the "noisy neighbor" problem. A QoS ceiling holds that down, and burst is the buffer that lets a workload briefly exceed the ceiling at startup or cutover. Being able to change the burst defaults on 9.19.1 means you can tune how the ceiling is exceeded per use case.

On AWS, here is who it helps. If you run databases like Oracle / SQL Server / SAP on Amazon EC2 with their data on FSx for ONTAP, you can cap the nightly batch and reporting side to leave bandwidth for daytime transaction processing (OLTP), while absorbing the database-startup spike with burst. If you operate virtual desktops (VDI, where employees' Windows sessions run on servers rather than their own PCs), you can absorb the morning login storm with burst and keep fairness with a per-group ceiling. If you run multi-tenant SaaS on one file system, you can set ceilings at each level of the storage hierarchy (SVM → volume → qtree, a set of nested units) and vary the burst per tenant to express different service levels (SLA tiers). Each is the same move: keep the ceiling itself fixed, change only how it is exceeded, per tenant or workload.

The use-case mappings and how to choose between QoS and the alternatives (application-layer rate limiting, separate file systems, separate throughput capacity) are organized in the companion note on GitHub (Which workloads the QoS ceiling and burst help). What follows is the premise behind all of that: whether FSx for ONTAP even lets you change the defaults.

What it does not cover: whether a downgrade (revert) is operationally possible on FSx for ONTAP (unverified); the effect of burst values on overall workload cost; interaction with other features such as SnapMirror.

What changed from the previous post

Earlier I measured the speed and monthly cost of using S3 as files versus FSx for ONTAP (I measured the speed and monthly cost of Amazon S3 Files). That time I measured the one-client speed limit and how it scales with client count. This time it is the same FSx for ONTAP, but the question is whether the defaults for how a QoS ceiling is exceeded (burst) can be changed on 9.19.1. Not the ceiling itself, but the design lever for how it is exceeded.

What QoS burst is

A QoS throughput ceiling caps a workload's throughput at a maximum (in IOPS — operations per second — or MB/s). Even when other workloads are competing for the same storage, it guarantees the workload does not exceed the cap. It is used to hold down a noisy neighbor.

But the ceiling is not a hard wall: to respond to short spikes, by default a workload may exceed the ceiling by 50% for 1 second. This absorbs temporary resource demands — starting a database, booting many VMs — without permanently adding capacity.

Since ONTAP 9.19.1, you can change this "50% / 1 second" default. There are three options.

When I measured it, with the ceiling at 2000 IOPS, the default (burst-percent 50 / duration 1) reached about 3000 IOPS (+50%) for the first second only and settled to 2000 by the second second. Setting burst-percent to 25 shrank the first peak to about 2500 (+25%), and setting burst-duration to 5 held about 3000 for 5 seconds. The width, duration, and absolute magnitude of the excess moved exactly as configured.

xychart-beta
    title "Burst behavior at ceiling=2000 IOPS (measured, 4k random read)"
    x-axis "seconds" [1, 2, 3, 4, 5, 6]
    y-axis "IOPS" 0 --> 3500
    line "default 50% / 1s" [3017, 2000, 2000, 2000, 2000, 2000]
    line "25% / 1s" [2507, 2000, 2000, 2000, 2000, 2000]
    line "50% / 5s" [3038, 2970, 3032, 2997, 2970, 2000]

Option	Default	Meaning
burst-percent	50%	the percentage over the ceiling (the peak excess width)
burst-duration	1 second	how many seconds the excess is sustained
burst-iops	0	the absolute IOPS cap during the excess (0 means unused)

burst-percent and burst-iops are mutually exclusive: you set the excess either as a percentage or as an absolute IOPS value.

Checking on a live file system

I reached a verification FSx for ONTAP file system in ap-northeast-1 (ONTAP 9.19.1P2, single-AZ) from a Linux client inside the VPC, using the commands and API that drive the storage engine (ONTAP) directly. That management connection is only reachable from inside the VPC, so I went through a bastion host placed in the VPC.

First, does the configuration go through? I created a QoS policy group with a 2000 IOPS ceiling, set burst-percent to 25, set burst-duration to 5. Both create and modify succeeded, and reading the values back confirmed them.

Then the real question: does changing the setting actually change the read/write behavior? With a load-testing tool (fio) I drove 4 KB random reads well past the 2000 IOPS ceiling (without QoS it does 24,400 IOPS) and observed IOPS at one-second resolution. Burst only kicks in once enough "credit" has built up, so I left an idle period before the load to accumulate it.

Setting	t=1s peak	Steady	What happened
default (percent=50, duration=1)	3017	2000	exceeded ceiling by +50% for 1 second, settled to 2000 by the 2nd
percent=25, duration=1	2507	2000	the peak excess shrank to +25%
percent=50, duration=5	~3000 held for 5s	2000	the excess lasted 5 seconds
burst-iops=3000, duration=5	~3000 held for 5s	2000	the excess IOPS pinned to the absolute value 3000
disabled (0/0/0)	2020 (~+1%)	2000	the excess all but disappeared

burst-percent governed the peak excess width, burst-duration the number of seconds the excess lasted, and burst-iops the absolute IOPS during the excess. Setting 0/0/0 removes the burst entirely. Which aspect of the behavior moves is cleanly separated by which option you set.

The NetApp documentation states that a ceiling may be exceeded by up to 10%, and by up to 50% to handle bursts. The default +50% peak and the ~+1% when disabled match that.

The errors along the way were not FSx constraints

Two errors came up, and both are ONTAP's own constraints, not something FSx for ONTAP blocks.

burst-iops and burst-percent cannot be specified together — they are mutually exclusive. Reset one to 0, then set the other, and it goes through.
burst-iops must be larger than the ceiling IOPS. Against a 2000 IOPS ceiling, burst-iops=500 was rejected and 3000 was accepted. The command reference also says that if burst-iops is below max-throughput, max-throughput is used.

Separating "FSx for ONTAP cannot do this" from "this is how ONTAP behaves" matters. A managed service can hide or restrict settings the underlying product exposes, so when something is rejected, unless you isolate which layer rejected it, you misattribute the cause.

Which way in lets you change it, and which does not

I only checked via the storage engine's own commands and API (ONTAP REST / CLI). On that path the burst defaults can be changed, and the change shows up in real read/write behavior.

The AWS Management Console and the AWS CLI (the AWS API) have no item for these burst options. As noted at the top, FSx for ONTAP has two ways in — the AWS-side consoles and APIs, and the path that drives the storage engine (ONTAP) directly — and QoS settings live on the second one. So the practical takeaway: looking in the usual AWS console will not find it; you set it on the storage-engine side.

Architecture and operational caveats

The intro covered who benefits. Here is the same setting as one picture of how you actually run it on AWS, with the points that trip you up in operation. The figure shows the flow from the workload to FSx for ONTAP, plus monitoring and automated configuration.

As the figure shows, setting QoS itself rides the path that drives the storage engine (ONTAP) — in the diagram, called from AWS Lambda / Amazon EventBridge. Setting the ceiling and burst by hand every time you add a tenant is not realistic, so you automate it as part of provisioning (the process that stands up each new environment). Per the two-ways-in split from earlier, this path is the storage-engine side; it is not on the AWS API side.

Four caveats I hit, or expect to hit, with this shape:

1. There is no burst-specific metric to monitor. What Amazon CloudWatch gives you is the throughput time series; there is no value that directly says "bursting now." Whether you are pinned at the ceiling or briefly exceeded it is something you read off the per-second series yourself. In this verification, the burst peak was invisible at anything coarser than one-second resolution — it averages away.

The finest unit the standard CloudWatch metrics can target is the volume (VolumeId), with the file system (FileSystemId) above it; there is no breakdown (dimension) that identifies a QoS policy group or workload, so from the standard metrics you cannot tell which policy group a number belongs to. If the unit you want to monitor is the volume or the file system, the standard metrics are enough — but when you want it per policy group or per workload, you close that gap yourself. ONTAP has a facility that exposes performance figures (its counter manager), reachable over the REST API, and a small Lambda can read it on a schedule and write per-workload FSxONTAP/QoS metrics into CloudWatch. I added exactly that collector to the Observability-integrations repo and verified the collection mechanism against the same live 9.19.1P2 file system. Two honest limits stay. First, this facility returns running totals (numbers that only ever grow since boot), so a single read does not tell you "how many per second"; the collector reads twice in one run and divides the difference by the elapsed time to get a per-second value (an average over a short window, not a long one). Second, there is still no burst-specific figure, so these metrics show effective throughput and IOPS and "pinned at the ceiling" — "currently bursting" remains inferred, not measured.

2. When ceilings are nested, the real overshoot is bounded by the tightest one. For multi-tenant SaaS where you set ceilings at each level of the storage hierarchy (SVM → volume → qtree), wherever you set burst, it only works up to the strictest ceiling. A generous burst on a lower-level qtree still tops out at a low SVM ceiling above it. Unless you manage which value sits at which layer consistently, you do not get the service-level (SLA) tiers you intended.

3. Ingest paths (AWS DataSync / SnapMirror) can crowd out production reads. Capping the migration/replication ingest keeps it from eating production reads at peak. One pattern is to relax it with burst only during catch-up windows. This is a design expectation, though — I did not measure specific burst values for ingest workloads.

4. ONTAP QoS burst and FSx-for-ONTAP-side burst are different layers. FSx for ONTAP itself also bursts above a baseline for a time (performance documentation). QoS burst governs "how far a workload may exceed its ceiling"; FSx-side burst governs "how much the file system itself can deliver" — different layers. In this measurement the peak was ~3,000 IOPS (~12 MB/s) against ~24,400 IOPS available without QoS, a large margin, so FSx-side bursting did not affect the result (this part I confirmed by measurement). As a design caution, though: if you size so the sum of per-workload (ceiling + burst) exceeds the file system's baseline (sustained) performance, a workload might not even reach its ceiling once FSx-side credits are exhausted, and the same QoS settings can behave differently depending on credit state (I did not measure this saturated case). Size the sum of ceilings to the baseline performance, not the burst.

Caveats 2 and 3, and the per-workload mappings from the intro, are cases where each service's role is real but the combination with QoS is my design expectation. What I measured is the first half: that the defaults can be changed and are reflected in real I/O. The detailed use-case mappings are in the GitHub note.

QoS is not the only way to cap throughput

Isolating workloads and capping throughput is not QoS alone. You can rate-limit at the application layer, separate the file systems themselves, or allocate throughput capacity separately — each with its own fit.

Roughly, it splits on "is the storage layer enough, or do you need to see the meaning of the requests?" If you want to control by request type or priority, choose the application layer; if you only want to suppress storage contention, choose QoS. If you need full physical isolation between tenants, separate file systems; if you need guaranteed bandwidth, separate capacity. QoS is not mutually exclusive with these — they can be combined. None is always right; you pick by requirement.

Summary

On FSx for ONTAP 9.19.1P2 the QoS burst defaults can be changed. Confirmed in both configuration and real read/write behavior.
You change it from the storage-engine (ONTAP) side. The AWS Management Console and AWS CLI (the AWS API) have no matching item.
burst-percent governs the peak excess width, burst-duration the sustained seconds, burst-iops the absolute IOPS during the excess.
The mutual exclusion and the value floor are ONTAP's own constraints, not ones FSx for ONTAP adds.

All the measured values are on GitHub with their environment (the FSx for ONTAP adoption playbook, performance domain). I hope this helps anyone who has set a QoS ceiling and wants to shape how it is exceeded to their own requirements.

Until next time.

Yoshiki Fujiwara

Seeing and restoring a RAM-shared logically air-gapped vault from another account and Region — measured

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Tue, 20 Oct 2026 10:17:27 +0000

The earlier post copied an Amazon FSx for NetApp ONTAP (hereafter FSx for ONTAP) backup from Tokyo to Osaka and restored it, all inside one account. In September 2026, AWS Backup added logically air-gapped vault support for FSx for ONTAP, and trying that is what prompted this post. It picks up where the earlier one stopped: an AWS Backup logically air-gapped (LAG) vault owned by another account, shared over AWS RAM so the recipient can see it, and restored into the recipient's own file system. The earlier post was a same-account cross-Region copy. This one is about visibility and restore across the account boundary.

Up front:

A LAG vault owned by one account becomes visible to another account over AWS RAM (V1). The recipient's --by-shared lists exactly one vault, its ARN account is the owner, and VaultType is LOGICALLY_AIR_GAPPED_BACKUP_VAULT. It is visible in both Tokyo (ap-northeast-1) and Osaka (ap-northeast-3).
The share is one-directional. The owner's own --by-shared is empty (V2) — shared-with-me is not shared-by-me.
The recipient must name the owner's account. A list-recovery-points call without --backup-vault-account-id is AccessDeniedException; with it, it succeeds and returns one recovery point (V4).
The portal visualizes, the CLI restores. With the shared-vault ARN in its IAM policy the portal shows the air-gap chip and the owner account (V3, V5), but the restore dialog's confirm button stays disabled — the portal does not start the restore.
Revoking the share drops the vault within 1–2 seconds in both Regions (V6); later calls return AccessDeniedException. An already-restored volume is untouched.
The shared recovery point restored into the recipient's existing file system, job COMPLETED in 1362 s (~22.7 min), the restored volume reached CREATED (1 GiB, junction /vol_restored) (V7).

Every figure is n=1, one environment, one run: 2026-10-12 (JST), Tokyo primary + Osaka secondary, SINGLE_AZ_1 128 MBps 1024 GiB, aws-cli 2.36.5. Not a generalization to another environment or scale.

Scope: visibility of a shared LAG vault from a recipient account across two Regions (V1, V2, V4), the portal's read path (V3, V5), revocation (V6), and a restore from the shared recovery point into the recipient's existing file system (V7).

Out of scope: multi-party approval (MPA — design only, not run, because it needs two AWS Organizations and a 3–20 person approval team that a single-org two-account lab cannot provide); the organization-sharing path (E13 — this environment used the invitation path with organization sharing off); a cross-Region restore into a recipient file system in that Region (E6 — this run kept to visibility and the same-path restore).

Shape of the setup

Layer	Owner account	Recipient account
Vault	LAG vault (compliance-mode lock, 1-week retention) + the RAM share	—
Visibility	`--by-shared` is empty (shares out, does not share in)	`--by-shared` lists the owner's one LAG vault
Restore	—	restores the shared recovery point into an existing file system → new volume
Revocation	revokes the RAM share	the vault leaves `--by-shared` in 1–2 s; the restored volume stays

The point is that the vault lives in the owner account and the recipient only sees and restores it. The owner holds the air-gapped copy; the recipient never owns the vault, only the volume it restores.

On vendor neutrality: this post evaluates the shared-LAG-vault path on technical grounds only. SnapMirror and the backup-copy / shared-vault paths each suit a different context, and neither is framed as better than the other. The recommended option's own constraints are stated alongside the others.

Who it's for

You want a tamper-resistant retention layer in a separate account from production
You want backup copies to land outside your own account
You are isolating the owning and the restoring account against ransomware or credential compromise
You want to know which operations the portal handles and which the CLI handles

What changed from the backup-copies post — where it is seen from

The earlier post was about where a backup can live: both source and destination inside one account. This post is about an owner's LAG vault being shared to another account, and the recipient seeing it and restoring it — across the account boundary.

	Earlier post	This post
What moved	a backup, cross-Region, same account	a vault, made visible cross-account over RAM
Who puts	the owner (same account as the restore)	the owner
Who sees / restores	the same account	a different account (the recipient)
Direction	n/a	owner → recipient, one-way; revoke is the owner's call

Blur the account boundary and the runbook comes out wrong: the recipient cannot list recovery points without naming the owner account, and the owner can revoke visibility at any time while the recipient's restored volume stays put.

Test environment

Item	Value
Date	2026-10-12 (JST)
Runs	One per item (n=1). Variance not measured
Regions	Primary `ap-northeast-1` (Tokyo), second Region `ap-northeast-3` (Osaka)
Config	`SINGLE_AZ_1`, 128 MBps, 1024 GiB
AWS CLI	`aws-cli/2.36.5`
Accounts	Single organization, two accounts (owner and recipient). Organization sharing off; shared via the invitation path
Vault type	`LOGICALLY_AIR_GAPPED_BACKUP_VAULT`, compliance-mode lock, 1-week retention
Interface	The lab's own scripts and templates, plus the deployed portal
Portal dummy account	`111122223333` (account IDs in the screenshots are masked)

Do not use these n=1 durations as an RTO basis for your environment. They are measured against a 1 GiB volume with one recovery point, where fixed overhead dominates.

What I measured (V1–V7)

One shared vault, seen from the recipient (V1, V2)

Running --by-shared in the recipient account returned exactly one LAG vault shared by the owner. Its ARN account is the owner, and VaultType is LOGICALLY_AIR_GAPPED_BACKUP_VAULT. 18 s.

Two quirks of the list: the by-shared row has no VaultState and shows NumberOfRecoveryPoints 0. The count populates on the vault detail, not on the list.

The recipient's detail view shows the vault type (logically air-gapped), the compliance-mode lock, a one-week retention, and one recovery point.

The same shared vault and recovery point were visible to the recipient in Osaka, the second Region. The share holds in both Regions.

The direction checks out too (V2): the owner's own --by-shared is empty in both Regions. Shared-with-me and shared-by-me are different lists, and this vault appears for the recipient, not the owner.

The owner side shows a shared status (V5)

Seen from the owner, the same vault carries a "shared" status and the RAM-share row to the recipient account.

The copy jobs — from the standard vault into the Tokyo and Osaka LAG vaults — both completed. Which Region's list a cross-Region copy job lands in is in a later section.

A call without the account ID is denied (V4)

When the recipient lists recovery points without --backup-vault-account-id, it is AccessDeniedException. With the owner's account ID it succeeds and returns one recovery point. The ARN shape is arn:aws:backup:<region>:<owner>:recovery-point:<id>. Being shared is not the same as being readable in the recipient's plain context — the owner account has to be named. 18 s.

The portal visualizes; it does not start the restore (V3, V5)

Tested with the shared-vault ARN added to the deployed portal's role. With it added, listSharedBackupVaults succeeds without an error and the screen shows the air-gap chip and the owner account.

This is the design line. The restore dialog opens, but its confirm button stays disabled, and the portal does not initiate the restore. The restore is the job of the CLI script (08); the portal handles visualization and selection. The arn-present spec passed both of its tests. 41 s.

Revoking the share drops it in 1–2 seconds (V6)

Revoking the share dropped the vault from the recipient's --by-shared list in about 1–2 seconds in both Regions. Later list-recovery-points calls return AccessDeniedException. A volume already restored before the revoke is unaffected — visibility and the data materialized on the recipient side are different things. 2 s.

Restore from the shared vault into the recipient's existing file system (V7)

The shared vault's recovery point restored into the recipient's existing file system. The job reached COMPLETED in 1362 s (~22.7 min). The restored volume vol_restored reached CREATED — 1 GiB, junction /vol_restored. The restore role was accepted and the Project tag was applied.

So the full path holds: a LAG vault owned by another account, seen by the recipient, and restored on the recipient side.

Constraints and out-of-scope

Stated as scope, not as defects.

MPA is design-only. The template ships an MpaApprovalTeamArn parameter (default empty), the docs mark it unverified, and it is tracked as a follow-up issue. It is not run here because MPA needs two AWS Organizations and a 3–20 person approval team, which a single-org two-account lab cannot satisfy.
The organization-sharing path (E13) is not recorded, because this environment used the invitation path with organization sharing off.
A cross-Region restore into a recipient file system in that Region (E6) was not run — this run kept to visibility and the same-path restore.

Where a cross-Region copy job appears

One manual observation, separate from the scripted V-items (observed once). The cross-Region copy job (Tokyo source → Osaka LAG vault) appeared in the source Region's list-copy-jobs, carrying the destination vault ARN in ap-northeast-3. It did not appear in the destination Region's list. If you watch only the destination side for progress, you will not find it — that is the operational catch. Both copies' recovery points share the same delete date (source-backup creation time + 7 days), cross-Region included.

Choosing between this and SnapMirror

The shared-vault path and SnapMirror carry different things. This is not a ranking.

	SnapMirror	Shared LAG vault
What sits at the destination	a `DP` volume, always present	a recovery point, no volume until you restore
Account boundary	same account (cluster peering)	owner and recipient can be different accounts
RPO	down to 5 min	the backup interval (guidance 60 min)
To use it	break and promote	restore → a new volume (22.7 min, n=1, 1 GiB)
Who starts the restore	—	the CLI; the portal visualizes only
Failback path	`snapmirror resync` the other way	none

Stated symmetrically, including the recommended option's constraints: the shared-vault path takes 22.7 min to restore (n=1, 1 GiB) and its restore does not complete from the portal; SnapMirror needs cluster peering and a destination file system standing at all times. Running both is coherent — SnapMirror for availability, the shared LAG vault for a tamper-resistant retention layer in a separate account.

What's next

Each of these needs a real environment or a different org/account shape than this run had, so they are tracked as follow-up issues rather than folded in here. The plan is to cover them in later posts in this series.

Measure the cross-Region restore (E6) into a recipient file system in that Region. This run kept to visibility and the same-path restore. Tracked in issue #479.
Measure the organization-sharing path (E13). This run used the invitation path only. Tracked in issue #479.
Run MPA on a two-org, approval-team setup. It is design-only here. Tracked in issue #471.

The most useful thing I learned was smaller than the feature: being shared and having the data materialized on the recipient side are different states. Revoking the share removes visibility — an owner-side action — but a restored volume belongs to the recipient and stays.

Thanks for reading.

Earlier post: AWS Backup now copies FSx for ONTAP backups across Regions and accounts — measured Tokyo to Osaka

Japanese version (はてなブログ): RAM で共有した論理エアギャップボールトを、別アカウント・別リージョンから見て復元する

Repository

The lab that produced these screenshots — the CloudFormation templates, the scripts, and the verification record — is here: infrastructure/shared-backup-vault-lab.

Resources

What's New: AWS Backup adds logically air-gapped vault support for Amazon FSx for NetApp ONTAP — the release this post verifies
AWS Backup: Logically air-gapped vault
AWS Backup: Creating backup copies across AWS accounts
AWS RAM: Sharing your resources
AWS Backup: ListBackupVaults / --by-shared
AWS Backup: Multi-party approval (MPA)
AWS: Restoring an FSx for ONTAP backup to a new volume
AWS Storage Blog: Inside the AWS Storage Assessment — how intelligent analysis replaces guesswork — related for planning: its workload classification and data-protection cost modeling inform the standing-file-system vs. backup-only cost trade-off discussed above

The file systems, SVMs, volumes, standard vault, and restored volume used for this test have been deleted and billing is stopped. The logically air-gapped vaults, under a compliance-mode lock, cannot be deleted until their recovery points' retention (source-backup creation time + 7 days) expires, and are deleted after that. The figures are measurements from one specific environment and configuration (n=1), and will differ with data volume and setup.

I measured the speed and monthly cost of Amazon S3 Files, which lets you use S3 as files (and how it differs from FSx for ONTAP)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Fri, 16 Oct 2026 11:20:30 +0000

Conclusion first: the one-client speed limit, scaling with client count, and a monthly cost order that splits by object size

Amazon S3 Files lets you open, read, and write the data in S3 like an ordinary folder. With plain S3, a program has to call S3-specific APIs. With S3 Files, tools built to open files can work on your S3 data as they are.

This post is a record of measurements and estimates for people who do not run storage performance tests as a routine job, to help decide whether S3 Files belongs in a design. I built the environment myself in ap-northeast-1 and measured speed on 2026-10-10. The monthly cost is not a measurement. It is arithmetic from list prices read on 2026-08-09. When I re-read the 2026-08-09 record with the Price List API on 2026-10-16, the values were the same (it covers 10 items and does not include the two PUT request prices).

Three things came out of it.

One server has a read speed limit. On a c5n.9xlarge (a 36 vCPU server) it was about 1.1 GB per second (1,104.9 MB/s). It stops when the relay program running on the server (efs-proxy) uses up one CPU core (96.8 to 97.9%). One run.
Adding servers raises the total. Going from 1 to 8 c5n.2xlarge servers raised the total from 784.4 MB/s to 6,282.6 MB/s (8.01x). All servers read the same 600 GiB file.
Under this model's base assumptions, the monthly cost order against FSx for ONTAP splits by object size. Amazon FSx for NetApp ONTAP (FSx for ONTAP below) is an AWS managed service that provides NFS and SMB shared folders, and with S3 Access Points attached it also serves the same data through the S3 API. I fixed the stored volume at 20 TiB and estimated five patterns that differ in monthly read and write volume (P1, P4, P5, P6, P8; the contents are in the table in the chapter "Monthly cost estimate"). At 64 KiB, FSx for ONTAP was cheaper in all five ($1,783.11 to $2,218.13 a month, against $7,928.32 to $12,030.81 via S3 Files). At 4 MiB, S3 Files was cheaper in all five ($512.14 to $543.52 a month, against $1,781.28 to $1,788.08 on FSx for ONTAP).

The monthly amounts are subtotals of the main line items for ap-northeast-1, with prices read on 2026-08-09. A main-line-item subtotal is the sum of storage, S3 Files high-performance storage (a low-latency layer inside the file system; you pay storage charges for what it holds), fixed bandwidth, requests, and so on, and it leaves out any item that is not in the tables. They are price x quantity arithmetic, not measurements, and they are not invoices. The FSx for ONTAP capacity pool (the tier that holds data that does not fit on SSD) read request count is an assumed value that I have not confirmed. The P5 write volume and the read and write volumes of P1, P6, and P8 are placeholders with no source.

Because the order was the same in all five patterns, what splits the order in this model is the object size, not the read and write volume. The candidate for the dividing line is the S3 Files import threshold (the file-size boundary for importing into high-performance storage; default 128 KiB). I estimated only two sizes, 64 KiB and 4 MiB, and I did not estimate the sizes at the boundary.

The takeaways: when one server is not fast enough, add servers (confirmed up to 8). Under the base assumptions, FSx for ONTAP was cheaper at 64 KiB and S3 Files was cheaper at 4 MiB. The order can flip when the assumptions change, so replace them with your own values and check. The assumptions to replace are the share of data on the capacity pool, the storage efficiency of the SSD tier, the active share of S3 Files high-performance storage, the S3 Files import threshold, and the FSx for ONTAP deployment. The table of values at which the order flips is in the section "Baseline assumptions and sensitivity" of the generated document in the repository (the link is in the "Assumptions under which the order can flip" section of the chapter "Monthly cost estimate"). This post gives only the directions. The estimate itself is in the chapter "Monthly cost estimate".

How to read this post by goal

You only want the conclusion: this chapter is enough.
You want to know whether it works in your setup: "The value of S3 Files and four prerequisites"
The mount does not proceed: "A hung mount"
You want to know how it differs from FSx for ONTAP: "Comparing two ways to use the same data through two entry points"
You want the cost: "Monthly cost estimate"
You want to know which setup to consider first: "Choosing between S3 bucket first and file system first" at the end of the "Monthly cost estimate" chapter, and "Closing"
You want the measurement conditions and limits: "The single-mount ceiling", "An aggregate that scales with client count", "An invalid warm / cold comparison", "Status of the four stated values", "What I have not measured"
You want to reproduce it: "Reproducing it yourself"

Introduction

There are times when you want to handle S3 data as files: an existing script that opens the files in a folder one by one, or a business tool built around a shared folder. Amazon S3 Files became generally available in April 2026, and it lets you mount an S3 bucket as an NFS file system (the network shared-folder mechanism of Linux) without migrating data.

Before using it, I think you wonder about three things.

What is it good for, in the end?
How fast is it, and what does it cost?
How does it differ from a file-server type of service (FSx for ONTAP)?

This post answers the second and third. The speeds are values I measured in my own environment, and the monthly cost is a unit price x quantity estimate. For the differences from FSx for ONTAP, I put my past measurement records (the environment differs per measurement) and the official documentation side by side in a table. I do not cover the first, the introduction. The Classmethod article (Japanese) compares I/O performance and sync lag with Amazon EFS, and the Serverworks article (Japanese) sorts out five services, including Mountpoint for Amazon S3.

In the previous post (S3 Burst Part 2: "The throughput ceilings of three paths") I compared three paths, S3 Files among them, side by side and asked where each one tops out. This time I narrow down to S3 Files alone. Last time the question was which path tops out where. This time it is how S3 Files behaves as a file system.

The mount failed before I could measure anything, so I recorded the cause first. Someone else can hit the same wall. After reading, you can decide, on numbers that come with their environments, whether S3 Files is a candidate, how far the number of servers can carry your speed, and which of S3 Files and FSx for ONTAP with S3 Access Points (the S3 API entry point that you attach to a volume) fits better.

What I cover / do not cover

Covered

The prerequisites of S3 Files and the value AWS states for it. Prerequisites decide first whether you can use it at all.
Why the mount failed. It was missing permissions, not the network, and knowing in advance avoids it.
The single-mount ceiling (c5n.9xlarge). The per-client ceiling is the starting point of capacity planning.
Aggregate growth with client count (c5n.2xlarge, 1 to 8 clients). It shows how far clients can carry you past one mount's limit.
The comparison with FSx for ONTAP S3 Access Points. Both give you the same data through files and the S3 API, but the source of truth, the speed of propagation, the growth with client count, and the billing items differ.
The monthly cost estimate. With the stored volume fixed at 20 TiB, five read and write patterns, and two object sizes, I line up the main-line-item subtotals of S3 Files and FSx for ONTAP. It separates this model's result, in which the size splits the order, from the assumptions under which the order can flip.
The accounting assumption for S3 Files metadata charges, and the reconciliation of usage against the bill. It separates what the reconciliation showed from the fact that the invoiced amount itself is unconfirmed.
Reading data on the capacity pool. It separates the existence of billing items from how far a latency observation can be read.
How my warm / cold comparison failed. Warm means reading data that sits in S3 Files' high-performance storage; cold means reading straight from the bucket. Someone repeating the measurement can fall into the same trap.
Which of the four values AWS states I could confirm, and which I could not. It separates what is safe to base an adoption on.
What I have not measured. It lets you judge where the numbers apply.
How to reproduce it in your environment. It points to the runbooks so you can follow the measurements.

Not covered

An introduction to S3 Files, and how it differs from Mountpoint for Amazon S3 and Amazon EFS. Articles already cover that, and it is outside this post's subject, measured speed and monthly cost.
Which configuration is faster. The paths were measured in different environments (NFS version, instance type, configuration), so no same-conditions winner exists.
Measurements that separate the propagation conditions. The window of about 60 seconds between writing a file and reading it through the S3 API appears as two rows in the comparison table of "Comparing two ways to use the same data through two entry points", and I explain it in four stages. Measurements that separate conditions, such as long continuous appends and concurrent updates, exist only as a plan and have no measured results. The conditions of that window are in another verification record (Japanese). It is outside this post's subject, which is throughput and client count, and that measurement covers a single condition.
An explanation of the values at which the order flips. The table of values is in the generated document, and this post gives only the directions, because the values have to be read with the sourced ranges and the mechanical search ranges kept apart. I did not estimate the boundary object size (around 128 KiB).
Invoiced amounts. The monthly cost is a unit price x quantity estimate, and I have not confirmed that it matches a bill.
For the warm / cold difference, which path a first read takes (straight from the bucket, or from high-performance storage after import). I re-measured the latency with a different design, but DataReadBytes was the same for the first and second reads, so I cannot distinguish the path. The values I measured are in the chapter "An invalid warm / cold comparison" (verification record).
The tiering difference from a client outside AWS, and from a standby-AZ client. I measured Single-AZ and the preferred AZ of Multi-AZ first generation from an EC2 instance inside a VPC (verification record). The chapter "What I have not measured" lists the remaining scope in a table.

Terms used in this post

Each term is explained at first use in the text. This table is an index for rereading.

Term	Meaning	Where it appears
S3 Files	A service that mounts an S3 bucket as an NFS file system. Built on Amazon EFS; the bucket remains the source of truth	All chapters
High-performance storage	The low-latency layer inside the S3 Files file system. It holds the data and metadata of files in active use, and you pay storage charges for what it holds	"The value of S3 Files and four prerequisites", "Comparing two ways to use the same data through two entry points", "Monthly cost estimate", "An invalid warm / cold comparison"
efs-proxy	A local proxy that runs on the mounting client's EC2 instance. The mount target is 127.0.0.1, and in the verification records this one process handles TLS termination and NFS translation	"The value of S3 Files and four prerequisites", "A hung mount", "The single-mount ceiling", "An aggregate that scales with client count", "Comparing two ways to use the same data through two entry points"
Mount target	The NFS endpoint of the file system that you create in a VPC (port 2049). One per AZ at most	"The value of S3 Files and four prerequisites", "A hung mount"
nconnect	A Linux NFS mount option that opens several TCP connections to one server. S3 Files does not support it, and specifying it hung the mount	"The value of S3 Files and four prerequisites", "The single-mount ceiling", "Comparing two ways to use the same data through two entry points"
warm / cold	In this post, used only for S3 Files reads. Warm is a read of data held in high-performance storage; cold is a read straight from the bucket. For FSx for ONTAP I write "on SSD" and "on the capacity pool"	"An invalid warm / cold comparison", "Status of the four stated values", "What I have not measured"
Capacity pool	The tier where FSx for ONTAP keeps data that does not fit on SSD. Its storage price is lower than SSD (Single-AZ, ap-northeast-1: $0.0238 vs. $0.15 per GB-month, read 2026-08-09)	"Comparing two ways to use the same data through two entry points", "Monthly cost estimate", "What I have not measured"
Tiering All	One of the tiering policies of an FSx for ONTAP volume. It places the data on the capacity pool	"Comparing two ways to use the same data through two entry points", "Monthly cost estimate"
FlexCache / Origin	FlexCache is an ONTAP feature in which a Cache volume holds only the data read from the Origin volume that owns the original data. This post uses it only to say that the FSx for ONTAP propagation measurement read the Origin volume directly	"Comparing two ways to use the same data through two entry points"
S3 Access Points	In this post, the S3 API entry point attached to an FSx for ONTAP volume. The same data can be read and written over NFS / SMB and the S3 API	"Comparing two ways to use the same data through two entry points", "Monthly cost estimate"
Import threshold	The size at which S3 Files decides whether to import a file into high-performance storage (`sizeLessThan`). The default is 128 KiB, and sizes below it are imported. Exactly 128 KiB is treated as not imported	"The value of S3 Files and four prerequisites", "Monthly cost estimate", "An invalid warm / cold comparison", "Closing"
Pattern (P1, P4, P5, P6, P8)	A name for a combination of monthly read and write volumes. The numbers follow the cost model's numbering, and the figures and tables carry five of them (P2, P3, and P7 are left out because they cost the same as P4 and P6)	"Monthly cost estimate", "Closing"
Working set / working-set ratio	The working set is the part that is read within the month. The working-set ratio is its share of the stored volume (10% for P4)	"Monthly cost estimate"
Churn rate	The share of the stored volume that is rewritten in a month (5% for P5)	"Monthly cost estimate"
Main-line-item subtotal	The sum of storage, S3 Files high-performance storage, fixed bandwidth, requests, and so on. Items not in the tables are not included. It is price x quantity arithmetic, not a measurement, and not an invoice	"Monthly cost estimate", "Closing"
Placeholder	A value with no source, put in to make the arithmetic work. It is not taken from a real workload	"Monthly cost estimate"
VDBENCH	A tool that generates storage I/O load (distributed by Oracle). The S3 Files measurements here used 5.04.07	"The single-mount ceiling", "An aggregate that scales with client count"
o_direct	A setting that reads and writes without the OS page cache. In VDBENCH it is `openflag=o_direct`	"The single-mount ceiling", "An aggregate that scales with client count"
disjoint / same	Disjoint: each client reads a non-overlapping range. Same: all clients read the same file	"An aggregate that scales with client count"
shared epoch	All clients wait until the same UNIX time (90 seconds after launch) and start measuring together	"An aggregate that scales with client count"

The value of S3 Files and four prerequisites

S3 Files lets Linux compute mount a bucket as an NFS file system and read and write it while the bucket stays the source of truth. It is built on Amazon EFS and became generally available in 34 regions in April 2026 (What's New). Four prerequisites decide first whether you can use it.

Four prerequisites

Prerequisite	Detail	Source
Bucket versioning	Required. Encryption is SSE-S3 or SSE-KMS	Prerequisites for S3 Files
Client compute	Linux on AWS (EC2 / Lambda / ECS / EKS / Fargate / Batch). I could not find documentation for mounting from outside AWS	S3 Files product page
Protocol	NFSv4.1 and NFSv4.2 only. NFSv3 and SMB are not listed as supported. nconnect (a Linux mount option that opens several TCP connections to one server) is unsupported; in my verification, specifying it timed out the mount after 90 seconds (one observation, c5n.2xlarge, 2026-09-02)	Unsupported features, limits, and quotas, verification record
File-to-bucket propagation	After you change a file, S3 Files waits about 60 seconds before copying it to the bucket	Performance specifications

How the mount works

When you mount, efs-proxy (the local proxy mentioned in the summary) starts on the client's EC2 instance. Your application mounts 127.0.0.1, and efs-proxy connects to the mount target in the VPC. The mount target is the NFS endpoint of the file system (port 2049), with a limit of one per AZ. In the verification records, this efs-proxy handles TLS termination and NFS translation in one process (verification record, Japanese). This structure sets the single-mount ceiling measured in a later chapter.

Reads take one of two paths. Small files (under 128 KiB by default) are imported asynchronously into high-performance storage, the low-latency layer inside the file system, on first access, and later reads are fast. Reads of 1 MiB or more go straight to the bucket even when the data sits in high-performance storage. The source is Performance specifications.

Values AWS states

The following four points are AWS's statements. I could get numbers for the second point, propagation time between files and the S3 API, and for the fourth point, single-mount throughput and aggregate growth with client count (up to 8). I did not measure the first, and the comparison for the third was invalid.

Existing file-based tools work as they are. The product page says "No code changes, no custom connectors, and no new APIs to learn" (product page).
The same data is usable from both files and the S3 API at the same time. The product page's wording is "nothing to reconcile between two separate systems", meaning you do not have to reconcile two systems.
Only frequently read data sits in high-performance storage, and you pay for what it holds. Reads of 1 MiB or more cost only the standard S3 GET price (product page).
Many clients can use it at once. The limit is 25,000 connections per file system, and the aggregate read throughput is "up to terabytes per second" in the AWS pages' wording (quotas, Performance specifications).

If your requirements include NFSv3 or fixed SMB devices or Windows workflows, S3 Files drops out at the prerequisites table. The AWS documentation does not mention mounting from outside AWS that I could find. If you need it, check in your own environment before you put S3 Files on the candidate list. If none of those apply, read the following chapters on the mount failure, the single-mount ceiling, and client scaling, and use them for capacity planning.

A hung mount: missing CloudWatch permissions on the client's IAM role

mount -t s3files timed out every 15 seconds. The cause was not the network or DNS. The IAM role on the client EC2 instance lacked logs:CreateLogGroup and cloudwatch:PutMetricData. Adding the permissions let the mount complete.

The environment was a c5n.9xlarge (Amazon Linux 2023, amazon-efs-utils 3.3.2) in a dedicated measurement VPC in ap-northeast-1 (a public subnet with an internet gateway). It happened while I prepared the measurement, and my troubleshooting notes carry no date. The throughput measurements in the same environment ran on 2026-10-10.

Symptom: timeouts every 15 seconds with a healthy network

efs-proxy: running, with a TCP connection to port 2049 on the mount target (ESTAB in ss)
Reachability: both S3 on 443 and the service endpoint on 2049 were reachable
Mount: mount.nfs4 127.0.0.1:/ still did not finish

Cause: efs-proxy restarting over and over because log creation failed

The efs-proxy log contained these lines.

WARN  cw_publisher Failed to create log group: AccessDeniedException ... logs:CreateLogGroup
ERROR nfs::nfs_reader Error sending message batch to dispatcher DispatchingFailure
INFO  controller Proxy incarnation restarted, restart_count=3

Before establishing the NFS connection, the mount helper tries to create a CloudWatch Logs log group (/aws/efs/utils). When that call fails for lack of permissions, efs-proxy keeps restarting and the mount never finishes. An AWS article names the same behavior as the root cause when a mount hangs forever without an obvious reason (Mounting S3 Files on EC2 in private subnets).

My subnet has an internet gateway, so CloudWatch Logs was reachable. Permissions were the blocker, not reachability. In a private subnet with no CloudWatch Logs VPC endpoint, unreachability produces the same symptom.

Original record: Mount pitfalls (causes and fixes) (Japanese)

Fixes: any of three options works

#	Fix	Fits when
1	Grant the IAM role `logs:CreateLogGroup`, `logs:CreateLogStream`, `logs:PutLogEvents`, and `cloudwatch:PutMetricData`	You have an internet gateway or CloudWatch endpoints and also want CloudWatch metrics. I used this one
2	Disable CloudWatch logging in the mount helper (`/etc/amazon/efs/s3files-utils.conf`)	You do not want to add permissions or endpoints
3	Create a CloudWatch Logs VPC endpoint (`com.amazonaws.<region>.logs`)	A private subnet with neither an internet gateway nor a NAT

I did not pick option 2 because the next chapter's measurement uses efs-proxy's CloudWatch metrics (connectivity and byte counts) to isolate the cause of the ceiling.

Pre-mount checks: three items

IAM role: it carries the permissions from fix 1.
botocore: installed with dnf install python3-botocore (pip3 does not work). Without it, the mount either stopped with access denied by server while mounting, or went through but CloudWatch metrics were unavailable. The two verification records show different symptoms, and neither records the condition that decides which one you get.
Mount target: it is in the same AZ as the client, and the security group allows 2049 from the client.

Check these three items before you mount, and decide beforehand whether to grant the CloudWatch permissions or disable logging.

The single-mount ceiling: one efs-proxy core saturated

A single mount stopped at 1,104.9 MB/s for 1 MiB sequential reads (c5n.9xlarge, one run). The limiter was the efs-proxy process's CPU: one core of 36 vCPU was full (96.8 to 97.9%). As noted in the prerequisites chapter, S3 Files does not support nconnect, so no mount option can add connections to efs-proxy and move this ceiling.

Measurement conditions

Environment: ap-northeast-1, 2026-10-10 (UTC), single AZ, c5n.9xlarge (36 vCPU), Amazon Linux 2023
Mount: nfs4 vers=4.2, rsize=wsize=1048576, hard. The server is 127.0.0.1 (through efs-proxy, TLS and IAM authentication always on)
Load: VDBENCH 5.04.07 (an I/O load generator distributed by Oracle), 512 threads, o_direct (bypasses the OS page cache)
Data: a single uncompressible 600 GiB file. 60 seconds of warmup, then 300 seconds measured

Results: four workloads

Workload	Throughput	IOPS
1 MiB sequential read	1,104.9 MB/s	1,105
1 MiB sequential write	852.0 MB/s	852
4 KiB random read	61.9 MB/s	15,839
4 KiB random write	14.4 MB/s	3,688

Each row is one run; I did not repeat any condition. Source of this table: Results (S-1 + S-2) (Japanese)

Isolating the cause: efs-proxy CPU saturated at one core

I re-measured 1 MiB sequential reads for 60 seconds and sampled CPU three times.

Sample	efs-proxy CPU	VDBENCH and java CPU
1	96.8%	near 0
2	97.4%	near 0
3	97.9%	near 0

The re-measured throughput was 1,109.4 MB/s, 0.4% away from the main run's 1,104.9 MB/s. The mount target is a local proxy at 127.0.0.1, and efs-proxy handles TLS termination and NFS translation in one process, so one mount tops out at one core of that process. Source of this table: Ceiling confirmation (Japanese)

Differences by instance type: three values under different conditions

Even on the same c5n.2xlarge the value changes with how I measured, so here are three side by side.

Measurement	Instance	Conditions	Read	Date
Stream-count sweep	c5n.2xlarge	8 streams, 1 GiB per stream, `O_DIRECT`	451.3 MB/s	2026-09-02
One client in the client-count test	c5n.2xlarge	VDBENCH, 64 threads, 1 MiB sequential, 600 GiB file	784.4 MB/s	2026-10-10
Single-mount test	c5n.9xlarge	VDBENCH, 512 threads, 1 MiB sequential, 600 GiB file	1,104.9 MB/s	2026-10-10

The three differ in conditions, so this is not a comparison that changes only the instance type. I have no measurement that varies only the instance type under the same conditions. I also did not isolate whether efs-proxy CPU or the client's network bandwidth sets the roughly 785 MB/s per c5n.2xlarge. Source of this table: Stream-count measurement (Japanese)

When one mount is not enough bandwidth, design for adding clients (the next chapter).

An aggregate that scales with client count: 8.01x at 8 clients

Going from 1 to 8 c5n.2xlarge clients raised the aggregate read from 784.4 MB/s to 6,282.6 MB/s (8.01x). Per client it stayed roughly constant at 784.4 to 802.2 MB/s. That fits the structure in which each client has its own efs-proxy.

Measurement conditions

Environment: ap-northeast-1, single AZ, 2026-10-10 (UTC), up to 8 c5n.2xlarge, Amazon Linux 2023
Mount: each client mounts S3 Files on its own (one efs-proxy per client)
Load: VDBENCH, 1 MiB sequential read, o_direct, fixed at 64 threads per client (only the client count changes)
Simultaneous start: shared epoch. All clients wait until the same UNIX time (90 seconds after launch) and start measuring together. 60 seconds of warmup, then 180 seconds measured
Range read: two patterns, same (all clients read the same 600 GiB file) and disjoint (each client reads a non-overlapping range)

Results: all clients read the same file

Clients	Aggregate	Per client	Ratio to 1 client
1	784.4 MB/s	784.4 MB/s	—
2	1,604.5 MB/s	802.2 MB/s	2.05x
4	3,148.4 MB/s	787.1 MB/s	4.01x
8	6,282.6 MB/s	785.3 MB/s	8.01x

Source of this table: Client-count measurement (same) (Japanese)

Results: each client reads a non-overlapping range

I measured disjoint at 8 clients only. The aggregate was 7,210.0 MB/s, and per client 901.3 MB/s (a spread of 770 to 1,332), higher than the 8-client same result of 6,282.6 MB/s. Each client reads different data, so sharing and fanning out one read cannot explain the number. I did not measure disjoint at 1, 2, or 4 clients, so the shape of disjoint scaling is unconfirmed.

Source of this table: Client-count measurement (disjoint) (Japanese)

Measurement limit: client-side totals only

The numbers are the sum of what the clients reported. I could not reconcile them with CloudWatch DataReadBytes: on the day, metrics arrived sparsely and did not land in the measurement window. The record also has no spread across repeated runs for each client count.

high-performance storage and direct bucket reads. A single mount tops out at one efs-proxy core, and the aggregate grows in proportion to client count up to 8 clients" width="800" height="686">

Figure 1: The internals of S3 Files. Each client has its own efs-proxy. A single mount tops out at one efs-proxy core, while the aggregate grows in proportion to client count (measured up to 8). The figure says the same thing as the tables above.

For capacity planning up to 8 clients, multiply roughly 785 MB/s per client (785.3 MB/s at 8 clients; c5n.2xlarge, all clients reading the same file) by the client count. I have not measured 9 or more. If you have a use case that shares it across 9 or more clients, tell me the conditions on the repository issue form, and I will measure from the conditions people use.

Comparing two ways to use the same data through two entry points: where the source of truth sits

Both S3 Files and FSx for ONTAP with S3 Access Points let you use the same data through a file mount and the S3 API. What differs is the source of truth: an S3 bucket for S3 Files, an ONTAP volume for FSx for ONTAP. That difference bears on the speed of propagation, how throughput grows as you add clients, and how the bill comes out. The measured values do not share an environment, though, so this post does not isolate whether the differences come from where the source of truth sits.

The two setups compared in this chapter are as follows.

S3 Files: attach S3 Files to an S3 bucket and mount it over NFS from EC2. The S3 API reaches the bucket directly.
FSx for ONTAP: use an ONTAP volume over NFS / SMB and attach S3 Access Points (the S3 API entry point attached to the volume) to the same volume. Data that does not fit on SSD can sit on the capacity pool, a tier with a lower storage price. Tiering All, one of the volume's tiering policies, places the data on the capacity pool.

The table below lines up the two setups by aspect. I wrote "not measured" where I did not measure. Measured values come from different environments, so each cell carries its environment.

Aspect	S3 Files (S3 bucket first)	FSx for ONTAP + S3 Access Points (file system first)	Source
Source of truth	The S3 bucket. Objects and files point at the same bucket	The ONTAP volume	Design differences
File-side entry	NFSv4.1 / 4.2 only, through a per-client efs-proxy. nconnect is unsupported	ONTAP serves NFS (v3, v4.0 to 4.2) and SMB directly	Protocol comparison, S3 Files quotas
S3 API-side entry	The bucket itself	An S3 Access Point attached to the volume	Design differences
Propagation: write as a file, read with the S3 API	p50 63,769.3 ms (a window of about 60 seconds). t3.small, NFSv4.2, 64 B, concurrency 1, n=30, 2026-09-01	p50 44 ms. NFSv3, 128 MBps Single-AZ, 64 B, concurrency 1, n=30, 2026-08-09	S3 Files measurement, FSx for ONTAP measurement
Propagation: write with the S3 API, read as a file	p50 1,533.1 ms (same environment as above)	p50 3 ms. A direct NFS read of the Origin volume that received the write (it does not go through a FlexCache Cache volume; same environment as above)	Same as above
Conflict and locking	When the same key is changed from the file side and the S3 API side, the S3 bucket side became the source of truth (3/3 converged on the S3 content). The file-side version was moved to `.s3files-lost+found-<fs-id>`. Only conflicts inside the window (5 seconds later) were measured	Not measured. I did not measure what happens when the same key is changed from the file side and the S3 API side, nor reconcile it with ONTAP-side file locking	Conflict behavior (Japanese)
Supported APIs, conditional writes, copy scope	The S3 bucket API. I did not measure conditional writes or copy scope in this post	Presigned URLs through S3 Access Points (`PutObject` / `HeadObject` / `GetObject`) are verified (2026-08-19, n=30x4, succeeded under both SigV4 and SigV2). The official support table also lists `Presign` as Supported (confirmed 2026-10-08, documented). Uploads are single `PutObject` 5 GiB and whole object 50 GiB, with the limit judged at `CompleteMultipartUpload`. The read side is asymmetric: a GET of 50 GiB + 1 byte succeeded (537 seconds, 2026-09-11)	Verification status (Japanese)
Versioning / Object Lock / Lifecycle	Bucket versioning is required. I did not measure Object Lock or Lifecycle in this post	On the S3 Access Points side, S3 event notifications, lifecycle, and versioning are not supported (documented). ONTAP-side data protection (SnapLock and the like) is a separate mechanism; do not equate it with S3 Object Lock	Design differences (Japanese)
IAM vs. file-side permissions	An object written through the S3 API appears as `root:root` on the file system (file 0644, directory 0755). A client mapped to non-root at the access point cannot write into a directory the S3 API created (reads go through). The official Synchronization page says POSIX permissions are stored in the S3 object's user-defined metadata, and that objects without metadata get default permissions. I have not found the specific values `root:root` or 0644 / 0755 in the pages I read	ONTAP NFS / SMB file permissions and the IAM on the S3 Access Points side are separate paths. I did not run a reconciling measurement in this post	Owner of S3-API writes (Japanese)
Where unread data sits	The S3 bucket. Reads of 1 MiB or more go to the bucket even when the data is in high-performance storage	SSD. Depending on the tiering policy, the capacity pool (Tiering All places everything there)	Performance specifications, Billing items table
Read billing items	S3 GET plus metadata reads ($0.04 / GB; the model counts 4 KiB x operations). Small files held in high-performance storage also incur a data read charge ($0.04 / GB). In two measurements, on 2026-10-16 and 2026-10-17, the Cost Explorer usage for `APN1-Files-Read` matched the sum of CloudWatch `DataReadBytes` and `MetadataReadBytes` to within 0.5% (daily, `Estimated: true`). Because cost is set by usage (bytes), the per-read operation breakdown does not change the subtotal. The invoiced amount is offset to a net zero by credits, so I have not confirmed it as a billed figure	GET through S3 Access Points ($0.000029 / 1,000). Data on the capacity pool can incur a read request charge ($0.00037 / 1,000). The billable requests per GetObject are not confirmed. In a separate environment (2026-10-20, Single-AZ), counting CloudWatch `CapacityPoolReadOperations` gave about 36 per object at 4 MiB, but that is a CloudWatch operation count, not a billable request count, and the correspondence between the two is unconfirmed	Billing items table, capacity pool operation-count measurement
Latency of reading data that is not held in the fast tier	Not confirmed. 64 KiB p50 was 5.33 ms from high-performance storage and 4.05 ms straight from the bucket (c5n.9xlarge, n=30, one run, 2026-10-10). A separate t3.small record shows 10.8 ms vs. 55.0 ms (2026-09-01), which did not reproduce on c5n.9xlarge	Measured from EC2 inside a VPC. For 4 MiB reads that reached the capacity pool, ALL was slower than SSD by a median difference of +122.4 ms (Single-AZ, 1 batch, n=30) and +110.4 ms (preferred AZ of Multi-AZ first generation, 4 batches, 120 pairs) (2026-10-20, ONTAP 9.19.1P3, 128 MBps, concurrency 1). A client outside AWS and a standby AZ are not measured	Three reads, Reads by tier, Tier latency inside a VPC
Adding clients: file path	1 to 8 clients: 784.4 to 6,282.6 MB/s (8.01x). c5n.2xlarge, all clients read the same file, 2026-10-10. 9 or more: not measured	1 to 8 clients: 2,664.58 to 11,916.29 MB/s (4.47x, all clients read the same file). With non-overlapping regions it does not grow, from 2,299.05 MB/s at 2 clients to 2,173.37 MB/s at 8. c5n.2xlarge, nconnect=16, second generation, 2,048 MBps, 2026-09-05	S3 Files client count, FSx for ONTAP client count
Adding clients: S3 API path	The S3 API reaches the bucket directly. Reads from Amazon S3 went from 495.4 to 2,030.5 MB/s at 1 to 4 clients (4.10x, c5n.2xlarge, 2026-09-01 to 02). Not measured on a bucket with S3 Files attached	At 1 to 4 clients, writes went 129.4 to 131.0 MB/s (1.01x) and reads 480.9 to 612.3 MB/s (1.27x). 128 MBps configuration, nearly flat from 2 clients. Clients share the purchased throughput capacity	Measurement record of the previous post
Published limits	25,000 connections per file system, 3 GiB/s read per client. Aggregate read is "up to terabytes per second" in AWS's wording	You buy throughput capacity. The minimum for Single-AZ first generation is 128 MBps	quotas, Performance specifications, Fixed-cost floor
Fixed cost	None. S3 storage plus what you use	Floor of $269.57 a month for Single-AZ first generation (SSD 1,024 GiB + 128 MBps, prices read 2026-08-09)	Fixed-cost floor
Prerequisites and constraints	Bucket versioning required. Clients are Linux on AWS. NFSv4.1 / 4.2 only. About 60 seconds from file to bucket	On the S3 Access Points side, S3 event notifications, lifecycle, and versioning are not supported. Capacity pool read charges. Clients share the throughput capacity	Design differences, Performance specifications

The propagation rows are not a same-conditions comparison, because the NFS version, client, and configuration differ. The client-count rows differ too: S3 Files used VDBENCH with 64 threads per client, and FSx for ONTAP used a second-generation 2,048 MBps configuration with nconnect=16. I read in full on 2026-10-16 the two AWS documents this chapter relies on: the S3 User Guide pages Performance specifications and Understanding how synchronization works.

Source of this table: Design differences table. For the measured rows, the sources are the links in the "Source" column.

The four stages of file-to-bucket propagation, and how to treat the maximum reflection time

The "window of about 60 seconds" in the propagation rows is not one wait but four stages. The measured p50 of 63,769.3 ms is the whole span from (a) to (d).

Stage	Definition	Category
(a) File write complete and persisted	The moment the write to NFS finishes through efs-proxy. Separate from propagation to S3	NFS semantics
(b) No-write wait	The wait counted from when writes to the same file stop. The official Synchronization page states the "60 seconds with no writes" condition	Documented
(c) Export start	The moment a file that satisfies the wait condition becomes an export target. The official Performance page states the roughly 60-second aggregation	Documented
(d) Visible through the S3 API	After the export completes. The measured p50 of 63,769.3 ms is this whole (a) to (d)	Measured (t3.small, 64 B, concurrency 1, n=30, 2026-09-01)

The export side agrees across two sources. Against the client-side max of 66.33 seconds, the CloudWatch ExportAge peak was 66.06 seconds, a 0.27-second difference (all 30 runs fell in 63.8 to 66.3 seconds).

The maximum reflection time cannot be used as a guaranteed value. On the import side (S3 to file), the client-side max of 14.36 seconds and the CloudWatch ImportAge peak of 62.80 seconds do not agree. I could not find a maximum reflection time in the official documentation, so I do not treat it as an upper bound, and I write the representative value as the client-side p50. I do not write "reflected within 60 seconds", "reflected every minute even under continuous updates", or "never synchronized" either, because none has grounds. This explanatory model (wait, then aggregate, then export) is not an SLA.

Three notes follow.

In a conflict (the same key changed from the file side and the S3 API side), the S3 bucket side became the source of truth (3/3 converged on the S3 content). The file-side version was moved to .s3files-lost+found-<fs-id> (matching the CloudWatch LostAndFoundFiles peak of 3.00).
That directory is not created on conflict; it is present from the start (0700 root:root), and what happens on conflict is the eviction event. The official Synchronization page says files in this directory are not copied to the S3 bucket (I did not measure this).
For large rename / move operations, a rename on the file side finishes immediately, while on the S3 side it becomes a per-object copy and delete. AWS says a rename of 100,000 files takes several minutes to reflect on the S3 side, and that the prefixes from before and after the rename are both visible midway. I have not measured this behavior separately.

Measurements that separate the propagation conditions (long continuous appends, after updates stop, while another file is being updated, concurrent updates) are deferred to a separate verification plan (a plan, not a new measurement).

What this comparison can settle is two things: whether the source of truth lives in an S3 bucket or an ONTAP volume, and whether you can accept the roughly 60 seconds from file to bucket. I estimate the monthly cost in the next chapter.

Monthly cost estimate: an order that splits by object size, and room for assumptions to move it

I apply the same list prices to the two setups from the previous chapter, with the stored volume fixed at 20 TiB. I estimated 10 cells: 5 patterns that differ in monthly read and write volume (P1, P4, P5, P6, P8) times 2 object sizes (64 KiB and 4 MiB). The amounts in this chapter are list price x quantity for ap-northeast-1, with prices read on 2026-08-09, and they are subtotals of the main line items. The subtotals add up the line items shown in the tables, and some items are not included. They are not measurements and not invoices. "Base assumptions" below means the values in the common-assumptions table in the next section, "Five patterns".

Under this model's base assumptions, in all 10 cells, the subtotal is lower on FSx for ONTAP at 64 KiB and lower via S3 Files at 4 MiB. At 64 KiB, FSx for ONTAP comes to $1,783.11 to $2,218.13 a month and S3 Files to $7,928.32 to $12,030.81. At 4 MiB, S3 Files comes to $512.14 to $543.52 a month and FSx for ONTAP to $1,781.28 to $1,788.08. Because the order was the same in all five patterns, what splits the order in this model is the object size, not the read and write volume. The candidate for the dividing line is the S3 Files import threshold (default 128 KiB). I estimated two sizes, 64 KiB and 4 MiB, so the tables have no rows near 128 KiB or at 1 MiB. The reading that the dividing line sits at the threshold comes from those two points and from how the price works (high-performance storage is charged only for sizes below the threshold), and I did not estimate sizes at the boundary. The import condition is sizeLessThan, so I treat exactly 128 KiB as not imported. This order is a result under the base assumptions, and it can flip when they change, so check the section "Assumptions under which the order can flip" in this chapter against your own values.

Five patterns: total monthly reads and writes

Writing "read N times a month" alone does not fix the total volume read. Whether the N reads cover all objects or only some of them changes the total several times over. I define each pattern with three numbers: the working-set ratio, the number of reads per working-set object, and the write volume. The table shows the monthly read and write volumes derived from them. The working set is the part that is read within the month. The working-set ratio is the working set's share of the stored volume, and the number of reads per working-set object is how many times each of its objects is read in a month. I set the write volume as a share of the stored volume (the churn rate).

Take P4 as an example. 10% of the 20 TiB stored (2 TiB) is the working set, and each of its objects is read 10 times a month. The volume read is 2 TiB x 10 = 20 TiB, which is 1.0x the stored volume. The other 90% (18 TiB) is not read at all within the month. With 4 MiB objects, that is 524,288 objects read 10 times each, or 5,242,880 reads. P4 has no writes, so the stored volume stays at 20 TiB.

Pattern	Working-set ratio	Reads per working-set object	Monthly read volume (multiple of stored volume)	Monthly write volume (multiple of stored volume)	Basis of the values
P1 Cold archive	5% (1 TiB)	1	1 TiB (0.05x)	None	Placeholder (no source)
P4 Re-read a small working set heavily	10% (2 TiB)	10	20 TiB (1.0x)	None	This model's estimate assumption (not a measurement)
P5 Read-mostly with light ingest	10% (2 TiB)	10	20 TiB (1.0x)	1 TiB (0.05x, churn rate 5%)	Reads as in P4. The write volume is a placeholder
P6 Mixed read and write	50% (10 TiB)	4	40 TiB (2.0x)	20 TiB (1.0x)	Placeholder (no source)
P8 Write-centred	10% (2 TiB)	1	2 TiB (0.1x)	20 TiB (1.0x)	Placeholder (no source)

Source of this table: The patterns

The placeholder values are numbers I put in to make the arithmetic work. They are not taken from a real workload. Replace them with your own monthly read and write volumes. The cost model also has P2, P3, and P7, but I left them out of the figures and tables because they have the same monthly read and write volumes, and so the same amounts, as P4 and P6 (P4 matches P2 and P3, and P6 matches P7).

The following assumptions are common to all 10 cells.

Item	Value
Region	ap-northeast-1
Prices	List prices read on 2026-08-09 (on-demand, excluding tax). I re-read the 2026-08-09 record with the Price List API (read-only) on 2026-10-16, and the values were the same. It covers 10 items and does not include the two PUT request prices
Stored volume	20,480 GiB (20 TiB). Common to all cells
FSx for ONTAP	Single-AZ first generation, 128 MBps. Storage efficiency 40% on the SSD tier and 20% on the capacity pool tier. 30% of the data on the capacity pool (common to all patterns). Reads are served from the file system, and writes are charged as PUT through S3 Access Points
S3 Files	Import threshold 128 KiB. For sizes below the threshold, the whole stored volume is assumed to be held as active in high-performance storage. Expiry is the default 30 days

Source of this table: Common baseline

Cost figure 1: monthly cost of P1 and P4, which have no writes

Cost figure 1: breakdown of the monthly cost of P1 and P4, which have no writes (ap-northeast-1, list price x quantity, not an invoice). The horizontal axis is shared with cost figure 2. The working-set ratio and read count of P1 are placeholders with no source. "Figure 1" and "Figure 2" inside the images mean these two cost figures, not the internals figure in a preceding chapter. The figure shows the same numbers as the table below (dark theme version).

Pattern / size	Monthly operations	Setup	Storage	S3 Files high-performance storage	Fixed bandwidth	Requests etc.	Of which S3 Files metadata charges	Main-item subtotal
P1 / 64 KiB	16,777,216 reads, no writes	Direct read with the S3 API	$512.00	—	—	$6.21	—	$518.21
P1 / 64 KiB	16,777,216 reads, no writes	Via S3 Files	$512.00	$7,372.80	—	$43.52	$2.56	$7,928.32
P1 / 64 KiB	16,777,216 reads, no writes	FSx for ONTAP	$1,665.28	—	$115.97	$1.86	—	$1,783.11
P1 / 4 MiB	262,144 reads, no writes	Direct read with the S3 API	$512.00	—	—	$0.10	—	$512.10
P1 / 4 MiB	262,144 reads, no writes	Via S3 Files	$512.00	$0.00	—	$0.14	$0.04	$512.14
P1 / 4 MiB	262,144 reads, no writes	FSx for ONTAP	$1,665.28	—	$115.97	$0.03	—	$1,781.28
P4 / 64 KiB	335,544,320 reads, no writes	Direct read with the S3 API	$512.00	—	—	$124.15	—	$636.15
P4 / 64 KiB	335,544,320 reads, no writes	Via S3 Files	$512.00	$7,372.80	—	$870.40	$51.20	$8,755.20
P4 / 64 KiB	335,544,320 reads, no writes	FSx for ONTAP	$1,665.28	—	$115.97	$37.25	—	$1,818.50
P4 / 4 MiB	5,242,880 reads, no writes	Direct read with the S3 API	$512.00	—	—	$1.94	—	$513.94
P4 / 4 MiB	5,242,880 reads, no writes	Via S3 Files	$512.00	$0.00	—	$2.74	$0.80	$514.74
P4 / 4 MiB	5,242,880 reads, no writes	FSx for ONTAP	$1,665.28	—	$115.97	$0.58	—	$1,781.83

Source of this table: P1 / 64 KiB, P1 / 4 MiB, P4 / 64 KiB, P4 / 4 MiB

"Requests etc." is the sum of S3 requests, S3 Files read and write charges (metadata charges included), and FSx for ONTAP capacity pool read requests and PUT. It is the purple category in the figure. "Storage" for FSx for ONTAP is SSD and capacity pool storage added together.

The FSx for ONTAP rows assume reads are served from the file system, and they do not include read charges through S3 Access Points. The direct-read rows for the S3 API appear only for P1 and P4, which have no writes. In the patterns with writes, the write path differs by setup (S3 API PUT, NFS write plus sync, PUT through S3 Access Points), and putting them in one table would mix the write assumptions.

In the 4 MiB rows, the S3 Files subtotal is nearly the same as the direct S3 API read subtotal ($512.14 vs. $512.10 for P1, and $514.74 vs. $513.94 for P4). 4 MiB is at or above the threshold, so the data is not held in high-performance storage and is read straight from the bucket. In the 64 KiB rows, S3 Files adds $7,372.80 of high-performance storage.

Cost figure 2: monthly cost of P5, P6, and P8, which have writes

Cost figure 2: breakdown of the monthly cost of P5, P6, and P8, which have writes (ap-northeast-1, list price x quantity, not an invoice). The horizontal axis is shared with cost figure 1. The write volume of P5 and the working-set ratio, read count, and write volume of P6 and P8 are placeholders with no source. The figure shows the same numbers as the table below (dark theme version).

Pattern / size	Monthly operations	Setup	Storage	S3 Files high-performance storage	Fixed bandwidth	Requests etc.	Of which S3 Files metadata charges	Main-item subtotal
P5 / 64 KiB	335,544,320 reads, 16,777,216 objects written	Via S3 Files	$512.00	$7,372.80	—	$990.66	$55.68	$8,875.46
P5 / 64 KiB	335,544,320 reads, 16,777,216 objects written	FSx for ONTAP	$1,665.28	—	$115.97	$55.36	—	$1,836.61
P5 / 4 MiB	5,242,880 reads, 262,144 objects written	Via S3 Files	$512.00	$0.00	—	$4.04	$0.87	$516.04
P5 / 4 MiB	5,242,880 reads, 262,144 objects written	FSx for ONTAP	$1,665.28	—	$115.97	$0.87	—	$1,782.11
P6 / 64 KiB	671,088,640 reads, 335,544,320 objects written	Via S3 Files	$512.00	$7,372.80	—	$4,146.01	$192.00	$12,030.81
P6 / 64 KiB	671,088,640 reads, 335,544,320 objects written	FSx for ONTAP	$1,665.28	—	$115.97	$436.88	—	$2,218.13
P6 / 4 MiB	10,485,760 reads, 5,242,880 objects written	Via S3 Files	$512.00	$0.00	—	$31.52	$3.00	$543.52
P6 / 4 MiB	10,485,760 reads, 5,242,880 objects written	FSx for ONTAP	$1,665.28	—	$115.97	$6.83	—	$1,788.08
P8 / 64 KiB	33,554,432 reads, 335,544,320 objects written	Via S3 Files	$512.00	$7,372.80	—	$3,117.79	$94.72	$11,002.59
P8 / 64 KiB	33,554,432 reads, 335,544,320 objects written	FSx for ONTAP	$1,665.28	—	$115.97	$366.11	—	$2,147.36
P8 / 4 MiB	524,288 reads, 5,242,880 objects written	Via S3 Files	$512.00	$0.00	—	$26.32	$1.48	$538.32
P8 / 4 MiB	524,288 reads, 5,242,880 objects written	FSx for ONTAP	$1,665.28	—	$115.97	$5.72	—	$1,786.97

Source of this table: P5 / 64 KiB, P5 / 4 MiB, P6 / 64 KiB, P6 / 4 MiB, P8 / 64 KiB, P8 / 4 MiB

Adding writes adds, on the S3 Files side, PUT to the S3 bucket and, at 64 KiB, import writes to high-performance storage. On the FSx for ONTAP side it adds PUT through S3 Access Points. For P6 at 64 KiB, "Requests etc." is $4,146.01 via S3 Files and $436.88 on FSx for ONTAP. Across the three patterns of cost figure 2, the 4 MiB subtotals stay within $516.04 to $543.52 via S3 Files and $1,782.11 to $1,788.08 on FSx for ONTAP.

Items that push up the S3 Files subtotal at 64 KiB: high-performance storage and the number of writes

S3 Files imports files below the threshold (default 128 KiB) into high-performance storage and charges $0.36 / GB-month for the amount held. Files at or above the threshold are not held and are read straight from the bucket. The model assumes the whole stored volume (20,480 GiB) is active at the small size. That makes high-performance storage $7,372.80 in all five 64 KiB cells and $0.00 in all five 4 MiB cells.

On the FSx for ONTAP side, storage of $1,665.28 and fixed bandwidth of $115.97 are the same in all 10 cells. Only the "Requests etc." column moves with size and with read and write volume.

Why the S3 Files subtotal is higher at 64 KiB splits into two cases by pattern. In P1, P4, and P5, the S3 Files subtotal falls below FSx for ONTAP once the $7,372.80 of high-performance storage is removed. In P6 and P8 it stays higher even after the removal. Those two write 335,544,320 objects, and the S3 bucket PUT price ($0.0047 / 1,000) is higher than the PUT price through S3 Access Points ($0.00108 / 1,000), which also contributes. Those two PUT prices are still the values read on 2026-08-09 and were not part of the 2026-10-16 re-read.

The high-performance storage amount follows from assuming a 100% active share. That share is a placeholder, not a measured value. I made one observation of a related quantity (described in the section on assumptions under which the order can flip).

Metadata charges: the accounting assumption and one observation

The model counts S3 Files metadata as 4 KiB x operations, in separate import-side and read-side lines, and includes both in the subtotal. The column "Of which S3 Files metadata charges" in the tables is the sum of those two lines, and across all 32 cells it matches usage x unit price ($0.04 / $0.07). In two measurements, on 2026-10-16 and 2026-10-17, the Cost Explorer usage for APN1-Files-Read matched the sum of CloudWatch DataReadBytes and MetadataReadBytes to within 0.5% (daily, Estimated: true). Ingest was billed as APN1-Files-Write and did not appear in the CloudWatch write metrics. Because cost is set by usage (bytes), the per-operation breakdown of a read (open, getattr, and so on) does not change the estimate. The invoiced amount is offset to a net zero by credits, so I have not confirmed it as a billed figure. I did not measure other sizes or other instance types. The primary record of this reconciliation is at billing reconciliation.

I have one observation (2026-10-16, ap-northeast-1, m5.large, not reproduced). I read 1,000 4 MiB objects once each with dd, and CloudWatch MetadataReadBytes registered about 6.0 times per object, with an average of about 4.1 KiB each. Changing the I/O size to 1 MiB or 4 MiB made no difference. The 64 KiB control with the same procedure registered about 2.0 times. I do not know where the roughly 6 comes from, so I have not applied that count to the "4 KiB x operations" count in the model. DataReadBytes for 4 MiB had 0 data points, so I could not confirm whether the reads went through NFS or directly to the bucket. For the ingest of 4 MiB objects written through the S3 API, MetadataWriteBytes had 0 data points. I cannot tell whether no ingest happened or whether it is not metered.

The roughly 6.0 figure is the number of data points the CloudWatch metric recorded. I later queried Cost Explorer and reconciled it against the billed usage. The APN1-Files-Read usage on 2026-10-16 and 10-17 matched the sum of CloudWatch DataReadBytes and MetadataReadBytes to within 0.5% (daily, Estimated: true). I did not capture the breakdown of the roughly 6.0 operations, but because cost is set by usage (bytes), the breakdown does not change the subtotal.

The following remain unconfirmed:

The invoiced amount itself. It is offset to a net zero by credits, so I have not confirmed it as a billed figure.
Other instance types, nconnect, multiple processes, and other object sizes. I did not measure them.

The metadata column in the tables keeps the assumption of one operation per read. The difference in the operation count does not affect a cost that is set by usage.

Assumptions under which the order can flip: what to replace with your values

The order in this chapter is a result under fixed assumptions. If you replace the following assumptions with values from your environment, the amounts and the order move. The table of subtotals when one assumption moves at a time and the table of values at which the order flips are in the section "Baseline assumptions and sensitivity" of the generated document in the repository. A figure that takes P4 as an example and shows where each assumption sits and what kind of evidence backs it is in the section "Break-even map for pattern P4" (images: light and dark). Line length does not represent uncertainty.

Under the base assumptions, the FSx for ONTAP subtotal is lower at 64 KiB and the S3 Files subtotal is lower at 4 MiB. At 4 MiB, the direction does not change when one assumption moves at a time over the values tried, and it changes for every pattern in the scenario that moves several assumptions together toward a lower FSx for ONTAP subtotal (a combination that includes a value with no source). At 64 KiB, lowering the active share of S3 Files high-performance storage from the base makes the S3 Files subtotal lower, starting with the patterns that have fewer writes, though the base value of that share is a placeholder.

Share of data on the capacity pool: fixed at 30% in every pattern.
Storage efficiency of the SSD tier: fixed at 40%.
Active share of S3 Files high-performance storage: fixed at 100%. A placeholder.
S3 Files import threshold: the default 128 KiB.
FSx for ONTAP deployment: Single-AZ first generation, 128 MBps.

There are also retention items that the base leaves out. Bucket versioning is required, so the S3 Standard storage charge for the non-current versions you keep is added to the S3 Files side. Keeping 1,024 GiB as an example raises the S3 Files subtotal by $25.60 a month in all 10 cells of the tables ($0.025 per GiB-month, model output). For snapshots on the FSx for ONTAP side, I have not confirmed which tier they occupy or how they are billed, so you can enter a capacity but it does not enter the subtotal.

I made one observation of a quantity close to the active share (2026-10-17, ap-northeast-1, 200,000 files of 64 KiB, the default import threshold). When I read 5% and 20% of the working set, the increase in StorageBytes, the billing dimension of high-performance storage, was 1.094 times the bytes read (5.47% and 21.89% of the stored volume). The working-set ratios are values the experiment set. StorageBytes is the sum of data and metadata, so it is not a direct measurement of the active share. In the same experiment ImportFailures registered about 200,000 per file system. I did not investigate the cause and did not separate its effect on StorageBytes. I have not confirmed the billed amount, so the ratio cannot be read as a dollar figure. It is one observation with no repeat in another environment, so I did not lower the 100% in the tables. The record is in the verification record (Japanese).

The capacity pool share and the active share of high-performance storage are fixed values decoupled from the working-set ratio. In P1, P4, and P5 the data touched within the month (the working set plus writes) does not reach the whole stored volume, so the two shares can be set again to match the volume actually touched. Fitting the active share of high-performance storage would lower the S3 Files subtotal at 64 KiB. For the FSx for ONTAP subtotal when the capacity pool share is fitted, the generated document only says that the two together would go down, and I have not checked each one separately. In P6 and P8 the data touched within the month adds up to the stored volume or more, so if the capacity pool share is assumed to cover the whole volume, the FSx for ONTAP subtotal would rise. The directions come from the generated document, and I have not calculated the change in amounts. One more point: for S3 Files at 64 KiB in P1 and P4, which have no writes, the model counts no first-read stream (the GET from the bucket and the import write into high-performance storage) and counts all reads as high-performance storage reads. Counting it would add some items and remove others, and the model does not settle the net direction.

The capacity pool read requests included in "Requests etc." for FSx for ONTAP assume one request per GET on the 30% of reads that hit the capacity pool. The actual count is unconfirmed.

An observation that helps comes from an adjacent repository (the FSx for ONTAP monitoring integrations). On a file system of the same type (Single-AZ first generation, 128 MBps, SSD 1,024 GiB), on 2026-10-11, I read volumes that had been created with ALL and written to directly, from a Lambda function outside the VPC. CloudWatch CapacityPoolReadOperations came to 0 per object at 64 KiB, about 5.4 to 5.6 at 1 MiB, and about 29.8 to 33.2 at 5 MiB (source: per-invocation reads, Japanese). The 1 MiB and 5 MiB values are for cases where operations registered, and 2 of the 8 runs built the same way registered none. The sizes measured were 64 KiB, 1 MiB, and 5 MiB; 4 MiB was not measured. What was counted is the CloudWatch operation count, not a billed count, and that record also leaves the relation between the two unconfirmed. The measurement date, the volume, and the client all differ from this post's measurements. For these reasons I did not apply these operation counts to the model's count (one request per GET). The capacity pool read request amount that follows from the count the model assumed is $0.03 to $1.16 across the five 4 MiB cells.

Reading data on the capacity pool: billing items exist, latency is unmeasured

Reading an object on the capacity pool of a Tiering All volume through S3 Access Points can incur two billing items: GET through S3 Access Points and capacity pool read requests (documented). This post's measurements did not produce the number of requests per GetObject or the latency of reading from the capacity pool. The capacity pool storage price is lower than SSD, $0.0238 vs. $0.15 per GB-month (Single-AZ, ap-northeast-1, read 2026-08-09).

These are the p50 observations of GET through S3 Access Points from a local machine.

Object size	SSD side	After changing to Tiering All	p50 difference
64 KiB	46.5 ms	56.4 ms	+9.9 ms
1 MiB	58.6 ms	74.0 ms	+15.4 ms
5 MiB	116.0 ms	121.3 ms	+5.3 ms

The environment was 2026-10-10, ap-northeast-1, FSx for ONTAP Single-AZ first generation, 128 MBps, SSD 1,024 GiB, from a local machine outside AWS over the internet (round-trip time included), concurrency 1. It is one measurement: 166 to 407 requests on the SSD side and 10 requests after the change.

I cannot read this difference as a delay from reading the capacity pool, for three reasons.

In the read window after the change, the per-volume CapacityPoolReadOperations was 0. The reads did not reach the capacity pool.
The 5 MiB difference (+5.3 ms) is smaller than the spread from p50 to p90 on the SSD side (9.0 ms).
I did not alternate SSD-side and post-change reads, and the request counts are asymmetric.

Source of this table: Reads by tier (Japanese)

Before you read much Tiering All data through S3 Access Points, look at the per-volume CapacityPoolReadOperations one-minute values in your own environment and then estimate.

Choosing between S3 bucket first and file system first

Laid out at the same granularity, the constraints of the two setups are as follows. For monthly cost, treat the rows as guidance under the base assumptions of the estimate above.

Setup	Fits when	Constraints
S3 Files (S3 bucket first)	You want the source of truth in an S3 bucket. Clients are Linux on AWS and speak NFSv4.1 or later. Objects of 128 KiB or more dominate. You do not want a fixed cost	Bucket versioning required. Linux on AWS only. NFSv4.1 / 4.2 only (NFSv3, SMB, and nconnect unsupported). About 60 seconds from file to bucket. Objects below 128 KiB incur high-performance storage charges
FSx for ONTAP + S3 Access Points (file system first)	You want the source of truth in an ONTAP volume. You have NFSv3 or SMB clients. You read from outside AWS by mounting over NFS or SMB (I did not measure this). Objects below 128 KiB dominate	A fixed cost of $269.57 a month or more. Capacity pool read latency is unmeasured, and read request charges can apply. Clients share the throughput capacity. S3 event notifications, lifecycle, and versioning are not supported on the S3 Access Points side

If your clients are Linux on AWS and you handle objects of 128 KiB or more while keeping the source of truth in an S3 bucket, start evaluating with S3 Files. If you have NFSv3 or SMB clients, read from outside AWS by mounting over NFS or SMB (I did not measure this), or mostly have objects below 128 KiB, start with FSx for ONTAP and S3 Access Points.

The object-size conditions are guidance drawn from the 64 KiB and 4 MiB results I estimated. I did not estimate sizes near 128 KiB. Decide the monthly order only after you replace the values in "Assumptions under which the order can flip" with those of your environment and check.

An invalid warm / cold comparison: lowering the threshold does not evict data already loaded

My first warm / cold comparison was invalid because both reads were warm. I have not confirmed the latency difference between warm and cold. Here warm means reading data held in high-performance storage, and cold means reading straight from the bucket.

I measured once with a different design (2026-10-19, ap-northeast-1, one c5n.9xlarge, Amazon Linux 2023, 64 KiB files placed one per directory, drop_caches before each read, default import threshold, concurrency 1). The p50 of a first read was 230.22 ms, and the p50 of re-reading the same file two minutes later was 18.71 ms (n=360 each). The paired difference on the same object was positive for all 360 pairs. But DataReadBytes was the same 65,972 B for the first and second reads, so I cannot distinguish the path of the first read. Of 40 objects in one directory, only the first was 438.9 ms, and the median of the other 39 was 13.12 ms (one observation; I did not investigate the reason). The conditions differ from the 4.05 ms (straight from the bucket) later in this chapter, and I did not investigate the cause of the difference. Details are in the verification record.

The goal was to check the claim that only frequently read data becomes low-latency. I assumed that lowering the import threshold (sizeLessThan, default: import files under 128 KiB) and then reading the same file would make the read cold, and I measured on that assumption.

Why it was invalid: the threshold only decides what the next first access imports

The threshold decides what gets imported the next time a directory is accessed for the first time. Data already in high-performance storage is not evicted when you lower it. Even in the window I meant to be cold, CloudWatch DataReadBytes (bytes read from high-performance storage) was nonzero, so both reads came from high-performance storage.

The numbers gave it away. The supposedly cold read was faster than the warm one, at 2.94 ms, which cannot be right. The record reads that difference as run-to-run variation and remount differences.

Three reads: two are valid

The environment was ap-northeast-1, 2026-10-10, c5n.9xlarge, 64 KiB, o_direct, concurrency 1, p50 of n=30.

Read	p50	Valid
High-performance storage (threshold 131072, 64 KiB already imported)	5.33 ms	Valid
Same file after lowering the threshold to 1024	2.94 ms	Invalid (both warm)
A newly created 64 KiB file with the threshold left at 1024 (direct from the bucket)	4.05 ms	Valid

Source of this table: Three reads (Japanese)

The two valid rows, 5.33 ms and 4.05 ms, differ by an amount within run-to-run variation. A separate t3.small record showed 10.8 ms from high-performance storage and 55.0 ms straight from the bucket (2026-09-01), but that did not reproduce on c5n.9xlarge. The environments and methods differ, so it is not a comparison that varies only the path in one environment.

To measure cold, read a new file that has not been imported after lowering the threshold, and confirm the path with DataReadBytes. The 2.94 ms in this post cannot serve as evidence for a warm / cold difference.

Status of the four stated values: I got numbers for two

Of the four values AWS states, I got numbers for two: propagation time when using both files and the S3 API, and aggregate growth with client count (up to 8). The comparison for "only frequently read data becomes low-latency" was invalid, so I cannot confirm it. I did not measure "existing tools work as they are".

Stated value	What I confirmed	Status
Existing file-based tools work as they are	The measurement tool, VDBENCH, ran on the mount. I did not look at compatibility with existing applications	Not measured
The same data is usable from both files and the S3 API	Propagation p50 was 63,769.3 ms (a window of about 60 seconds) from writing a file to reading it with the S3 API, and 1,533.1 ms from writing with the S3 API to reading it as a file (t3.small, NFSv4.2, 64 B, concurrency 1, n=30, 2026-09-01)	Measured. File to S3 has a window of about 60 seconds
Only frequently read data sits in high-performance storage and gets low latency	64 KiB p50 was 5.33 ms from high-performance storage and 4.05 ms straight from the bucket (c5n.9xlarge, n=30, 2026-10-10), within run-to-run variation. The first comparison was invalid	Not confirmed
Many clients can use it at once	Aggregate 784.4 to 6,282.6 MB/s (1 to 8 clients, 8.01x, c5n.2xlarge, 2026-10-10). I did not measure the 25,000-connection and TB/s-class limits	Measured up to 8 clients

Sources of this table: Propagation measurement, Three reads, Client-count measurement (same) (Japanese)

The values come with costs. Propagation from file to bucket takes about 60 seconds, the supported protocols are only NFSv4.1 and NFSv4.2, bucket versioning is required, and clients are limited to Linux on AWS. The two confirmed points are usable as grounds for adoption. Treat the low-latency value as an assumption only after you measure a cold read in your own environment.

What I have not measured

This post's measurements cover a single S3 Files mount (c5n.9xlarge), S3 Files client count (c5n.2xlarge, 1 to 8 clients), S3 Files 64 KiB cold and warm latency (one client), and latency by tier (SSD vs. capacity pool) through FSx for ONTAP S3 Access Points for Single-AZ and the preferred AZ of Multi-AZ first generation, inside a VPC. I did not measure the following four things.

What I have not measured (subject and object stated)	Why not	When not knowing hurts	How to measure
For S3 Files, I cannot distinguish whether the latency of a first read is a direct read from the bucket or a read from high-performance storage after import.	`DataReadBytes` was the same 65,972 B for the first and second reads, so this metric does not distinguish the two. The values I measured are in the chapter "An invalid warm / cold comparison".	You use first-access latency as a direct-from-bucket value in design.	Lower the import threshold below the file size, create new files, place them one per directory, and read each after a `drop_caches`. Compare the latency in a window where `DataReadBytes` does not rise against the first-read value in the body.
For FSx for ONTAP S3 Access Points, I did not confirm how the CloudWatch read operation count corresponds to the billed request count. I also do not know the conditions under which an ALL volume's reads reach the capacity pool.	I measured the operation count (the comparison table in the body). The Cost Explorer value on the measurement day (`APN1-Requests.SAZ_2N:CPoolStdRd`) is daily and account-wide, read before aggregation completed. The account has another FSx for ONTAP, so I cannot compare it against the measured window's operation count. Batches where the reads did not reach, or only partly reached, the capacity pool numbered 2 / 12 at 4 MiB, 8 / 13 at 1 MiB, and 10 / 10 at 64 KiB (supplementary batches included). I did not investigate the cause.	You take the operation count directly as the billed request count to estimate read cost on a Tiering All volume.	After the date is settled, re-read the daily `CPoolStdRd` value, subtract the same day's `CapacityPoolReadOperations` (CloudWatch) of the other file systems, and compare against the measured file system. For the reach conditions, repeat with the time from write to read, the read method (ALL only, alternating), and the size separated.
I did not measure the latency difference of reading an SSD volume and a capacity pool volume through S3 Access Points from an on-premises client. The result from EC2 inside a VPC is in the comparison table in the body.	I measured only from EC2 inside a VPC and did not set up a client outside AWS. The single reading from a local machine outside AWS over the internet (+5.3 to +15.4 ms) did not reach the capacity pool and cannot serve as an on-premises value.	Your client is on premises and you apply the in-VPC difference as is.	From an on-premises client, alternate reads of the same two volumes at the same size and confirm that the per-volume `CapacityPoolReadOperations` rose in the read window.
For Multi-AZ first generation, I cannot distinguish whether latency from a standby-AZ client differs from a preferred-AZ client.	The standby-AZ client had one batch per size (n=30), and the SSD side at 4 MiB was bimodal, making the difference negative. It cannot be read as an AZ difference.	Your client is in an AZ other than the preferred one and you apply the preferred-AZ result.	From a standby-AZ client, read several batches per size. Investigate the cause of the bimodal SSD side first.

If any row has conditions that differ from yours, measure that row first.

Reproducing it yourself: three runbooks from creation to teardown

You can follow this post's measurements in your own AWS account with the same steps. I put runbooks from creating the environment to tearing it down on GitHub. The links are pinned to the commit of the measurement.

Create the environment and mount: S3 Files verification environment setup (Japanese). It builds with CloudFormation and runbook.sh. The most frequent cause of a silently hanging mount (missing CloudWatch permissions on the host role) is written at the start of these steps.
Measure throughput and client count: from the section "Measuring throughput and client count" on the same page, go to throughput-runbook.sh (single mount) and s4-ladder.sh (1 to 8 clients). VDBENCH requires accepting Oracle's license, so obtain it separately.
Results and conditions: S3 Files file system performance (Japanese). This is where this post's numbers come from, each with its environment. It also states that each is a single sample.

Leaving the verification environment up keeps billing. When you finish measuring, tear it down with the runbook's destroy and stop, then confirm by count that nothing remains.

Closing

Two things hold from the measurements, and one from the estimate.

The single-mount ceiling was one efs-proxy CPU core. It was 1,104.9 MB/s for 1 MiB sequential reads on a c5n.9xlarge (one run, 2026-10-10, ap-northeast-1).
When bandwidth falls short, adding clients raised it. From 1 to 8 c5n.2xlarge clients, the aggregate went from 784.4 MB/s to 6,282.6 MB/s (8.01x). I have not measured 9 or more.
In the estimate, under this model's base assumptions, the monthly cost order split by object size, not by read and write volume. The candidate for the dividing line is around the S3 Files import threshold (128 KiB), and I estimated only two sizes, 64 KiB and 4 MiB. With the stored volume fixed at 20 TiB and five patterns (P1, P4, P5, P6, P8), 64 KiB comes to $1,783.11 to $2,218.13 a month on FSx for ONTAP and $7,928.32 to $12,030.81 via S3 Files, and 4 MiB comes to $512.14 to $543.52 via S3 Files and $1,781.28 to $1,788.08 on FSx for ONTAP (ap-northeast-1, subtotals of the main line items with prices read on 2026-08-09; list price x quantity, not an invoice). This order is a result under the model's base assumptions and can flip when they change, so check it against your own values.

Which one to consider first narrows down by where the source of truth lives and where clients read from. Because the monthly order by object size can change with assumptions, treat the object-size conditions in the next table as guidance under the base assumptions of the estimate.

Condition	Setup	Check first
The source of truth is an S3 bucket. Clients are Linux on AWS and can use NFSv4.1 or later. Objects of 128 KiB or more dominate	S3 Files	Bucket versioning, about 60 seconds from file to bucket, and that NFSv3 and SMB are not supported
The source of truth is an ONTAP volume. You have NFSv3 or SMB clients, read from outside AWS by mounting over NFS or SMB (I did not measure this), or mostly have small objects below 128 KiB	FSx for ONTAP with S3 Access Points	A fixed cost of $269.57 a month or more, capacity pool read request charges, and that clients share the throughput capacity
Warm / cold latency, 9 or more clients, or the capacity pool read count will decide your design	Measure in your own environment, then decide	For 9 or more clients, the end of the chapter "An aggregate that scales with client count"; otherwise the rows of the "What I have not measured" table whose conditions differ from yours

I keep every measured value, each with its environment, on GitHub. I hope this helps anyone with a similar need (collect in S3, read as files).

Until next time.

Yoshiki Fujiwara

Why Amazon FSx for NetApp ONTAP block performance differed by about 2x on the same specified values, isolated by measurement

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 30 Sep 2026 05:43:42 +0000

Why Amazon FSx for NetApp ONTAP block performance differed by about 2x on the same specified values, isolated by measurement

While measuring, I hit a case where the templates and the specified values matched, yet the number
moved every time I measured. Rather than let it go, I wanted to isolate whether it was a
measurement mistake or something about the environment, so I eliminated the causes one at a time.

The same thing shows up when a post-migration performance test does not agree, and when you want to
settle on one representative value while sizing an initial build. It is a question that eats a lot of
time, so this article is the record of that isolation.

What this article does and does not cover: it covers investigating the cause of the performance
swing, and how to isolate the bottleneck using four CloudWatch metrics. It does not cover mount
procedures, how to confirm multipathing, or figures usable for sizing (details under "What this
article does not cover" below). All figures are from a Single-AZ configuration; Multi-AZ is
unverified.

dev.to series: FSx for ONTAP Block Protocols, Measured.

A 1 MiB sequential read on Amazon FSx for NetApp ONTAP (hereafter FSx for ONTAP), measured with the
same template, the same specified values and the same workload, varied by 2.64x across
environments. Each one was stable within its own run. This is not measurement scatter.

This is the record of eliminating the causes one at a time. It was not the path, not how the
connection was made, not the queue count, not the distribution, and not the iopolicy (Linux's
multipath setting for distributing I/O across paths to an NVMe device). What was
acting on it was the layout on disk, which breaks after 300 seconds of 4 KiB random writes and
comes back when the data is rewritten sequentially.

Three times along the way I nearly settled on the wrong explanation. First: "if it exceeds the
specified value it must be served from memory." Second: "the layout must also be setting the
amplification factor for 4 KiB random reads." Third: "the amplification factor must be decided by
whether 4 KiB random writes ever ran on that device." All three were disproved once I aimed a
measurement that could decide them. All three are in here. The third had five observations
lined up cleanly behind it and survived until a control experiment killed it.

What was measured

Assuming no prior knowledge of FSx for ONTAP, here's what varies between the deployments.

Deployment 1: AWS Cloud, single VPC, single AZ. One EC2 client -> (NVMe/TCP, 1 MiB sequential read) -> Amazon FSx for NetApp ONTAP (same template, same specified values)
Deployment 2: same AWS Cloud, single VPC, single AZ. One EC2 client -> (same connection, same workload) -> Amazon FSx for NetApp ONTAP (same template, same specified values)
The invisible difference between Deployment 1 and 2: layout on disk. It never appears in the template or the list of specified values

(Deployment 1, Deployment 2), each measured from one EC2 client with the same connection and the
same workload. Shows what can be held identical (path, connection setup, queue count,
distribution, iopolicy) against what cannot (layout on disk)" width="800" height="726">

Figure: **matching the CloudFormation template and the specified values doesn't make the layout
on disk match across a redeploy.* Path, how the connection is made, queue count, distribution,
and iopolicy can all be held to the same value across deployments. Layout is the one condition
that changes every deployment and never appears in the list of specified values.*

Item	Role in this article
Deployment 1 / Deployment 2	Separate FSx for ONTAP file systems, built from the same template and the same specified values
Conditions that can be matched	Path, how the connection is made, queue count, distribution, iopolicy
Condition that can't be matched	Layout on disk. Breaks after 300 seconds of 4 KiB random writes, comes back when rewritten sequentially

What this article does not cover. Mount procedure is out of scope
(repository record, Japanese).
Confirming whether multipathing is usable is also out of scope (separate article).
The figures in this article cannot be used for sizing either. Why they cannot is the subject of
the article. All figures are from a Single-AZ configuration; Multi-AZ is unverified.

One of three, but there is no reading order. The block-protocol measurements are split by
reader.

Article Reader

A Sessions and queues People about to mount

B Confirming whether multipathing is usable People who followed the procedure and got stuck

C What moves the numbers (this article) People who measure the numbers, and people who cite them

None of the three presupposes the other two. Read only the part you need.

The same property on the file-protocol side (NFS / SMB / S3 API) is covered in
S3 Burst Part 2, with the primary record of those figures in
throughput, IOPS and concurrency (Japanese).
Those figures are also iorate=max values from this environment, so the way of measuring
described here applies to them unchanged.

The environment measured

Item	Value
File system	FSx for ONTAP second generation SINGLE_AZ_2, 6,144 MBps x 1 HA pair, SSD 4,096 GiB, provisioned 200,000 IOPS
Region	ap-northeast-1, single AZ
ONTAP	Not one version. It spans 9.18.1P5 (the early deployments), 9.18.1P6 (the per-path measurements) and 9.18.1 (the last three deployments). This is exactly the unresolved candidate left at the end of this article (the cross-article version mapping is in the ONTAP version matrix)
NVMe read cache	Disabled (confirmed on both nodes)
LUN / namespace	600 GiB, on a 900 GiB volume
Before measuring	All 600 GiB was written once. Unwritten blocks on a thin-provisioned volume return zeros, so reading without writing first measures nothing
Client	c5n.9xlarge; 50 Gbps network is the guaranteed figure
Instrument	VDBENCH 5.04.07 directly, `iorate=max` (unlimited), 512 threads, 60 s warm-up + 300 s measurement
Test data / efficiency settings	Not recorded whether the VDBENCH payload was compressible, or whether volume inline efficiency (`StorageEfficiencyEnabled`) was enabled. This article's subject is on-disk layout, so whether efficiency settings affected results is an unconfirmed variable

The window is 300 seconds, so the figures in this article include burst. They cannot be cited as a
baseline.

Every figure in this article is an iorate=max saturation point, not an operating point.
No target IOPS was given; it is what the path would take. This distinction bears on the subject
of the article itself. A saturation point answers "where does it jam", and does not answer "at
what response time can it be used".

The file-protocol side of the same repository measured that gap. With a target of 4,400 IOPS the
result was 4,406 MB/s at 8.86 ms; unlimited, it was 4,204 MB/s at 121.79 ms.
Unlimited is 5% lower in throughput and 14x worse in response time.
Recording one iorate=max point as "the ceiling" leaves behind a figure that is lower than what
is reachable and an order of magnitude worse in response time.

Response time was recorded as measured in only the two series under "not concurrency" below.
Elsewhere the values VDBENCH printed were not kept, and the environments are torn down. But with
512 threads fixed and no rate limit, response time is the identity 512 / IOPS (derived 216.6 ms
against measured 216.588 ms, confirmed to agree). The figures below can be read through that
identity: 1,288.87 MB/s is about 397 ms, 3,407.60 MB/s about 150 ms, 546.02 MB/s about
937 ms. All three are derived, not measured.

The response-time-versus-load curve was not measured. The file-protocol side produced a curve
across target IOPS values and a cutoff (the IOPS at which a given latency is reached) with
auto_vdbench, but the same thing could not be
produced on the block side. auto_vdbench creates test files on a mount path and cannot drive a
raw device or an NVMe namespace. Open work.

The same configuration varied widely

This is the part of the article I would most like read carefully.

As F-3, the same shape was measured three times within one connection.

Run	1 MiB sequential read
1st	3,401.49 MB/s
2nd	3,407.60 MB/s
3rd	3,407.59 MB/s

Within one connection it reproduces to 0.18%. The second and third differ by 0.01 MB/s.

But three environments built with the same procedure, the same template and the same negotiated
result (4 queues) produced different values.

Environment	1 MiB sequential read	Stability within the window
First	1,288.87 MB/s	stable
Second	2,366.49 MB/s	stable
Third	3,401-3,408 MB/s	stable (0.18%)

The spread is 2.64x. Each was stable within its run, so this is not measurement scatter.

So you cannot size from a single figure of the form "NVMe/TCP throughput on this configuration is
N MB/s". At the point where I had measured the second environment, I was about to write 2,366.49
as the representative value. I had not followed my own rule of repeating three times, so I noticed
one environment later than I should have.

I had put forward one candidate at the time: Asymmetric Namespace Access (the mechanism that
tells the host which of several paths to the same namespace is the shortest one, referred to as
multipathing throughout this article) is unusable, so
which of the two controllers the device under measurement sits behind can change per deployment.
Measurement said otherwise.

The path candidate is gone

I brought in a kernel with multipathing enabled (Rocky Linux 9.7) and measured within one deployment, on
one namespace, with the paths narrowed to one at a time.

Path	1 MiB sequential read
optimized only (the node holding the namespace)	1,278.11 MB/s
non-optimized only	1,279.07 MB/s

0.08% apart. Either controller gives the same figure, so the path cannot explain that spread.

How the connection is made was eliminated too. Connecting from optimized first, from non-optimized
first, and using connect-all as the AWS procedure has it, gave 1,276.60 / 1,277.38 / 1,277.28
MB/s. A spread of 0.06%. The queue count was the same in every configuration
(queue_count=5, 10 established sockets to :4420).

What was acting on it was when the data being read was last written

Within one deployment, one device, one path and one policy, rewriting the 600 GiB brings it back.

State	1 MiB sequential read
Immediately before the rewrite (control)	1,239.15 MB/s
Immediately after the rewrite	2,298.54 MB/s
Once more, right after that	2,296.12 MB/s

1.85x. And the second read does not fall back. This is not a property that is consumed by one
read.

This is the only place in this article where a ratio is written. The numerator and denominator
come from the same file system, the same device, the same path and the same policy: the reason a
ratio can be written here is the same reason none is written in the other sections.

CloudWatch answers what the cold side is running into.

State	`DiskIopsUtilization`	Disk throughput utilization	Network utilization
Cold read	avg 57-60 / max 95.0-95.4	30 / 42	8.7 / 11.6
Warm read	avg 13.6-14.4	40 / 41	19.2 / 19.5

The ceiling on the cold side is SSD IOPS. It is not the 6,144 MBps of throughput capacity that
was specified (utilization 42% or below). It is not the network either (11% or below).
And at 2,298 MB/s on the warm side, none of the utilization figures FSx for ONTAP publishes is at
its ceiling. What is acting on it there is not known.
(What is being read here is DiskIopsUtilization / disk throughput utilization / network
utilization from the
second-generation file system metrics.
The utilization metrics emit one point per HA pair and per aggregate.)

Not reads, and not elapsed time

Since it moves with "when it was written", I aimed separate measurements at what brings it down.
On top of one fill, reads, elapsed time and writes were applied in turn.

Immediately after	1 MiB sequential read
The fill	2,263.8 MB/s
A second read straight after	2,267.5
A third read straight after	2,268.4
30 minutes idle	2,258.5
300 seconds of 4 KiB random writes	1,272.8
Refilling	2,235.4

Reads do not bring it down. Three full passes over 600 GiB, a spread of 0.2%
Time does not bring it down. 30 minutes with nothing running, within 0.4%
4 KiB random writes bring it down. 2,258.5 to 1,272.8, a fall of 1.77x

Do not generalize this to "writes bring it down". What was aimed was one point: 300 seconds of
4 KiB random writes. The result of separating shape from duration and aiming again is in
the section below, and the generalization
did not hold.

What falls is not cache but the layout on disk

I first explained this as "cache retention". That was wrong. Read CloudWatch interval by
interval and the mechanism comes out plainly.

State	Client	One read on the disk side	Disk IOPS	Against the specified 200,000	`DiskRead / DataRead`
Right after the fill (5 intervals)	2,219-2,263 MB/s	87.3-89.6 KiB	25,743-27,404	13%	97-105%
After 4 KiB random writes (6 intervals)	1,151-1,267 MB/s	11.7-11.8 KiB	177,527-189,629	89-95%	169-188%

DiskRead / DataRead is about 100%, so it is reading from disk right after the fill too.
It is not being served from memory. What changed is the size of one read on the disk side, by a
factor of 7.5. Disk IOPS rises 7.3x as a result, and saturates at 89-95% of the specified value.

So: 4 KiB random writes break the layout on disk, and the same 1 MiB read becomes many small
reads. Reads do not change the layout, so they have no effect, and neither does time. Rewriting
sequentially restores a contiguous layout. This is also why only 4 KiB random reads are
unaffected: a random 4 KiB is one read on the disk side whatever the layout is.

DiskIopsUtilization alone does not get you here. It tells you whether something is
saturated; it does not tell you why it saturated (because each read is small).

And it goes back and forth within a single run. I think this is the strongest evidence.

What was running	One write	One disk read
Fill, then 3 reads	1,024 KiB	89.1 / 89.6 / 88.3 / 87.4 KiB
The read after 30 minutes idle	—	87.9 KiB
4 KiB random writes, then read	4.0 KiB	58.7 to 11.8 to 11.6 KiB
Sequential refill, then read	1,024 KiB	88.2 / 87.7 KiB

30 minutes idle does not move it, a 4 KiB write brings it down, a sequential write brings it back.
This is neither a difference between deployments nor measurement scatter.

Note: this d_io depends on the workload too. In 1 MiB sequential read intervals it is 88 or
11.7 KiB; in 4 KiB random read intervals it is 4-5 KiB. When comparing, match the client-side
size of one operation. Without that, the layout difference and the workload difference mix.

As a benchmarking practice, I think this is the most portable thing here.
If you publish a sequential read figure, record whether small random writes were interposed
between writing and reading. Without them, the value does not move however many times you read or
however long you wait. With them, it changes by nearly 1.8x. As the next section shows, 300 seconds
of sequential writes does not move it.

What breaks it is not writes but small random writes

This section was measured later. The wording above reads as "writes break it", and that was
too broad. On the same device, I separated the shape of the write from its duration.

Step	1 MiB sequential read	Against the reference
Reference (not broken)	2,364.78 MB/s	—
4 KiB random writes for 30 seconds	2,343.63 MB/s	-0.9% (not broken)
4 KiB random writes for 300 seconds	546.02 MB/s	-77% (broken)
After a sequential refill	2,364.18 MB/s	-0.03% (fully restored)
Sequential writes for 300 seconds	2,365.59 MB/s	+0.03% (not broken)

Sequential writes do not break it. 300 seconds of writing, 0.03%. What breaks it is small random writes only, not "writes" in general
30 seconds does not break it, 300 seconds does. The threshold is somewhere in between, and where is not known
The unbroken side does not depend on the specified IOPS. 2,364.78 at 100,000, 2,366.23 at 200,000
For the broken side, proportionality cannot be written. The 546.02 MB/s here (specified 100,000) and 1,272.8 MB/s on another deployment (specified 200,000) give a ratio of 2.33 — it moves in the direction of the specified IOPS but not by 2.00, and the two are not from the same deployment. No run varied the specified IOPS while in the broken state, so this is not a control experiment

In practice, record whether you "read after small random writes", not whether you "read after
writes". A workload that writes sequentially and reads sequentially is not subject to this.

The conclusion that you cannot size from a single point is unchanged. What changed is the
reason. Not "an unexplained 2.64x", but "even on the same configuration it moves 1.8x
depending on whether small random writes were interposed after the data being read was last
written".
Why the 3,401-3,408 MB/s environment came out higher still remains unexplained.

The warm-side ceiling sits on the client side

In the warm state, none of the
published utilization metrics
was at its ceiling. I used two EC2 clients to decide which side it stops on.

What was measured	1 MiB sequential read
Rocky 9.7 (concurrent)	2,040.69 MB/s
RHEL 9.7 (concurrent)	2,163.86 MB/s
Total	4,204.55 MB/s
One EC2 client alone (same state)	2,265.7 MB/s

The total is 1.86x one client. The ceiling sits on the client side. The storage side still had
headroom; it was stopping at what one client could produce.

What determines the per-client ceiling was not measured. It is about 16-17 Gbps per client, short
of the 50 Gbps of a c5n.9xlarge.

Before you write "I measured the storage ceiling", try adding an EC2 client.
If the total grows, what you were measuring was the EC2 client's ceiling.

The same shape appeared on a path that is not block. On 2026-09-19, on the NFS path (first
generation 2,048 MBps, SSD IOPS 40,000, two c5n.9xlarge reading 300 GiB that does not overlap),
1,195.27 alone gave a total of 1,901.48 MB/s, 1.59x. Alone, disk throughput utilization was
63% and not saturated; with two clients it reaches 102-103%.

There, the single-client side was followed up the next day, and it turned out to be two settings
rather than a resource. Transfer size (rsize 64 KiB to 1 MiB, +21%) and concurrency (8 to 16
threads, +14%) took a single client to 1,646.17 MB/s, at which point the file system side pins at
102.6%
(measurement, Japanese).

It was followed further (2026-09-20), and "the client side" split into parts. On second
generation 6,144 MBps / 200,000 IOPS with 4 KiB random reads, varying only how the working set is
divided, five ways:

Division	Against the reference	Disk side / specified 200,000
1 EC2 client / 1 SVM / 1 volume (reference)	—	60.7%
Threads doubled only	+8.9%	67.6%
2 volumes (same SVM)	+1.0%	66.8%
2 SVMs	+26.3%	83.1%
2 EC2 clients	+52.7%	96.6%

What acts on it is neither thread count nor volume count, but the number of IP addresses the client
talks to and the number of clients. With two clients the disk side reaches 96.6% of the specified
value, and only there does the file system side become the limit
(measurement, Japanese).

The same separation was not done for the 16-17 Gbps per client in this section. There is no
block-side measurement varying transfer size, concurrency and the number of peer IP addresses.
Had I followed it that far instead of stopping at "the client side", the block side might also
have turned out to be settings and division.

"Measuring with one client does not determine the kind of ceiling" holds across protocols, so the
procedure in this section is portable.

At least it is not concurrency

The candidate "there must not be enough outstanding requests" fell. In the unbroken state, I
varied thread count alone by 4x.

Threads	1 MiB sequential read	Response time (measured)
512	2,363.91 MB/s	216.588 ms
1,024	2,364.28 MB/s	433.283 ms
2,048	2,365.58 MB/s	860.258 ms

Throughput moves by 0.07% and only response time scales precisely. Applying 4x the threads does
not raise the per-client ceiling; only the time requests spend waiting goes up 4x.

This table is also where the hazard of iorate=max is most visible in this article.
"2,364 MB/s" is the same on all three rows, but the response time behind it spans 216 ms to
860 ms. Write throughput alone and that difference disappears.

The file-protocol side was the same. NFS 4 KiB random reads at 512 / 1,024 / 2,048 threads gave
94,949.6 / 97,915.3 / 97,014.5 IOPS (a spread of 2.2%), with response times of
5.391 / 10.456 / 21.108 ms (measured). That ceiling is not concurrency either.
Note that 94,949.6 is from a deployment with half the SSD capacity (4,096 against 8,192 GiB), and is
0.7% from the original measurement of 95,630. Between those two points, capacity was not
acting on it.

Do not cite those two figures as a same-conditions reproduction. Re-measured on 2026-09-20 with
a 600 GiB working set, the reference was 81,313.9, 14% below the figures above.
The original measurement did not record its working set, so the conditions cannot be matched.

Match the time of day when comparing configurations

My first comparison measured four configurations in sequence, across time. And as the section
above shows, a sequential read figure moves 1.85x with elapsed time since the write. I thought I
was changing the policy; I was changing elapsed time as well.

The same shape is not confined to policies. I once wrote a 23% difference for multipathing on versus off,
and that too measured the two at different times. When comparing configuration A against
configuration B, match the elapsed time from write to read, and if you cannot match it, it is better
not to compare.

On choosing between protocols

There is no protocol difference between iSCSI and NVMe/TCP at one connection. That conclusion
is easy to miss if you compare figures measured on different file systems, so the comparison is kept
below to show why.

Comparison	iSCSI	NVMe/TCP
Different file systems side by side (misleading)	1,135.18 MB/s	591.64 MB/s
As the documented procedure has it	1,135.18 MB/s	1,288.87-3,407.60 MB/s (3 deployments)
4 KiB random reads under that procedure	44,593 IOPS	126,964 / 128,154 IOPS (2 deployments)

Read on its own, the top row looks like "the ordering inverts at one connection." But those two
figures were measured on different file systems, and side by side the difference reads as a protocol
difference that isn't there.

I re-measured on one file system. A 600 GiB LUN and a 600 GiB namespace were placed together on
a single 1,800 GiB volume, leaving lun= as the only line differing between the parameter files.

One connection, same file system	1 MiB sequential read
iSCSI, 1 session	1,135.19 MB/s
NVMe/TCP, 1 I/O queue	1,135.88 MB/s
Difference	0.06%

The ordering does not invert. There was no protocol difference at one connection.
591.64 belongs to the deployment tier: one-connection figures appear at only two positions,
4.73 Gbps and 9.08 Gbps, and within one deployment iSCSI and NVMe/TCP appear at the same
position. The connect-all side gave 2,366.32 against the published 2,366.49, agreeing to 0.007%
(measurement, Japanese).

The question left over is what 591.64 was. In the single mode as published, all four workloads
landed between 591.5 and 591.7, the shape of hitting the ceiling of a single path. The
re-measured run does not pin there. The cause has not been confirmed.

No ratio is written. The as the documented procedure has it row still has its numerator and
denominator from different file systems, and the NVMe/TCP side scatters across deployments.
Computed from one pair it comes to "2.08x", but recomputed over the measured spread it is
1.14-3.00x. Pick one figure out of that spread and write "N times", and the picking is itself the
conclusion.

The iSCSI side cannot be treated as a fixed value either. For sequential reads, one session and
both portals agreed across two deployments (1,135.18 and 1,135.19, 0.01 MB/s apart), but two
points are not a distribution. And 16 sessions is still one deployment only. There is no
basis for "iSCSI is stable". The evidence is simply not there.

One thing can be kept as a comparison. 4 KiB random reads agreed to within 1% on the same two
deployments where sequential reads scattered (126,964 and 128,154). It is the sequential read
that scatters, and nothing else.

This was later corroborated. On the same deployment where a rewrite moved sequential reads by
1.85x, 4 KiB random reads gave 131,506 (warm) against 131,550 / 131,749 / 131,736 (cold), a spread
of 0.2%. Whatever moves sequential reads by 1.85x does not act on 4 KiB random reads.

4 KiB random reads do, however, scatter across deployments for a different reason (three tiers
observed: 131k / 203k / 242k). That cause could not be confirmed. Two things were established.

One is that 4 KiB random reads scale precisely with the specified SSD IOPS.
Only the specified value was changed, within one deployment.

Provisioned SSD IOPS	4 KiB random read	Ratio to the specified value
200,000 (before lowering)	131,413.5 IOPS	0.657
100,000 (after lowering)	66,016.9 IOPS	0.660
100,000 (created that way on another deployment)	65,070-65,947 IOPS	0.651-0.659
200,000 (after raising it on that deployment)	131,529.7 IOPS	0.658

The ratio going down is 1.990 and going up is 2.02. The measured value lands at about 66% of the
specified value, the same ratio at all four points. The 1 MiB sequential read in the same
states was 2,265.7 / 2,366.23 MB/s and does not change: sequential reads do not depend on the
specified IOPS.

The reverse control was taken differently, in the following run. The first attempt lowered from
200,000, and the request to restore it was refused with "cannot start until at least 6 hours have
passed since the last storage capacity update".
A provisioned IOPS change is treated as a storage capacity update, with a six-hour cooldown.
Billing continues while you wait, so the next run was created at 100,000 and raised to 200,000
partway through: in the increasing direction it takes one update and does not hit the cooldown.
An update stays in UPDATED_OPTIMIZING for about 18 minutes (about 17 minutes when lowering).
If you measure by lowering, plan to throw the environment away; if you need a round trip, design
it in the increasing direction.

The other is that what sets the spread between deployments is the disk read amplification factor.

The specified value was 200,000 on every deployment. What differed was how many disk-side reads one
client read becomes. CloudWatch retains 5-minute values for 63 days, so the intervals whose
average read I/O is 4 KiB can be pulled out and checked without rebuilding the environment.

Deployment	Client IOPS	Disk IOPS	Against the specified 200,000	Amplification (byte ratio)
High side	203,961	162,562	81%	96.5%
High side (another deployment)	206,582	172,158	86%	83.3%
Low side	131,617	195,905	98%	193.7%
Low side (another deployment)	130,080	195,941	98%	190.2%

The IOPS the client sees is the specified IOPS divided by that environment's amplification factor.
131,617 x 1.49 = 196,109; 203,961 x 0.797 = 162,557. Both check out.

At an amplification of about 1.5, the disk saturates at 98% of the specified value and the client stops at 131k
At an amplification of about 0.8, the disk has headroom at 81% and the client reaches 204k

The "about 66% of the specified value" from the proportionality above is the reciprocal of that
1.5. The same relation was being seen two different ways.

Here I nearly settled on a wrong explanation once. I thought "if it exceeds the specified value
it must be served from memory", but the way to decide that is the ratio of DiskReadBytes to
DataReadBytes (a few percent or less means it is served from cache). Applied for real it was
104-181% across all five deployments: none of them was being served from cache. The cause
was looking only at DiskIopsUtilization. It tells you whether the disk is saturated; it tells
you nothing about the relation to what the client sees.

If you measure, take four: DataReadBytes / DataReadOperations / DiskReadBytes /
DiskReadOperations. Taken in 5-minute intervals, the average I/O size lets you cut out workload
intervals after the fact. You do not have to rely on the timestamps in your own notes.

So what is the reason the amplification factor splits between 0.8 and 1.5? This is what remained to
the end.

Here another of my hypotheses was disproved. I thought "the layout above must also be setting the
amplification factor", but even right after a sequential refill had restored the layout to 88 KiB,
random read amplification stayed at 190-208%. The layout for sequential reads and the
amplification factor for random reads are separate variables.

There was one candidate. Line up the five deployments by "did 4 KiB random writes ever run on that
device" and they split cleanly.

Deployment and interval	4 KiB random write history	Amplification	Client IOPS
Never ran (a run that measured reads only)	none	96.5%	203,961
An interval before it had run	none	83.3%	206,582
After it ran	yes	170.1%	140,423
After it ran	yes	193.7%	131,617
After it ran (a sequential refill does not restore it)	yes	190-208%	97,769-130,080

All five followed that ordering. Even so, this is an ordering of observations, not a control
experiment. I wrote it down as something that would be settled by applying the steps in turn on a
single device.

And that candidate was disproved too

I applied it. It was disproved. On a new deployment I began measuring right after a fill, with
4 KiB random writes never having run.

State	4 KiB random read	Against the specified 100,000
Right after a fill on a new file system (no random write history)	65,947.6 IOPS	0.659
After 300 seconds of 4 KiB random writes	65,430.8 IOPS	0.654
After a sequential refill	65,070.0 IOPS	0.651

It is already 0.659 with no history. Where the "none" rows in the table above were 0.83-0.97,
the same "none" condition produced a value on the 1.5 side. The spread across the three states is
1.3%, and the amplification factor moves with neither layout nor write history.

Five observations lining up cleanly was a coincidence. That is the third disproved hypothesis.

An ordering of observations is not a substitute for a control experiment. All five followed it
without exception, the mechanism was plausible, and it was wrong. When you write "all n cases
follow" as a basis, check at the same time how many unvaried variables were lining up in the same
direction. In these five, the patch version and the client were lined up the same way too.

What remains is those two. The difference between the 0.83-0.97 of F-6 / F-7 and the 1.5 side
later on comes down to the ONTAP patch version (9.18.1P6 against 9.18.1) and the client used
for the measurement. Neither was varied within a single run, so they are named as candidates and
nothing more.

One of those two cannot be varied by a reader either. CreateFileSystem's
FileSystemTypeVersion
is documented as being for FSx for Lustre, and FSx for ONTAP has no parameter for specifying a
version. The list of updatable properties in
Updating file systems
does not include the ONTAP version either. So the patch version can be neither chosen nor
pinned, and a control experiment varying that variable cannot be assembled on the user side.
The client is the only one that can be varied. If you want to press on here, what you can do is
show that the amplification factor does not move when the client is changed, narrowing the
candidate to the patch version. It does not establish causation.

The policy, incidentally, had nothing to do with it. Switching queue-depth to numa to
round-robin to queue-depth during a 4 KiB random read produced no step, and against the 132,044.8
IOPS average of the run containing all three policies, queue-depth alone gave 131,413.5 IOPS —
a difference of 0.5%.

Five things I hit on the instrument and the teardown

It took nine failures to reach a measurement. All of them gates, none of them numbers.
Four of those happened before reaching a mount, so they are in a
separate article
(A: sessions and queues); what belongs here are the five on the instrument and the teardown.
The detail is in the
block measurement runbook
as a table of symptom, cause and how to check.

vdbench resolves include= against the current directory, not against the path given to -f. The fill "completed successfully without writing anything". Had I gone on without noticing, I would have been measuring reads against unwritten blocks (which return zeros).

6. key=value on a vdbench command line is not an override of the workload definition.
It is an assignment to a placeholder inside the parameter file. Passing xfersize=1024k failed with
Unused parameter substitution.

I reported an SSM wait limit as a failure. A 600 GiB write did not fit inside a 7.5-minute
wait, I saw status: InProgress, judged it a failure and tore the environment down.
A wait limit is not a failure.
A sentinel string matched my own explanatory text. Against an implementation that "prints
MISSING if the module is absent", I was deciding with grep -q 'MISSING', and it matched the
explanatory line "A MISSING line means…" in the same output, so F-2 was skipped although NVMe/TCP
was available. A sentinel that your own explanatory text satisfies is not a sentinel.
The teardown trap died partway through the teardown. Under set -euo pipefail, reading
describe-stacks after a delete always fails once the stack has finished disappearing, and that
failure kills the trap itself. The delete ran but the retry did not, and a DELETE_FAILED stack was
left behind. Put the delete call first in the trap, and set +e immediately after it.

Three more hit during the rewrite

The nine above are from the first measurement. The run that recounted things per path added three
more. All of them the same shape as the nine: looking successful while measuring nothing, or
not being able to tell that nothing was measured.

A counter table returned empty. nvmf_lif had zero rows even after 600 GiB had been written,
and reading zero rows as zero bytes gets you the right conclusion for the wrong reason. Which
table to look at, and the detail of that trap, are in the separate article
(B).
A socket aggregation matched my own metric. To sum bytes per destination I treated any field
ending in :4420 as a destination, and bytes_acked:4420 (a byte count) happened to end in
4420, growing one spurious path named bytes_acked. The value was 0 so the sum was not broken, but
it is the same mistake as the eighth.

12. "It passed locally so it will pass in CI" did not hold. This is the repository side rather
than measurement, but the structure is the same. A check scans tracked files, and a new file is
outside the scan while it is untracked. Green locally, red in CI. "Check after the last edit" is
not enough; it is also "after the last git add".

Summary

The same configuration varied by 2.64x across environments. Do not size from a single measured value
What is acting on it is the layout on disk. One disk-side read goes from 87-89 KiB to 11.7 KiB (a factor of 7.5), disk IOPS rises 7.3x and saturates at 89-95% of the specified value. DiskRead / DataRead is about 100%, so the faster side is reading from disk too. It was not being served from memory
What breaks it is small random writes only. Three reads in a row, a spread of 0.2%; 30 minutes idle, within 0.4%; 300 seconds of sequential writes, 0.03%. 4 KiB random writes cost 0.9% at 30 seconds and 77% at 300 seconds, and a sequential refill restores it. It goes back and forth within a single run. Where the threshold sits between 30 and 300 seconds is not known
The warm-side ceiling sits on the client side, and it is not concurrency. Two clients concurrently total 4,204.55 MB/s, 1.86x one client. Meanwhile 512 to 2,048 threads moves throughput by only 0.07% and only scales response time from 216.6 to 860.3 ms. NFS 4 KiB random reads are the same, a spread of 2.2% (5.391 to 21.108 ms). Before you write "I measured the storage ceiling", try adding an EC2 client. Adding threads will not tell you
4 KiB random reads scale precisely with the specified SSD IOPS (about 66% of the specified value, the same ratio at four points). It holds in both the decreasing and the increasing direction. Sequential reads do not depend on the specified IOPS
What splits IOPS between 131k and 204k across deployments is the disk read amplification factor. Client IOPS = specified IOPS / amplification factor. DiskIopsUtilization alone will not tell you. Take DataReadBytes / DataReadOperations / DiskReadBytes / DiskReadOperations. What sets that amplification factor is not known. That it is neither layout nor write history is settled; the remaining candidates are the ONTAP patch version and the client. The patch version cannot be specified on FSx for ONTAP, so a control experiment varying that variable cannot be assembled on the user side
Observation order is not a substitute for a control experiment. I had a candidate that five cases followed without exception; I aimed a measurement at it and it was disproved. Unvaried variables were simply lined up in the same direction
When comparing configuration A against B, match the elapsed time from write to read. I compared without matching it, wrote "1.73x from the policy", and withdrew it later
Which protocol comes out ahead changes with the procedure and the I/O shape. Publish which procedure measured what, not which one is faster
Do not write throughput alone. The figures in this article are iorate=max saturation points, and the same 2,364 MB/s sits at a 216 ms point and at an 860 ms point. Without response time alongside it, that difference disappears

Notes on the AWS documentation

This article contains no remarks about the documentation, and there is nothing to submit.

What it contains are errors in my own way of measuring and measurements that the published
specification cannot account for. Neither is a claim that the documentation is wrong, so I am
setting them out separately.

Content	Kind
2.64x across the same configuration / 1.85x from layout	Measurement. The published specification does not promise this granularity
The cause of amplification 0.8 against 1.5 is unknown	Left unresolved. The candidates are the ONTAP patch version and the client, and the former cannot be chosen on FSx for ONTAP, so no control experiment can be assembled
The per-client ceiling (block side)	Not measured. On the NFS side it was followed through on 2026-09-20 (the table above), so the same way of varying things should apply to the block side
Comparing policies across time, writing 1.73x, and withdrawing it	My error
`nvmf_lif` returning no rows	ONTAP-side behaviour. A question for the vendor, not a matter of AWS documentation

The remarks about the documentation are in the other two: A and
B, each with its own submission status.

Submission status across the three block-protocol articles and S3 Burst Part 2 is collected in
AWS documentation notes and inquiries across the block-protocol articles.

Primary record of the figures

The primary record of the figures and their conditions is in
per-protocol measurement results (Japanese),
and the reproduction procedure is in the
block measurement runbook (Japanese).
Whatever is unconfirmed is written as unconfirmed, so please do not drop that when citing.

Pitfalls with the VDBENCH measurement tool, and items to always record, are collected in
Considerations when measuring FSx for ONTAP's block protocols.

Moving on-premises block storage to AWS? The "divide by 625 MBps" sizing approach comes out wrong

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 30 Sep 2026 05:43:26 +0000

Moving on-premises block storage to AWS? The "divide by 625 MBps" sizing approach comes out wrong — measured session and queue counts

What this article does and does not cover: it covers the measured failure of the "divide by
625 MBps" sizing approach, and four configuration failures hit before mounting. It does not
cover performance conclusions, a comparison against file protocols, or distributing block storage
to multiple sites (details under "What this article does not cover" below).
All figures are from a Single-AZ configuration; Multi-AZ is unverified.

When using FSx for ONTAP over iSCSI or NVMe/TCP, how many sessions should you open? AWS's
procedures surface one number: "Amazon EC2 single client maximum of 5 Gbps (~625 MBps)",
which appears on both the
iSCSI and
NVMe/TCP pages as
the lead-in to the procedure for adding sessions.

One thing first: AWS does not instruct you to divide. The source text says "if you want
throughput beyond this value, follow the EC2 network bandwidth page's procedure to add
sessions," and it's marked (Optional). No formula is given. "required bandwidth ÷ 625 =
session count" is a sizing approach I built after reading that figure. This article tests
my sizing approach, not a formula AWS published.

Measuring it, that sizing approach missed in both directions. 1.82x with one session, 0.20x
with sixteen. And going from one session to two moved sequential throughput by zero bytes.

So this figure can't be used as a divisor to determine session count. What's documented as a
single flow's ceiling in EC2 network bandwidth
turned out to be a separate thing from what a block protocol session measures per session.

NVMe/TCP has nothing equivalent to nr_sessions; the analogous quantity is queue count. That
doesn't honor a request either — asking for 36 produced 4.

What was measured

Assuming no prior knowledge of FSx for ONTAP, here's where this article's measurement happens.

NetApp ONTAP (second-generation SINGLE_AZ_2) file system over either iSCSI (multiple sessions) or
NVMe/TCP (multiple queues)" width="800" height="614">

Figure: what this article measures is the connection count between one EC2 client and one FSx for
ONTAP file system, not multiple clients pooled together.

The figure states the same thing as the table below. What this article measures is the
connection count between one EC2 client and one FSx for ONTAP file system, not multiple
clients pooled together.

Item	Role in this article
VPC / AZ	Single AZ, no network detour, so what's measured is the protocol itself
EC2 client	Varies session/queue count and measures 1 MiB sequential reads
FSx for ONTAP	Provides the iSCSI LUN or the NVMe/TCP namespace
Connection count	For iSCSI, TCP sessions counted with `ss`; for NVMe/TCP, requested vs. effective queue count

What this article does not cover. No performance conclusion is drawn; the numbers here only
serve as evidence that the division misses. The same configuration swings 2.64x across
environments, so don't use these figures for sizing. Why, and what actually drives the numbers,
is in a separate article.
There's no comparison against file protocols (NFS/SMB) either — there's a reason they don't belong
in the same table. And there's no discussion of distributing block storage to multiple sites — this
architecture's distribution layer (FlexCache) distributes volumes, not LUNs or namespaces.

This is one of a set of three, but there's no order to read them in. The block protocol
measurements are split by reader.

Article Reader

A Session and queue counts (this article) Anyone about to mount

B Checking whether ANA is available Anyone who followed the procedure and got stuck

C What moves the numbers Anyone measuring or citing figures

None assumes the other two. Read only what you need.

Measurement environment

Item	Value
File system	FSx for ONTAP second-generation SINGLE_AZ_2, 6,144 MBps x1 HA pair, SSD 4,096 GiB, provisioned 200,000 IOPS
Region	ap-northeast-1, single AZ
ONTAP	9.18.1P5 (the three divisor-table data points). The queue negotiation section used a separate deployment, and its version was not recorded — it wasn't read at measurement time and can't be recovered after the fact. I can't write "all figures in this article are the same version." (Multiple versions coexisting across this configuration is covered in the separate article (C). The cross-article version mapping is in the ONTAP version matrix.)
NVMe read cache	Disabled (confirmed on both nodes)
LUN / namespace	600 GiB, on a 900 GiB volume
Before measuring	Wrote the full 600 GiB once. Thin-provisioned unwritten blocks return zero, so reading without writing first measures nothing
Client	c5n.9xlarge; 50 Gbps network is the guaranteed value
Measurement tool	VDBENCH 5.04.07 direct, `iorate=max` (unlimited), 512 threads, 60s warmup + 300s measurement
Test data / efficiency settings	Not recorded whether the VDBENCH payload was compressible, or whether volume inline efficiency (`StorageEfficiencyEnabled`) was enabled. On the file-protocol side, efficiency settings measurably affected figures; whether the same applies to block is unconfirmed

Because the window is 300 seconds, this article's figures include bursts. They can't be cited as
a baseline.

This article's figures are the saturation point of iorate=max, not an operating point in
production. No target IOPS was given — this is however much could be pushed through. On the
file-protocol side of the same repository, giving a target of 4,400 IOPS produced 4,406 MB/s at
8.86 ms, while unlimited produced 4,204 MB/s at 121.79 ms. Unlimited is 5% lower
throughput and 14x the response time.

With threads fixed at 512 and unlimited, response time is an identity of 512 / IOPS (confirmed
against a separate measurement: 216.6 ms derived versus 216.588 ms measured). The values below
can be re-read through that identity — 1,135.18 MB/s is about 451 ms, 1,970.43 MB/s is
about 260 ms. Both are derived values, not recorded measurements.

The divisor division doesn't hold

Over iSCSI, I varied the counted number of TCP connections and measured 1 MiB sequential
reads. These are actual counts from ss, not the requested value.

TCP counted	Measured	Predicted from 625 MBps	Measured / predicted
1	1,135.18 MB/s	625 MB/s	1.82x
2	1,135.18 MB/s	1,250 MB/s	0.91x
16	1,970.43 MB/s	10,000 MB/s	0.20x

With one session it's faster than predicted; with sixteen it's a fifth of the prediction.

And going from one session to two moves sequential throughput by zero bytes — 1,135.18 appears
twice in a row. So the ceiling hit with one session is not a single flow's capacity.

1,135 MB/s is 9.08 Gbps. In the same environment, the file protocols landed near the documented
5 Gbps on a single connection — NFS 591.62 MB/s (4.73 Gbps), SMB 574.24 MB/s (4.59 Gbps).
A single iSCSI session carries roughly twice that. Why is unconfirmed. (This comparison is
against single-connection figures from other protocols in the same environment, not a direct A/B
test against iSCSI.)

Here's the split between what's documented and what's an unverified candidate. The single-flow
5 Gbps ceiling itself is documented. AWS defines a single flow as "a TCP or UDP flow identified
by a unique 5-tuple (source IP, destination IP, source port, destination port, protocol)" and
states that outside a cluster placement group, a single flow is limited to 5 Gbps
(Amazon EC2 instance network bandwidth).
The same page documents the workarounds too — up to 10 Gbps inside a cluster placement group,
up to 25 Gbps within the same AZ with ENA Express, or spreading traffic over multiple paths with
Multipath TCP (MPTCP)
(ENA Express).

From here on, these are unverified candidates. Whether this measurement environment was
inside a cluster placement group was not recorded and is unconfirmed. 9.08 Gbps would fit inside
a 10 Gbps allowance, but whether this configuration was inside that allowance is unknown.
Whether iSCSI, NVMe/TCP, NFS, and SMB all count as "one" the same way against the 5-tuple
definition is also unconfirmed. ENA elastic network interfaces are documented as having up to 8
RSS queues
(Optimize network performance on EC2 Windows instances —
that page covers Windows; I haven't checked the equivalent for Linux), and if protocols
distribute across CPU cores differently, the apparent ceiling could differ as a result. This
article does not test these candidates. A separate measurement would be needed.

The requested queue count doesn't hold

NVMe/TCP has nothing equivalent to nr_sessions. It opens a TCP connection per queue, so the
quantity that corresponds to session count is queue count.

Requesting via --nr-io-queues produced this. NCQA (Number of Completion Queues Allocated) and
NSQA (Number of Submission Queues Allocated) are the actual completion and submission queue
counts the NVMe Get Feature command reports back to the host.

Mode	Requested queue count	Effective NCQA / NSQA	Controller count
Single flow	1	1	1
AWS's procedure as written (one `connect-all`)	unspecified	4	2
Explicit queue count	36 (vCPU count)	4 (unchanged)	2

Requesting 36 produced 4 regardless.

This isn't speculation — it's documented on the ONTAP side. vserver nvme subsystem show's
description of -default-io-queue-count states "The actual value used when a connection is
established may vary depending on the host and transport protocol used," and
vserver nvme show-host-priority
states that I/O queue count is determined per combination of node, transport, and host
priority. NetApp's KB has an example where setting 15 on a subsystem produced 2 on the host
side.

Don't use the example's value as your own environment's value. In the example above,
nvme-tcp / regular is 2, but in this environment it was 4.

As a byproduct, one thing left as "unconfirmed" in planning got resolved. queue_count includes
the admin queue and matches NCQA + 1 (1 -> 2, 4 -> 5). With one queue, there was also exactly
one established TCP connection carrying I/O.

Four failures hit before mounting

Nine failures were hit before reaching a measurement. All were gates, not numbers. Each one
looks like it succeeded while measuring nothing. Four of those happened before mounting, so
they're here; the rest concern the measurement tool and teardown and are in the
separate article (C:
what moves the numbers). Details, as a table of symptom / cause / how to check, are in the
block measurement runbook.

JunctionPath turned out to be required. I had assumed it wasn't needed since a LUN isn't read through a junction, but the API rejects that. Worse, the property reference's entry for JunctionPath says This parameter is required. in the body while the same property's Required: column says No. That's a literal self-contradiction (still present as of 2026-09-17). Trust the API. It fails 25 minutes after file system creation starts.

The same page has an example of writing this without contradiction. StorageEfficiencyEnabled
uses Required: Conditional and explains the condition in the body (required when the volume is
RW). Conditional is a value this reference already uses, so JunctionPath could take the
same form.

I can't state exactly which condition makes it required. What I measured is one case,
OntapVolumeType: RW, where it was required. I did not test whether it's optional for
DP. Determining the condition belongs to whoever holds the API's implementation;
what I can report stops at "the body and the Required: column disagree."

2. AL2023 does not create /etc/nvme/hostnqn. AWS's procedure assumes this file exists and
reads it with cat (RHEL creates it). Generating it with nvme gen-hostnqn was necessary.

3. iSCSI login is asynchronous. iscsiadm --login returns before iscsid actually connects. In
this environment it took about 100 seconds. Counting with ss right after returned 0, so
every alias and fill attempt right after also came up empty. "The command returned" is not
"the connection is up."

4. find_multipaths yes does not create a map for a single-path device. When intentionally
measuring a single flow, the path count is 1 by design, and with the default setting
/dev/mapper/<alias> never appears. Explicit registration via multipath -a <wwid> was
necessary. That wwid is also more reliably read from the device with scsi_id than assembled as
3600a0980+serial (assembling it is useful as a cross-check, not as the primary source).

Summary

Sizing session count by dividing required bandwidth by 625 MBps missed in both directions. 1.82x with one session, 0.20x with sixteen.
Going from one session to two doesn't move sequential throughput. The ceiling hit with one session is not a single flow's capacity
NVMe/TCP queue count is the result of negotiation. Requesting 36 produced 4. Read the effective value with nvme get-feature --feature-id 7. queue_count includes the admin queue, so it equals NCQA + 1
All four configuration failures looked like success while measuring nothing. JunctionPath is required, AL2023 doesn't create hostnqn, iSCSI login is asynchronous at about 100 seconds, and find_multipaths yes doesn't map a single-path device.
Don't use this article's figures for sizing. The same configuration swings 2.64x across environments
What drives that 2.64x swing is out of scope for this article. It's covered in a separate article

AWS documentation notes and support inquiries

This article touches two points in the documentation. One has been filed; the other is not
being filed.

Finding	Source	Status
`JunctionPath`'s body and `Required:` column disagree. Traced to the API reference, which propagates into CloudFormation and the CDK	same	Filed (2026-09-17). AWS's reply (2026-09-22): explained that when OntapVolumeType is RW, JunctionPath is required; when DP, it cannot be specified at all — so "Required: No" has a rationale when read across both volume types combined. The proposed change to `Required: Conditional` was described as feedback to be shared with the internal team, with no commitment on adoption. The question of whether the API reference is upstream of CFN/CDK went unanswered ("internal AWS information, cannot disclose"); instead AWS said it would file feedback against the API reference, CFN, and CDK pages individually
Sizing by dividing 5 Gbps (~625 MBps) doesn't match measurement	iSCSI / NVMe/TCP procedures	Not filed. AWS never states a formula, so this isn't a documentation error

Why the second one isn't filed. This is a record of my own sizing approach missing, not a
finding about the documentation. The figure itself is correctly documented as a single flow's
ceiling; I was the one who used it as a divisor for session count, and that's where it broke
down.

The first was filed with AWS Support on 2026-09-17. The submission included the observed
requirement (omitting it fails stack creation 25 minutes in) and the fact that the CDK inherits
it as an optional type, requesting a fix on the API reference side.

AWS's 2026-09-22 reply settles only the rationale behind "Required: No" (it holds when RW and
DP are read as one combined attribute). It doesn't change the underlying fact that JunctionPath is
required when creating an RW volume, and whether Required: Conditional will actually be adopted
remains unconfirmed.

Primary record of the figures

The primary record of the figures and conditions is in the
per-protocol measurement results
(Japanese); the reproduction steps are in the
block measurement runbook
(Japanese). Anything marked unconfirmed there is marked unconfirmed for a reason — don't drop
that qualifier when citing it.

Considerations before sizing session and queue counts, and pitfalls hit before mounting, are
collected in
Considerations when measuring FSx for ONTAP's block protocols.

Submission status across the three block-protocol articles and S3 Burst Part 2 is collected in
AWS documentation notes and inquiries across the block-protocol articles.

Amazon S3 Files vs. Amazon FSx for NetApp ONTAP: two designs for using S3 as a file (S3 Burst Part 2)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 30 Sep 2026 05:43:06 +0000

Amazon S3 Files vs. Amazon FSx for NetApp ONTAP: two designs for using S3 as a file, measured on the same host (S3 Burst Part 2)

Keeping the primary copy of your data in one place while reaching it both as a file (NFS/SMB)
and over the S3 API: have you had to weigh which design to pick before committing to one?

There is more than one way on AWS to combine the two. Worth pausing on this, though: every
design has a point where "fast enough" turns into a ceiling, and that point sits in a different
place for each one. It helps to know where each one bottlenecks before choosing. This article
measures three of them from the same host.

The same configuration as the image above, reproduced so it reads in environments where images
don't render.

flowchart LR
    subgraph AWS["AWS Cloud"]
        subgraph VPC["VPC, single AZ"]
            EC2["EC2 client"]
            subgraph A["A: This architecture"]
                AP["S3 Access Point"]
                FSXN[("Amazon FSx for<br/>NetApp ONTAP")]
                AP --> FSXN
            end
            S3["Amazon S3<br/>(bucket)"]
            S3FILES["Amazon S3 Files<br/>(via efs-proxy)"]
            EC2 -->|"write over S3 API"| AP
            FSXN -->|"read over NFS / SMB"| EC2
            EC2 <-->|"read/write over S3 API"| S3
            EC2 <-->|"read/write over NFS"| S3FILES
        end
    end

The figure states the same thing as the table below. Only A uses a different protocol for
writes versus reads; B and C have the same client reading and writing over the same protocol.
The figure's three panels map directly to the three below.

	Path	Write	Read
A	FSx for ONTAP's S3 Access Point (this architecture, below)	S3 API	NFS / SMB
B	Amazon S3	S3 API	S3 API
C	Amazon S3 Files (an S3 bucket mounted over NFS)	NFS	NFS

Only A uses a different protocol and client for writes versus reads. That's the substance of
this architecture: something written over the S3 API is read over NFS/SMB with no copy job in
between. That's why the figure shows two arrow types for A, while B and C are a single
bidirectional arrow — the same client reads and writes over the same protocol in B and C.

Measure throughput and lay it out in a table, and the three numbers sit side by side. Sitting side
by side reads as "this one is faster," but each of the three was bottlenecked for a different
reason. Under each panel of the figure is where that path was actually bottlenecked. A throughput
value provisioned on the file system, client-side network bandwidth, the CPU of a process running
on the client. What the bottleneck is matters more to design than what the ceiling number is.

The measuring side has its own prerequisite. There are five settings that, left at default,
measure something other than what you intended. That's covered in the
previous article (Part 5, "five defaults that can invalidate a measurement").
This article's figures were measured after changing all five from default.

Scope of this article

What it covers

The three ceilings that bind on the FSx for ONTAP side, and which one actually applied
What bottlenecks each of the three paths
What FlexCache does for reads
What becomes the ceiling as client count scales to 8
Whether a 15-minute sustained write decays
The bottleneck shift with small objects
What each tool can and can't observe
What hasn't been measured yet

What it does not cover

Which configuration is faster. The three figures in this article mix a provisioned ceiling with an unprovisioned elastic ceiling. They aren't laid out to be read side by side for a winner, so wherever a number appears, the type of ceiling it's hitting is stated alongside it.
How to choose a configuration itself. Where to place the read side, which side holds the primary copy, when you want writes visible on the bucket side — these are decided by other factors and throughput numbers alone don't decide them.
The semantics of Amazon S3 Files propagation. That file-to-bucket propagation converges within roughly a 60-second window, and the reverse direction is on the order of seconds, is a different subject and belongs in a separate article.
IAM policy and the authorization model. The AWS side of an S3 Access Point (IAM identity policy, access point policy, VPC endpoint policy, SCP) and ONTAP's two-layer authorization are a different axis from this article's subject (throughput). Minimum-privilege policy examples are in the S3 AP design guide's authorization design section.

Measurement environment

Figures can't be reproduced without the environment they came from, so it's placed first.

Item	Value
Measured on	2026-09-01 and 2026-09-02
Region	ap-northeast-1
ONTAP	9.18.1P3D1 (the second-generation NFS 2-host measurement used 9.18.1P6. The cross-article version mapping is in the ONTAP version matrix)
File system	SINGLE_AZ_1 (first-generation Single-AZ). Measured at two throughput capacities, 128 MBps and 2048 MBps
SSD	1024 GiB, `DiskIopsConfiguration` at two points, 3,072 and 40,000
Client	c5n.9xlarge (single-client measurements), c5n.2xlarge x8 (client-count scaling test — adding one client at a time and observing how the combined throughput grows)
Volume	UNIX security style, NFSv3, `actimeo=0`
Test data	Uncompressible (from `/dev/urandom`). Zero-filled data was not used. These are figures from a world with efficiency features off
AZ configuration	All figures are from a Single-AZ configuration. Multi-AZ is unverified

Both NFS and SMB were measured, but the scope differs. The per-protocol comparison table and
per-path bottlenecks are measured over NFS. What was measured over SMB is the growth pattern with
increasing client count, the 15-minute sustained write, and Multichannel channel count. The
seven scenarios under cache control were not measured over SMB. This architecture can be read
over either NFS or SMB, but this article's figures are all NFS. If you plan to read over SMB,
the figures below don't directly apply. What carries over is the measurement approach and the
procedure for isolating which type of ceiling is binding.

Reproducibility under identical conditions was 0.02-0.14%. With that baseline established, a
"moved 30%" statement later in this article can be said not to be run-to-run noise.

Unless otherwise noted, figures in the body are single measurements. Where something was
measured twice, both values are shown as 415.1 / 415.7 rather than averaged or taking a median.
Totals in the client-count tables are the sum of each host's value.

And every figure in this article is a saturation point — however much could be pushed through at
a fixed concurrency. No point gives a target throughput or target IOPS. This distinction
matters as much as stating which type of ceiling applies — a saturation point answers "where does
it choke," not "what response time does it deliver at that load." Measuring both on a separate file
system (second-generation, 6,144 MBps), a target of 4,400 IOPS produced 4,406 MB/s at 8.86 ms,
while unlimited produced 4,204 MB/s at 121.79 ms. Unlimited is 5% lower throughput and 14x
the response time. Source:
the series with a varied target value
(Japanese).

Response time exists for the S3 API path and not for the file path. The S3 API side's body
notes p50 alongside throughput (363.7 ms at concurrency 1, 1054.9 ms at concurrency 64, and so on).
The file path's load generator didn't retain the values it output, and the environment has since
been torn down. Because concurrency was fixed, it can be re-derived through the identity
concurrency / IOPS, but that's a derived value, not a measurement.

A separate article pushes further on the same property on the block protocol side
(separate article). It
covers a case where quadrupling thread count moves throughput by only 0.07% while response time
alone quadruples, and the same 2,364 MB/s appears at both a 216 ms point and an 860 ms point.
That same caution applies when citing this article's figures.

The block protocol side is a set of three.
Session and queue counts
(for anyone about to mount),
checking whether ANA is available
(for anyone who followed the procedure and got stuck), and
what moves the numbers
(for anyone measuring or citing figures). There's no order to read them in.

The measurement host was c5n.9xlarge. The reason is isolation. With an "Up to"-labeled instance,
once a measured value plateaus, you can't tell whether you hit the storage-side ceiling or the
client-side bandwidth ceiling. c5n.9xlarge's 50 Gbps is a guaranteed value. Every figure below
sits well under 50 Gbps, so the possibility that the client-side bandwidth was the cause of a
plateau can be ruled out.

Only the client-count scaling test used eight c5n.2xlarge instances instead, since that test wants
more clients rather than a bigger single client.

The client-count scaling test and the sustained write used a separate file system

The table above is first-generation. Only the client-count scaling test and the 15-minute
sustained write were measured on a separate, second-generation file system. The first-generation
one is a shared environment, and raising or lowering its provisioned throughput capacity would
affect other measurements.

Item	Value
Measured on	2026-09-05
File system	SINGLE_AZ_2 (second-generation Single-AZ), 6,144 MBps x1 HA pair
SSD	4,096 GiB, provisioned 200,000 IOPS
Client	c5n.2xlarge at 1 / 2 / 4 / 6 / 8 hosts. 64 threads and `nconnect=16` fixed per host

Because the environment differs, these figures can't be placed in the same column as the table
above. Load per host was fixed and only the host count was varied, because fixing total thread
count and dividing by host count would change two things at once per data point, making it
impossible to separate which one caused the effect.

Six settings to change from default before measuring throughput (Part 5)

Each one, left at default, measures something other than what you intended. And the number
that comes back doesn't look unnatural, so measuring alone won't reveal it. The table and how each
was noticed are in Part 5; only the names are listed here.

Linux's NFS mount opens only one TCP connection by default (plateaus around 590 MB/s)
Data created with dd if=/dev/zero (all-zero blocks never reach disk)
Volume inline efficiency (compression/deduplication) enabled
DiskIopsConfiguration at AUTOMATIC
Reads slightly exceeding the read cache's capacity

More on item 5 — the read cache's two-tier structure

Item 5 is covered in detail because it bears directly on this article's figures.

FSx for ONTAP's file server has two tiers of read cache: in-memory and NVMe
(performance specs). For
ap-northeast-1's first-generation Single-AZ, what's published differs by tier.

Tier	Capacity at 2048 MBps	Source
In-memory	256 GB	Performance specs, the "other regions" table
NVMe read cache	Not stated	No column exists in that same table

For NVMe, only a condition is stated in a separate section: "A Single-AZ 1 created on or after
2022-11-28 with a throughput capacity of 2 GBps or more gets a read cache." This file system meets
that condition. Checking with the ONTAP CLI directly confirmed it was enabled.

128 MBps configuration : system node external-cache -> 0 records
2048 MBps configuration: is_enabled: true (both nodes)

And how SSD IOPS gets used is stated explicitly in the same documentation. SSD IOPS is used only
when reading data that's in neither the in-memory cache nor the NVMe cache.

With that in mind, the measurement can be read this way. A 280 GiB read exceeds the in-memory
cache's 256 GB (= 238 GiB) by 18%, but there's an NVMe read cache behind it. This
configuration's NVMe capacity isn't published (in the four regions with a column, it's 1,900 GB at
the same 2048 MBps). The measured 98.5-99.9% of read bytes not going to disk is consistent with
coming back from these two tiers (breakdown under "Per-tool visibility" below).

That's why the observation "raising SSD IOPS from 3,072 to 40,000 made reads 7.14x faster"
didn't square with the documentation. If it's coming back from cache, SSD IOPS shouldn't matter.

That mismatch went away on re-measurement (2026-09-17). Setting the working set to 512 GiB —
2.15x the in-memory cache's 238 GiB — and disabling the NVMe cache, re-measuring showed
96-126% of read bytes coming from disk, with the low end saturating at DiskIopsUtilization
112%. "SSD IOPS determines the disk throughput level" is consistent with measurement under
this condition. So the mismatch's cause wasn't the documentation — it was my measurement:
a 280 GiB working set only exceeded the cache by 18%.

The ratio itself changed, though. 256.47 -> 1,246.67 MB/s, 4.86x. 7.14x doesn't
reproduce. And what's capping the high side, around 1,250 MB/s, is none of the published
utilization metrics (disk throughput 67% / IOPS 43% / network 43%). It isn't concurrency
either — going from 8 to 32 to 64 threads drops it from 1,071 to 999 to 958, with response time
alone scaling proportionally (6.9 -> 31.0 -> 59.3 ms). This is left unresolved.

To check in your own environment, the order is two steps. Check for an NVMe cache with
system node external-cache, and then read a volume that reliably exceeds the sum of both
tiers in one pass. A first-generation Single-AZ at 2 GBps or more gets an NVMe cache by
default — there's no way to prevent it from being created, only to create it and then disable
it. I forgot this once and measured four points with it left enabled.

Item 6 — SMB Multichannel disabled by default

ONTAP's SMB Multichannel is disabled by default. This is a separate setting from SMB 3.x
itself, so a channel can stay at one even while negotiated at dialect=3.1.1. That state held in
this measurement environment: while one host was reading, only one established TCP connection to
the SVM existed.

Enabling it doesn't reach connections already established. Enabling only affects new
connections. Removing and re-mapping the drive kept the same local port, and it only became four
channels after restarting the SMB client service.

max_connections_per_session is a separate value. With Multichannel disabled, setting it to 32
still yields one channel, and you can misread "raised the limit but it didn't increase" as a
result.

The mechanism, sources, and where to read channel count are in
SMB Multichannel is disabled by default, and enabling it doesn't reach already-established connections
(Japanese).

Three ceilings that limit writes on the FSx for ONTAP side

Before getting into the three-path comparison, here's what limits this architecture's own side.
There's a ceiling that doesn't move even when throughput capacity is raised, and not knowing it
leads to a sizing miss.

One term first. Throughput capacity is a value — 128 / 256 / 512 / 1024 / 2048 MBps and so on —
that you set on the file system. It isn't metered usage; you're billed by time against the
value you set. This article measures at two points, 128 MBps and 2048 MBps.

For ap-northeast-1's first-generation Single-AZ, three ceilings bind on writes, and the one that
actually applies is whichever is smallest.

Ceiling	This environment's value	How it's determined
Provisioned throughput capacity	128 / 2048 MBps	Set by you
Disk throughput out of SSD	768 MBps (768 MBps/TiB x 1 TiB, when SSD IOPS is `AUTOMATIC`)	Driven by the provisioned SSD IOPS
Per-HA-pair ceiling	Write 750 MBps (read is 2,048 MBps)	Determined by the file system's generation and configuration

Source: performance specs.

The second row isn't a fixed value determined by SSD capacity alone. 768 MBps/TiB is what you
get when SSD IOPS is set to AUTOMATIC (3 IOPS/GiB). Raising SSD IOPS with USER_PROVISIONED
raises that 768 MBps/TiB figure right along with it. Conversely, raising only throughput
capacity to 2048 MBps while leaving SSD IOPS at AUTOMATIC leaves disk throughput capped at 768
MBps/TiB. There are two levers, and raising only one doesn't move the ceiling.

Laying out which configuration hits which ceiling:

Throughput capacity	Ceiling that applied	Measured (uncompressible, 8 MiB, concurrency 64)	Against that ceiling
128 MBps	The provisioned 128 MBps	133.4 MB/s	104%
2048 MBps	The per-HA-pair write ceiling, 750 MBps	415 MB/s (measured twice: 415.1 / 415.7)	55%

Landing at 104% of the 128 MBps configuration — slightly over the provisioned value — is a
burst effect. This configuration's disk throughput can sustain 128 MBps continuously, or 600
MBps briefly. Bursting was available but the measurement stopped at 133.4, so the provisioned
throughput capacity was the binding ceiling in this configuration.

On second-generation, the meaning of the provisioned value itself is different. In a separate
measurement, a second-generation Single-AZ's provisioned value points to a disk throughput
baseline, and the network-side baseline is roughly twice that. On a configuration provisioned
at 1,536 MBps, 1 MiB sequential reads produced 2,882 MB/s for 27 minutes, then 1,439 MB/s.
Carrying this table's first-generation reading over to second-generation misses a read sizing
estimate by 2x
(measurement record)
(Japanese).

At the 2048 MBps configuration, the provisioned value stopped being the ceiling. The denominator
for percentage changes here, so both are given. 415 MB/s is 20% against the provisioned 2048
MBps. Against the per-HA-pair write ceiling of 750 MBps that actually applied, it's 55%. The
surprise of "provisioned 2048 MBps, only got 415" is explained by the provisioned value no longer
being the applicable ceiling.

What's consuming the remaining 45% — the unused portion of the 750 MBps — hasn't been isolated.
Two candidates come from the published specs.

Writes use twice the network bandwidth of reads. This is documented explicitly, because a write replicates to the secondary file server, so a single write turns into twice the network throughput.
A fixed cost per request. Raising concurrency only stretched p50 (median latency) out to 1054.9 ms (at concurrency 64). If there's a fixed cost per single request, raising concurrency doesn't raise requests-per-second. If request rate doesn't rise, total throughput doesn't rise either.

Which one is doing how much hasn't been isolated. This isn't a case where "the reason it doesn't
reach 750 is this one thing" can be stated.

Bottlenecks sitting in different places per path

This is the main point. Three numbers that come out in the same unit, MB/s, were each plateauing
somewhere different. The figure's A / B / C map in that order.

Path	Where the bottleneck sits	What increases it
This architecture (FSx for ONTAP S3 Access Point)	Over the S3 API path, the provisioned throughput capacity on the file system, shared across all clients	Raise the provisioned value
Amazon S3	Not the service side — client-side network bandwidth	Add more clients
Amazon S3 Files	The CPU of the proxy process running on the client	Use a bigger client

Note: the row above is scoped to the S3 API path. It's based on measurement (1.01x with 4
clients). On the file path (NFS/SMB) of the same file system, reads exceeded the provisioned
value of 6,144 MBps by more than 2x. Reads served from cache don't go through disk, so stating
"the provisioned value is the ceiling" without naming the path would contradict that
measurement. Details in
per-protocol measurement results
(Japanese).

Here's the decision flow for telling which of the above a measurement is hitting, in your own
environment.

flowchart TD
    START["A measured value plateaus"] --> Q1{"Does the total grow<br/>when adding clients?"}
    Q1 -->|"No"| C1["Provisioned throughput capacity<br/>on the file system<br/>(this architecture's S3 API path)"]
    Q1 -->|"Yes"| Q2{"Is the per-client value<br/>near the client's network bandwidth?"}
    Q2 -->|"Yes"| C2["Client-side network bandwidth<br/>(Amazon S3)"]
    Q2 -->|"No"| Q3{"Is CPU high on a process<br/>running on the client?"}
    Q3 -->|"Yes"| C3["CPU of a proxy process<br/>on the client<br/>(Amazon S3 Files)"]
    Q3 -->|"No"| C4["A factor outside these three.<br/>Isolate it separately"]

The figure states the same thing as the table below. A single-host measurement alone can't
determine the type — telling them apart needs either adding clients, or holding transfer size
and concurrency fixed while varying something else at one host (details under
a single-client figure that was actually the client's own limit
above).

FSx for ONTAP — a shared ceiling that wasn't one thing

This is the left side of the figure's panel A, the S3 API write side. Adding clients doesn't
raise the total. With the same setting and varying client count, 4 clients produced 1.01x.
Since the total doesn't change, each client's share is roughly 1/n of it.

The reason is that the ceiling is a value provisioned on the file system. This is a
configuration where capacity planning is done per file system, not per client. If more clients
are expected, plan for it on the throughput capacity side.

Note that reads came out higher than the provisioned value. On a 128 MBps configuration, reads
reached 579.3 MB/s (8 MiB, concurrency 16), 4.5x the provisioned value. Using the write-side
ceiling to size reads will miss.

How the ceiling splits as client count grows to 8

The result above, up to 4 clients, is first-generation with the provisioned value as the ceiling.
Scaling to 8 clients / 128 connections on second-generation, with the same file system, same
setting, and same connection count, the total swung 5.5x. The only thing changed was "where each
host reads from."

With every host reading the same file:

Hosts	Connections	Client total	Port-measured	Per host
1	16	2,664.58 MB/s	2,698.7 MiB/s	2,664.58
4	64	8,562.94 MB/s	8,635.2 MiB/s	2,140.74
8	128	11,916.29 MB/s	12,173.0 MiB/s	1,489.54

With the working set held at a fixed 600 GiB, split into 75 GiB per host at 8 hosts, so no block
is shared:

Hosts	Connections	Client total	Port-measured	Per host
2	32	2,299.05 MB/s	2,323.4 MiB/s	1,149.53
8	128	2,173.37 MB/s	2,191.7 MiB/s	271.67

Quadrupling connections from 32 to 128 doesn't grow the total. It's already saturated at 32.

The difference comes down to how much is served from the file server's read cache. SSD IOPS is
used only when reading data in neither the in-memory nor the NVMe cache, per the
performance specs. Overlap
in read ranges pulls toward the former; no overlap pulls toward the latter. There isn't one
ceiling — there are two, and which one a workload hits depends on the workload.

A hypothesis that turned out wrong

Before this measurement, I had assumed the SVM's single data LIF was a roughly 4,400 MB/s
ceiling. That was wrong.

Since client-reported totals aren't evidence on their own, I cross-checked against ONTAP's
physical port cumulative counters (nic_common's transmit_bytes), differenced. The
port-measured value exceeded the client total by within 1.3% at every point, and all 8 points went
through the same single port. That means a single LIF was carrying 12,173 MiB/s — 102 Gbps.

One thing tripped up this cross-check. The instantaneous-value counter returned 0 even while
traffic was actually flowing. Only the differenced cumulative value was usable.

This changes how capacity planning should work. Rather than counting hosts, decide first whether
each host's read range overlaps. A non-overlapping design pulls toward the disk path's ceiling,
so adding hosts doesn't grow the total — that's what the 32-connection and 128-connection rows
above show.

A single-client figure that was actually the client's own limit

A second, similarly shaped mistake, this time by me. In a separate round, on a first-generation
2,048 MBps configuration, I got roughly 1,250 MB/s and treated it as "this configuration's read
ceiling." I checked with two clients (2026-09-19).

First-generation 2,048 MBps, SSD 1,024 GiB, SSD IOPS 40,000 (USER_PROVISIONED), NVMe read
cache disabled, volume 800 GiB, client c5n.9xlarge x2, each reading a non-overlapping 300
GiB, 1 MiB sequential, 8 threads, o_direct, ONTAP 9.18.1P6, ap-northeast-1.

Condition	Client total	Disk throughput utilization
Single host only	1,195.27 MB/s	63% (SSD IOPS also 40%)
Two hosts simultaneously	981.90 + 919.58 = 1,901.48 MB/s (1.59x)	102-103% (burst)

With a single host, the disk path was only at 63% — the file system side had headroom. Only
with two hosts did the disk side saturate.

From here I chased "so what's the per-host ceiling?" The answer was two of my own settings, not
a resource. On the same file system, I varied rsize to 1 MiB and thread count (the next day).

Condition (1 host)	Measured	Response	Disk throughput utilization
Transfer 64 KiB, 8 threads	1,195.27 MB/s	6.69 ms	63%
Transfer 1 MiB, 8 threads	1,446.66 MB/s	5.443 ms	77.2%
Transfer 1 MiB, 16 threads	1,646.17 MB/s	8.208 ms	102.6%
Transfer 1 MiB, 32 threads	1,647.42 MB/s	16.420 ms	102.6%
Transfer 1 MiB, 64 threads	1,648.71 MB/s	32.838 ms	102.6%

Transfer size added +21%, concurrency added another +14%, and then it stopped. Past 16 threads,
only response time keeps doubling while throughput doesn't move. It's queueing, not gaining.

And a single host reaches the same 102.6% that two hosts reached. IOPS sat at 50.9% and network
at 66.3%, neither saturated. I could not name a specific client-side device that was saturated.

This "can't name it" was traced to a different tier and resolved separately (2026-09-20). On
second-generation, 6,144 MBps / 200,000 IOPS, measuring five ways of splitting the working set
shows adding volumes moves it only +1.0%, while adding SVMs moves it +26.3% and adding
clients moves it +52.7%. So what matters is the number of distinct IPs the client talks to
and the client count, not thread count (+8.9% going from 512 to 1,024). With two clients, the
disk side reaches 96.6% of the provisioned value, and only then does the file system side become
the actual limiter. No single device can be named because the limiter isn't one counter — it's
how receive processing is distributed
(measurement)
(Japanese).

This reinforces this article's claim. The table at the top says to confirm the type of ceiling;
a single-host measurement can't determine that type. Using a value measured alone as a ceiling
for capacity planning under-buys. There are two ways to check — adding hosts and fixing transfer
size/concurrency — and both arrive at the same conclusion.

I also chased the discrepancy between them, without using any additional resources. Both file
systems had already been deleted, but CloudWatch retains data for 63 days after deletion.

This architecture doesn't publish a single burst-remaining metric (zero out of 77 metrics contain Balance). My earlier explanation, "because burst remainders weren't aligned," doesn't hold, because nothing to align exists.
The denominator for utilization was the provisioned value itself (2,048 MB/s). DiskReadBytes / (utilization / 100) converges to 2,048 across every minute. On first-generation, over 100% just means "more than the provisioned value came out," not evidence of spending a burst allowance. Second-generation's 1,536 config at 203.5% is against a baseline, so the same metric has a different denominator by generation.
And both sessions were hitting the same server-side ceiling of about 2,040 MB/s. The 15% client-reported difference wasn't a difference in what the file system delivered.

As one discrepancy closed, a second one — between two counter systems — opened, and it closed
the same day too. At saturation, the server reported about 2,043 MB/s while vdbench reported
1,646-1,649 (19% short). The cause was how the measurement was taken. Aligning the window's
start to a minute boundary and comparing only the minutes fully contained in the window, the
two systems agree within 0.2% (300s window: 271.04 vs 271.57; 900s window: 326.70 vs 326.07).
The 19% arose from taking a 60-second average of a minute where load was only partially applied
as the server-side value. Against a steady-state 271 MB/s, the start minute was 216.85 and the end
minute 58.47. There's nothing to restate about FSx for ONTAP here. Making the two agree
requires both aligning the window and discarding the partial minutes
(measurement)
(Japanese).

The shared ceiling split two ways over SMB too

The same shape showed up over SMB. Varying only client count across 8 clients didn't produce
one single answer.

Everything below the share is identical between the two. Same single CIFS share, same SMB 3.1.1
with 4 Multichannel channels, same physical port on the same node of the same SVM. Only what each
host reads differs.

Total at 8 hosts	Read range
10,721.19 MB/s (port-measured 11,194.7 MiB/s)	Every host reads the same range of the same file
4,225.31 MB/s	Each host reads its own 1/8 of the range

The former was still climbing at 8 hosts; the latter plateaued between 4 and 8. The former's
growth is the result of ONTAP's memory serving the overlap. Citing only one of them leads to the
opposite conclusion.

Client-side totals aren't evidence on their own here either. A single read can satisfy multiple
clients, so bytes never sent over the wire can get counted. Every point here is cross-checked
against ONTAP's physical port cumulative counters.

Per-point figures, and what this measurement doesn't answer (the 180-second window doesn't
separate burst from baseline), are in the
SMB host-count test
(Japanese).

Amazon S3 — a total proportional to client count

The figure's panel B is made of just this one bidirectional arrow.

Hosts	Write total	Per host	Read total	Per host
1	581.5 MB/s	581.5	496.3 MB/s	496.3
2	1158.0 MB/s	579.0	977.5 MB/s	488.8
4	2362.5 MB/s	590.6	1974.7 MB/s	493.7
6	3601.4 MB/s	600.2	3038.5 MB/s	506.4
8	4783.2 MB/s	597.9	3992.2 MB/s	499.0

Going from 1 to 8 clients, write totals grew 8.23x and reads 8.04x. The per-host value doesn't
drop. The total reached 38.3 Gbps at 8 hosts, and this range never showed a point where it
stopped being proportional to host count. Each host used a distinct prefix, so per-prefix request
ceilings weren't tested.

The roughly 500 MB/s visible with a single host was not Amazon S3's ceiling — it was the
client's. Since adding hosts grows the total, capacity planning here comes down to counting
client-side bandwidth.

Amazon S3 Files — the CPU of a proxy process on the client

This path's structure differs from the other two, so the structure is described first. The
figure's panel C shows the client and efs-proxy sharing the same box — that's it. Mounting S3
Files starts a transfer process called efs-proxy on the client, and the NFS client connects to
that process. It's visible in the mount info.

127.0.0.1:/ nfs4 rw,vers=4.2,rsize=1048576,wsize=1048576,proto=tcp,port=20812,addr=127.0.0.1

The mount target is 127.0.0.1 — itself. Data is then carried from efs-proxy to the service
side. FSx for ONTAP's mount target is the file system itself, so this is structurally different.

With that structure in mind, I measured what
not supporting nconnect
means in practice. nconnect is an option that increases the number of TCP connections an NFS
mount opens.

Stream count	Read	Write
1	67.9 MB/s	34.6 MB/s
4	264.3 MB/s	140.5 MB/s
8	451.3 MB/s	281.0 MB/s
16	404.8 / 450.0 MB/s	531.8 MB/s

Reads reach about 450 MB/s at 8 streams and don't grow further at 16 (writes are still growing
at 16, 531.8 MB/s). What follows covers the read plateau. 450 MB/s is 3.6 Gbps, under the
single-flow ceiling
(roughly 5 Gbps per network flow).
So the single-flow ceiling isn't what's binding.

The plateau sits at efs-proxy's CPU. While reading at 16 streams, CPU usage on the
efs-proxy process was 67.3% + 18.1% + 14.9% (an 8-vCPU host).

This is where the meaning of not supporting nconnect becomes clear. What nconnect increases
is the connection count of the NFS mount — between the NFS client and efs-proxy. Since the
plateau is efs-proxy's own CPU, even if it were supported, it wouldn't reach where the plateau
actually is.

So it isn't "slow because nconnect isn't supported." Since the bottleneck is a process's CPU on
the client, the fix is a bigger client, not a mount option. efs-proxy --tls being on this path
is the cost of a configuration where in-transit encryption is the default, not a defect.

One practical note. Specifying nconnect hung the mount (90-second timeout). Not a rejection,
not silent ignoring. The first measurement was entirely lost to this. When trying an unsupported
option, wrap it with timeout.

Absence of decay over a 15-minute sustained write

Writes had been measured in a 120-second window, so I re-measured at 900 seconds, on the
second-generation environment.

Item	Value
Steady-state window average	2,063.00 MB/s (900 seconds after a 60-second warmup)
10-second interval min / max	1,749.0 / 2,128.4 MB/s
Cumulative written	Roughly 1.77 TiB
Decay	Not observed. The first minute and the last minute sit in the same range

Absorption somewhere can be ruled out. In-memory cache is 256 GB and NVMe cache is disabled;
the amount written is 7x that. Volume efficiency was disabled, with 0 savings.

Stating the relationship to the published value directly: the documentation says for
second-generation, "reads get the full throughput capacity, writes get one-third of it," and
it separately names 6,144 MBps as an exception to that general rule, listing it in a separate
table with Single-AZ writes at 1,024 MBps. So the published value that should apply to a 6,144
configuration is 1,024 MBps. The measurement was 2.05x that.

This single point reproduced on a separate file system later. Same 6,144 MBps provisioned
value, 1,800 seconds (twice this measurement's window), 2,097 MB/s, a 0.3% difference between
the first 300 seconds and the last 300. Different day, different file system, different window
length: 2,063 and 2,097, a 1.6% difference. The no-decay conclusion is supported by two
observations.

Lowering the provisioned value and measuring again still exceeds it in the same direction.

Provisioned	Published value that should apply	Measured (sustained, median)	Against the published value
1,536	One-third of the general rule -> 512 MB/s	1,333 MB/s	2.60x
6,144	Exception table's 1,024 MB/s	2,097 MB/s	2.05x

Both points exceed the published write ceiling. Same direction, ratios of 2.60 and 2.05.

This used to be described a different way. It previously read "the general rule holds at
6,144 and misses at 1,536," but 6,144 / 3 = 2,048 being close to the measured 2,097 is a
coincidence. The documentation names 6,144 as an exception to the general rule, so applying
the general rule there doesn't hold in the first place. Corrected.

Which figure refers to what still isn't determined. The table header reads "Maximum
throughput from SSD storage," which may refer to a ceiling scoped to the SSD path only.
But this measurement closed off the likely absorption points — the cumulative 1.77 TiB is
7x the 256 GB in-memory cache, the NVMe cache was confirmed disabled on both nodes, efficiency
was disabled with 0 savings. Why it still comes out roughly 2x can't be explained from this
side.

The design-level effect is over-, not under-provisioning. Working backward from the general
rule to hit a write of 1,333 MB/s looks like it needs a 4,000 MBps-class provisioned value, but
1,536 was measured to be enough. The second-generation provisioned rate is $2.013/MBps-month
(ap-northeast-1, fetched 2026-09-04), so this misreading translates directly into overbuying.

Only two points were measured: Single-AZ, 1 HA pair. Multi-AZ, 2+ pairs, and provisioned values
other than 1,536 and 6,144 weren't measured
(measurement record)
(Japanese).

Capacity growth from continued writing

Trying to continue into a read-side measurement after this, it didn't complete. The cause is a
documented specification — a gap in my own design, not a documentation error.

Before starting, snapshot count was 0 and the snapshot policy was none. 30 minutes into the run, a
snapshot named starting with backup- appeared, holding 739 GiB, and the write rate dropped from
2,200 MB/s to 267 MB/s. Volume usage went from 88% to 100%, with 9.9 GiB free. Deleting files
with rm didn't return the free space.

The cause is the automatic daily backup.
Volume backups state that a
backup first takes a volume snapshot, and that snapshot sits inside the volume, consuming
capacity, and stays until the next backup. For an overwrite-heavy workload, the pre-overwrite
blocks are retained, so the amount written becomes capacity as-is.

Why it doesn't return on deletion is
also documented.
Data deleted from the active file system is not released as long as a snapshot still references
it.

Checking for snapshots before starting doesn't prevent this. Before the run it reads normally,
because no snapshot exists yet. There was a recovery mechanism, though.
It's documented
that volume autosize and Snapshot autodelete are designed to work together. Neither was active
by default in this setup. The actual situation was that the default was in effect because
neither was specified in the template. "Not specified" isn't "disabled."

15 minutes over SMB

No decay over SMB either. The 900-second steady-state window averaged 1,488.03 MB/s, with a
10-second-interval min of 1,452.70 and max of 1,514.40; the first-third average was 1,487.55
against a last-third of 1,486.57.

Changing the client from an 8-vCPU type to a 36-vCPU type only moved it 0.6% (1,478.95 versus
1,488.03). This value isn't determined on the client side.

The initial observation that a 300-second measurement (1,698.42 MB/s) was 12.4% higher than the
900-second one doesn't reproduce (re-measured 2026-09-20). Varying only window length at 6,144
MBps, the longer window actually came out 4.0% higher (1,375.4 versus 1,430.3 MB/s). Channel count
stayed constant regardless of measurement duration, and even writes moved only 1.9% across window
lengths. The original 12.4% gap looks like a one-off value, not a property of window length.
That said, the longer window's vdbench format was interrupted partway through, so it wasn't a
comparable pair
(re-measurement record)
(Japanese).

Against the same physical port on the same node of the same file system, NFS measured 2,063.00
MB/s and SMB 1,488.03 MB/s. Since the source of concurrency differs (16 connections specified by
the client versus 4 negotiated channels), this can't be attributed to a difference in the
protocols themselves.

Conditions and the 10-second-interval progression are in
the SMB sustained write
(Japanese).

Additional read capacity via FlexCache

One more thing was measured on the read side. This section covers the figure's panel A, where
Origin and Cache sit as separate file systems. FlexCache isn't a mechanism for sharing the
origin's (the replication source volume's) performance — it's a mechanism that adds separate
capacity on the read side.

Measurement point	Measured, uncompressible
FlexCache first read (cache not yet filled; read while filling)	306.8 MB/s
FlexCache resident (second read onward; cache already filled)	882.7 / 907.1 MB/s
Direct origin read (control — the comparison baseline; already read once and cached)	383.3 MB/s

Resident is 2.31x a direct origin read. The cache is a separate file system from the origin,
with its own throughput capacity, its own memory, and its own burst allowance. So while reading
from the cache, the origin's own ceiling doesn't apply.

The cost is the first read. Data that's never been read comes in at 306.8 the first time — 1/2.88
of resident. That's the figure to design around.

One qualification. This was measured with both origin and cache at 128 MBps. Raising the
origin to 2048 MBps puts a direct origin read into the 2000 MB/s class, so depending on
configuration, the 2.31x could reverse. Unmeasured.

Bottleneck shift with small objects

Raising throughput capacity 16x and SSD IOPS 13x didn't raise request rate. The six measured
points were 4 KiB reads and writes at concurrency 64 and 256, and 64 KiB reads and writes at
concurrency 64. None of the six grew, and writes dropped 25%. Against 40,000 IOPS, 279-605
req/s — 0.7-1.5% of provisioned IOPS.

The bottleneck sits on the S3 Access Point's request-handling side. For designs that handle a
large volume of small objects, raising throughput capacity and SSD IOPS doesn't help.

Per-tool visibility

What's available to check "what is this actually limited by" while measuring differs by path.
Not knowing this means missing the numbers needed to identify a cause.

What you want to see	Reaches it	Doesn't reach it
Per-volume throughput	ONTAP's volume counters	Client-side measurement (turns into a single-TCP-flow ceiling)
Node CPU	CloudWatch (`AWS/FSx`)	`fsxadmin`'s REST doesn't reach it
Fraction that went to disk	CloudWatch's `DiskReadBytes` / `DataReadBytes`	Can't be determined from the client side
S3 Files propagation lag	CloudWatch (`AWS/S3/Files`'s `ExportAge`)	—

Measuring from the client side turns into roughly 590 MB/s. Linux's NFS mount opens only one
TCP connection per server by default, and that's what limits it first. If you want to measure
the storage-side ceiling, read the storage-side counters.

CPU on both paths was around 22%. The NFS path delivers twice this architecture's throughput at
the same CPU level. So this architecture's cost isn't inside ONTAP — it's upstream, on the S3
translation side.

And on reads, DiskReadBytes / DataReadBytes was 1.4 / 0.9 / 1.5%. 98.5-99.9% of read bytes
didn't go to disk. That means it came back from cache, which links directly to Part 5's fifth
default.

AWS documentation notes and support inquiries

This article touches three points in the published specs. Two were filed on 2026-09-17. The
source for all three is
the performance specs.

Content	Type	Status
Second-generation writes exceed the published ceiling at both points measured (2.60x the general rule's 512 at a 1,536 provisioned value; 2.05x the exception table's 1,024 at a 6,144 provisioned value)	Gap between published value and measurement. Likely absorption points (two-tier cache, efficiency) closed off	Filed (2026-09-17). AWS's reply (2026-09-24): the "up to a third" figure is not a hard limit but a sizing indicator. A write uses bandwidth twice over — once to the active server, once to sync the standby server — but network burst capacity can push measured throughput above a third of the provisioned value, and AWS considers that expected behavior, not an anomaly. The table heading's "Maximum" is described as the ceiling to expect once burst is not a factor and the same file system is shared with other workloads. For sizing, AWS pointed back to the original rule of thumb: provision three times the required write throughput (e.g., a 1,333 MB/s requirement calls for a 6,144 MBps configuration)
ap-northeast-1's NVMe read cache capacity is missing from the table (the four regions with a column have it listed). The management page is also written scoped to "second-generation" only, so a first-generation user with a qualifying config can't tell they have a cache	Missing content. Presence was confirmable via ONTAP; capacity was not	Filed (2026-09-17). AWS's reply (2026-09-21): no documentation change. AWS's position is that a first-generation Single-AZ configuration at 2,048 MBps lacking a cache is not something the documentation promises against, and that enumerating every such corner case isn't practical, so the existing wording stands. Pointed instead to `system node external-cache show` for checking presence and capacity per file system. Both the capacity-publication ask and the management-page scope ask were effectively declined
The observation that raising SSD IOPS made reads 7.14x faster didn't square with "SSD IOPS is used only for reads not served from cache"	My own measurement gap. Hadn't read a volume that reliably exceeded the sum of both cache tiers	Not filed. Re-measuring 2026-09-17 with the working set at 2.15x the cache size squared with the documentation (ratio came out 4.86x; 7.14x doesn't reproduce). Not a documentation error

Why the third isn't filed. The mismatch's cause was on my side. Re-measuring settled it
(2026-09-17). With the working set at 512 GiB — 2.15x the in-memory cache — and the NVMe cache
disabled, 96-126% of read bytes came from disk, with the low end saturating at
DiskIopsUtilization 112%. This matches the documentation. Not a documentation error, so not
filed. Only the ratio changed — 4.86x, and 7.14x doesn't reproduce. And what's capping the
high side, roughly 1,250 MB/s, remains unresolved — that isn't a finding against the published
spec, it's a range I haven't measured.

The first and second were filed. Both were filed without me deciding "the correct value is
this" — as a gap or omission for whoever holds the values to judge. The first was framed as the
question "what does 'Maximum throughput from SSD storage (write)' actually bound?" and the second
as two points: publishing the capacity, and the management page's stated scope.

The first used to be described a different way in this article. It previously read "which of
the general rule and the exception table applies flips depending on the value," but the
documentation names 6,144 as an exception to the general rule, so applying the general rule
there doesn't hold in the first place. Corrected before filing.

Both first replies came in on 2026-09-21. The first stopped at confirming AWS's own measurement
clears a third of the provisioned value; the second landed on a decision not to change the
documentation. The underlying fact — a first-generation Single-AZ configuration at 2,048 MBps can
lack a cache — is something the documentation genuinely never promised against, and not
enumerating every corner case is a reasonable line to draw. What didn't go through, though, is
separate from that: asking for the capacity to be published, and asking the management page's
stated scope to cover first generation too, are both requests that a documented gap doesn't
answer by itself. Recording that they didn't land. The per-file-system way to check
(system node external-cache show) is documented, so checking it yourself is unaffected.

The first one got a follow-up on 2026-09-24. The "up to a third" figure turned out to be a
sizing indicator rather than a hard limit — a write spends bandwidth twice over (once to the
active server, once to sync the standby), and network burst capacity can push it above a third of
the provisioned value, which AWS considers expected rather than anomalous. The table heading's
"Maximum" is described as the ceiling to expect once burst isn't a factor and the file system is
shared with other workloads. For sizing, AWS pointed back to the original rule of thumb: provision
three times the required write throughput. This explanation doesn't dispute that the
measurement exceeded the published value — it only says how to read that published value.

The block protocol side's findings and filing status can be traced from the
separate article.
Submission status across the three block-protocol articles and this article is collected in
AWS documentation notes and inquiries across the block-protocol articles.

What hasn't been measured yet

Meaning not yet measured, not "can't be."

Reading more than 2x the cache size in one pass. At 512 GiB (2.15x the cache), 4.86x (256.47 -> 1,246.67 MB/s). The low side saturating on SSD IOPS is confirmed. What was capping the high side, around 1,250 MB/s, turned out to be transfer size and concurrency, not a resource (details under a single-client figure that was actually the client's own limit above)
Effect of measurement duration. Measurement duration itself is not the cause. What carries the gap is provisioned IOPS, payload compressibility, and unit-to-unit variance (measurement) (Japanese)
Measuring FlexCache uncompressed on a 2048 MBps configuration. The "2.31x" could reverse depending on configuration
What's capping the roughly 300 MB/s on the 128 MBps configuration. Reads on the same configuration converge to 297-317 MB/s for both warm and cold (values from the same measurement set as this article, not shown in the body table). This used to be described as "consistent with plateauing on a burst," but reading burst remainders ruled that out. The remainder plateaued at 99 -> 95% and returned to 99% after the load stopped — not exhausted. Against a disk burst ceiling of 600 MBps, the measured 254 MB/s is 42%; against a network burst ceiling of 1,250 MBps, roughly 285 MB/s is 23% — neither ceiling was reached. It isn't the roughly 590 MB/s single-flow ceiling either. Where it sits is unresolved (measurement) (Japanese). Note that DiskReadBytes / DataReadBytes in the same window was 87-97%, so reads on this configuration are coming from disk — a contrast with the 2048 MBps configuration's 1.4%, consistent with an in-memory cache of only 16 GB
The host count at which Amazon S3's total stops being proportional to host count. It's still proportional at 8 hosts / 38.3 Gbps
Adding S3 Files proxies. A configuration with separate mounts running more efs-proxy instances hasn't been tested
More than 8 hosts. The side sharing the same file dropped to a 1.12x increment at 8 hosts, so where it stops hasn't been observed
A point below 32 connections on non-overlapping ranges. It was already saturated at 32, so the connection count where saturation begins hasn't been bracketed
A sustained write longer than 900 seconds. No decay over 15 minutes, but nothing beyond that has been checked
Reads with the fraction served from cache varied. Capacity filled up and the run was aborted, so none of the 7 planned points were captured
Time-of-day effects. Two different days were checked, but not different times of day
SMB's 7 scenarios under cache control. Stopped by fill, not capacity. format=yes ignores the run block's xfersize and threads, writing at 128 KiB and queue depth 2, so overwriting the same file that otherwise reaches 1,488 MB/s only produced 165 MB/s (315 MB/s with formatxfersize=1m; queue depth stayed at 2). The parameter file points 8 SDs at a single file; splitting each range into its own file looks likely to solve both the speed and the design problem at once. This is a prediction, not a result. Background in two places stuck under cache control (Japanese)
Why SMB's 300-second and 900-second values differ. Doesn't reproduce. Re-measuring at 6,144 MBps removes the window-length difference; the longer window's vdbench run had also been interrupted, so it wasn't a comparable pair in the first place (measurement) (Japanese)
Conditions for raising Multichannel's channel count above 4. Setting max_connections_per_session to 32 still produced 4. The 4 comes from the client side's ConnectionCountPerRssNetworkInterface, which determines channel count per RSS-capable NIC. The server side's 32 is a per-session ceiling, not the limiter on the increasing side. Adding NICs or raising this value hasn't been tested
The discrepancy between server-side counters and the client-side measurement tool. The cause was measurement method — aligning the window's start to a minute boundary and comparing only the minutes fully contained in the window, the two systems agree within 0.2% (measurement) (Japanese)
What the 4 KiB random read ceiling actually is. It's on the client side. What matters is the number of distinct IPs the client talks to and the client count, not thread count. When citing this figure, write "per client" — it isn't the file system's ceiling (measurement) (Japanese)
Why 4 KiB random reads plateau at 47.5% of provisioned SSD IOPS. The ratio doesn't reproduce.
Smaller configurations are disk-IOPS-limited; larger ones follow a different formula.
So "47.5% of provisioned" is a value specific to that one configuration, not an
invariant with scale. When citing it, always state the 6,144 MBps / 200,000 IOPS condition
alongside it
(measurement)
(Japanese)
Metadata-operation performance under a high file count (creating, enumerating, and stat-ing a
large number of files). What this article measured is throughput, IOPS, and concurrency, not
metadata operations when the file count is large. NetApp's High File Count NAS Workloads
Technical Report suggests that inode capacity and a single directory's maximum size bear on
enumeration cost, but there's no figure measured on this architecture (S3 API collect -> origin
-> FlexCache distribution). Measuring it would need an origin volume holding a sufficient file
count and a measurement setup for metadata operations over it
The time taken to enumerate a directory holding a large number of files (ListObjectsV2, or an
NFS / SMB directory walk). The enumeration cost when files concentrate in a single directory
is something this project hasn't measured on this architecture
FlexCache's metadata-replication cost under a high file count. How the cost of replicating
metadata from origin to cache scales with file count is something this project hasn't measured
on this architecture. It's a different axis from this article's throughput measurements

Closing

Measuring three paths from the same host, the bottleneck sat in three different places for all
three. A throughput capacity provisioned on the file system, client-side network bandwidth, and
the CPU of a process on the client. This difference doesn't close as you add more clients.

And on the FSx for ONTAP side, there wasn't one ceiling. With the same file system and the same
connection count, the total swings 5.5x depending only on whether each EC2 client's read range overlaps.

Which one to choose comes down to where you place the read side, which side holds the primary
copy, and when you want a write visible on the bucket side. Throughput numbers are one input into
that decision, and they come later in the order of things to check.

What's worth taking away is a procedure: check the type of ceiling before comparing
throughput. Whether it's a provisioned value, client-side bandwidth, or a process's CPU on the
client. Line up numbers of different types in the same column, and you get a table shaped like a
comparison without being one.

And a value measured with a single EC2 client can't determine that type. On the same file system,
adding a second EC2 client grew the total 1.59x, and fixing transfer size and concurrency while
staying at one client grew it 1.38x. Either way arrives at the same ceiling. There are two ways
to check the type, and neither costs much extra.

A measurement that reproduces within 0.02-0.14% under identical conditions can still swing 30%
from a single misunderstanding about conditions. "Stable" isn't "correct." And this article's
figures are for one specific throughput capacity and one specific payload. They don't transfer
directly as your environment's numbers. What transfers is the measurement approach.

Hope this helps anyone running the same kind of measurement.

Defaults to check before measuring and items to always record are collected in
Performance testing considerations.

Until next time.

Amazon FSx for NetApp ONTAP NVMe/TCP assumes RHEL 9.3: why multipathing fails on Amazon Linux 2023

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Wed, 30 Sep 2026 05:42:42 +0000

Amazon FSx for NetApp ONTAP's NVMe/TCP procedure assumes RHEL 9.3: why multipathing is unavailable on Amazon Linux 2023, and how to check before you are billed

Followed the AWS procedure and run cat /sys/module/nvme_core/parameters/multipath only to find
the file isn't there?

Moving your target Linux distribution to Amazon Linux 2023 can stop a documented command from
working exactly as written. You are reading the procedure correctly, yet your environment alone
gives a different result — and tracking the cause down to a kernel build setting takes time.
This article is for anyone who got stuck at that point.

What this article does and does not cover: it covers how to confirm multipathing is unusable on
AL2023, and how to count whether it is actually using the second path. It does not cover
performance comparisons or a generalized conclusion for distributions other than AL2023 (details
under "What this article does not cover" below). All figures are from a Single-AZ configuration;
Multi-AZ is unverified.

dev.to series: FSx for ONTAP Block Protocols, Measured.

What was checked

Assuming no prior knowledge of FSx for ONTAP, here's what this article is confirming.

Location: a single AWS VPC, single AZ
Client: one EC2 instance (Amazon Linux 2023)
Target: one Amazon FSx for NetApp ONTAP file system (second-generation SINGLE_AZ_2). The HA pair's two controllers hold two paths to one namespace
- Path 1: to the optimized controller (shortest path)
- Path 2: to the non-optimized controller. Only when multipathing is available can the host tell that path 2 is a standby

the shortest route) and Path 2 (non-optimized, which looks like a normal route without ANA), both
reaching the same one namespace" width="800" height="682">

Figure: an FSx for ONTAP HA pair has two paths to one namespace, and **Asymmetric Namespace
Access is the mechanism that tells the host which one is the shortest path* (referred to as
multipathing throughout this article). Without this mechanism in the kernel, those two paths
don't appear as "one shortest path plus one standby": they appear as two separate devices
pointing at the same data.*

Item	Role in this article
VPC / AZ	Single AZ, so both paths to the HA pair's controllers sit on the same network
EC2 client	Checks for multipathing and the kernel configuration behind it
optimized path	The shortest path as determined by multipathing. Real data is expected to flow here
non-optimized path	The failover path. Without multipathing it appears as an ordinary path too

AWS's
Provisioning NVMe/TCP for Linux
includes a step that confirms multipath has come up: the fourth item under "To discover the target
NVMe nodes". It says that a returned Y means success (wording as of 2026-09-17; every
reference to that page below was checked on the same day).

cat /sys/module/nvme_core/parameters/multipath
# Y means success

On the Amazon Linux 2023 AMIs I used, that file did not exist. The kernel does not have
Asymmetric Namespace Access (a mechanism that tells the host which of several paths to the same
namespace is the shortest one) in it. The range I checked is the three kernel series returned by
al2023-ami-kernel-default-x86_64, so I cannot write "AL2023 always behaves this way". Checking
takes one command, and a later section gives it.

The procedure is written for RHEL 9.3. The same page's "Before you begin" says
Create an EC2 instance running Red Hat Enterprise Linux (RHEL) 9.3. Immediately after that there
is a statement about other distributions. In summary: on AMIs other than RHEL 9.3 some of the
utilities may already be present or the install command may differ, but apart from installing
packages, the commands in this section are valid on other EC2 Linux AMIs too (the original wording
is at the end of "Before you begin" on that page).

What this article reports is two counterexamples to that statement. Neither of them is
explained by package installation.

Command from the documentation	Result on AL2023	A packaging problem?
`cat /etc/nvme/hostnqn`	File does not exist	No. `nvme-cli` is installed; whether it creates this file differs by distribution
`cat /sys/module/nvme_core/parameters/multipath`	File does not exist	No. This is a kernel build option (`CONFIG_NVME_MULTIPATH`) and a package cannot change it

You can check this before any billing starts. That is the most practical part of this article,
I think.

This is not a claim that "the documentation is wrong". It is a report that these two do not
fall inside the stated range (apart from installing packages, valid on other AMIs too).
The state of my feedback to AWS is at the end of this article.

Then, after bringing in a kernel that does have multipathing, I found that the optimized /
non-optimized output of nvme list-subsys says "there are two paths", not "two paths are in
use". Whether they are in use has to be counted separately, and there is a trap in choosing
which table to count.

What this article does not cover. It does not compare performance. The figures here exist to
show that two clients agreed under the same conditions, and they cannot be used for sizing. What
moves the numbers is in a separate article. All figures are from a
Single-AZ configuration; Multi-AZ is unverified.

This is one of three, but there is no reading order. The block-protocol measurements are split
by reader.

Article Reader

A Session and queue counts Anyone about to mount

B Checking whether multipathing is available (this article) Anyone who followed the procedure and got stuck

C What moves the numbers Anyone measuring or citing figures

None of them assumes the other two. Read only the part you need.

The environment I measured

Item	Value
File system	Amazon FSx for NetApp ONTAP, second generation SINGLE_AZ_2, 6,144 MBps × 1 HA pair, 4,096 GiB SSD, 200,000 provisioned IOPS
Region	ap-northeast-1, single AZ
ONTAP	9.18.1 (the reads and writes in the agreement table). For the 12 sequential-write points I failed to record the version. `FileSystemTypeVersion` returned `null` and I tore the environment down before re-reading it from the cluster API. The kernel investigation also spans deployments on other versions. The cross-article version mapping is in the ONTAP version matrix
NVMe read cache	Disabled (confirmed on both nodes)
LUN / namespace	600 GiB, on a 900 GiB volume
Before measuring	Wrote the full 600 GiB once. Unwritten blocks on a thin-provisioned volume return zeros, so reading without writing first measures nothing
Client	c5n.9xlarge; 50 Gbps network is a guaranteed figure
Instrument	VDBENCH 5.04.07 directly, `iorate=max` (unlimited), 512 threads, 60 s warm-up + 300 s measurement
Test data / efficiency settings	Not recorded whether the VDBENCH payload was compressible, or whether volume inline efficiency (`StorageEfficiencyEnabled`) was enabled. On the file-protocol side, efficiency settings measurably affected figures; whether the same applies to block is unconfirmed

The window is 300 seconds, so the figures in this article include burst. They cannot be cited as a
baseline.

The figures in this article are iorate=max saturation points, not operating points. No target
IOPS was given; they are what the system took when everything available was pushed at it. On the
file-protocol side of the same repository, giving a target of 4,400 IOPS produced 4,406 MB/s at
8.86 ms, while unlimited produced 4,204 MB/s at 121.79 ms. Unlimited was 5% lower in
throughput and 14 times higher in response time.

With 512 threads fixed and no rate limit, response time becomes the identity 512 ÷ IOPS
(derived 216.6 ms against measured 216.588 ms in a separate run). The figures below can be read
through that identity: 1,277.48 MB/s is about 401 ms, and 131,736.5 IOPS is about
3.9 ms. Both are derived values and were not recorded as measurements.

For this article's purpose, being a saturation point is not a disadvantage. The figures are
here to show that Rocky and RHEL agreed under the same conditions, and as long as both are
saturated the same way, the agreement test holds. Those values still cannot be used for
sizing.

Behaviour on this AMI

On this AMI the file named at the top does not exist. Reading the kernel configuration gives
# CONFIG_NVME_MULTIPATH is not set, and modinfo nvme_core has no multipath parameter either.
So multipathing is not in the kernel.

As a result the two controllers are not merged into one device; they appear as separate devices
pointing at the same namespace UUID (/dev/nvme2n1 and /dev/nvme3n1). nvme list-subsys shows
no optimized / non-optimized either.

The AWS procedure is written for RHEL 9.3, where it is present.

Please do not generalise this to "multipathing is not available on AL2023". What I observed were the
AMIs that the al2023-ami-kernel-default-x86_64 SSM parameter returned on 2026-09-12 and
09-13, and I did not record the kernel versions. The template said "record the kernel version
at measurement time" in place of pinning the AMI, and the block phase did not implement that.
That is a gap in my records.

Checking in your own environment is more reliable, and it is one command.
grep -i NVME_MULTIPATH /boot/config-$(uname -r)

I later widened the observation to three kernel series. All three returned by
al2023-ami-kernel-default-x86_64 (6.1.186 / 6.12.103 / 6.18.48) gave
# CONFIG_NVME_MULTIPATH is not set. You can check this before creating a file system, by
extracting the config from the kernel package in the repository. It is known before billing
starts, so checking in that order is cheaper.

To measure multipathing I then brought in another distribution. The AWS procedure is written for RHEL 9.3,
so I put both Rocky Linux 9.7, a rebuild, and RHEL 9.7 itself on the same namespace and compared
them.

Workload	RHEL 9.7	Rocky 9.7
1 MiB sequential read	1,277.48 MB/s	1,276.60–1,279.22 MB/s
4 KiB random read	131,736.5 IOPS	131,550.6 / 131,749.4 IOPS
4 KiB random write	195,439.8 IOPS	194,960.6 IOPS

Only the reads agreed in the same state (within 0.1%). Same minor version, same nvme-cli
(2.16-1.el9), same cold state. The kernels differ in z-stream (5.14.0-611.55.1 against 611.5.1).
For reads, there is now one piece of grounds for reading a rebuild's figure as the figure for the
product itself.

Both write workloads had the Rocky side measured warm, so the states are not aligned. Counting
the 0.3% difference on 4 KiB random write as "agreement" was my error, and the note in that same
table contradicted my own conclusion (corrected 2026-09-20). For that shape I can say neither
that it holds nor that it does not.

Sequential write was re-measured later, in the same state, for that shape alone. It does not
agree (2026-09-17). On a different file system (second generation, 1,536 MBps, 50,000 provisioned
IOPS), same namespace, same state, same parameters, six runs each.

Client Min Median Max Spread

Rocky 9.7 842.70 1,040.26 1,092.50 29.6%

RHEL 9.7 1,105.75 1,112.65 1,129.30 2.1%

Units MB/s (1 MiB sequential write, 8 threads, o_direct, 120-second window). These 12
points were taken under conditions that differ from the environment described at the top: a
different file system (second generation, 1,536 MBps, 50,000 provisioned IOPS), 8 threads rather
than 512, and 120 seconds rather than 300. The 512 ÷ IOPS reading given at the top cannot be
applied to these 12 points. RHEL's minimum is above Rocky's maximum: six against six, with no
overlap in range. RHEL's median is 7.0% above Rocky's, and the asymmetry in spread is larger than
the difference in medians.

So "a rebuild's figure can be read as the figure for the product itself" is limited to the
reads. When citing a sequential-write figure, say which of the two it was measured on.
4 KiB random write was not re-measured in the same state, so for that shape I can say neither
that it holds nor that it does not. I cannot state a cause. The only difference on record is
the kernel z-stream, and I did not vary it on its own. During the runs the file system side was at
72–81% disk throughput, 16–19% IOPS and 31–38% network, none of them saturated (so this is not a
case of the two meeting at a ceiling). For these 12 points I failed to record the ONTAP version.
FileSystemTypeVersion returned null and I tore the environment down before re-reading it from
the cluster API.

`iopolicy` on a kernel with multipathing, and the value the procedure states

This is the third counterexample. iopolicy (Linux's multipath setting that decides how I/O
is distributed across multiple paths to an NVMe device: round-robin cycles through them,
queue-depth sends to whichever has the shortest queue) is what step 5 of the same procedure
says to confirm as round-robin (and the example nvme list-subsys output at step 7 also shows
iopolicy=round-robin).

	Value
Value the documentation says to confirm	`round-robin`
Measured on Rocky Linux 9.7 / RHEL 9.7	`queue-depth`
What set it	the udev rule `71-nvmf-netapp.rules` shipped by `nvme-cli` (2.16-1.el9)

The kernel default is numa, neither round-robin nor queue-depth. It was queue-depth in
this environment because the NetApp-oriented udev rule bundled with the nvme-cli package set it.
Following the confirmation step returns a value other than the expected one.

I measured the effect on performance. It is negligible: switching policy mid-run produced no
step change, and the difference was 0.5%. The figures and the method are in
a separate article (C). So this is not a story about round-robin being
required for speed. The only part worth reporting is that the confirmation step returns a value
that differs from the documentation.

This can change with the nvme-cli version. What I observed here is one version, 2.16-1.el9,
and I did not confirm whether other versions ship the same udev rule.

How to check whether multipathing is using the second path

This section was added afterwards. It became necessary in order to do the analysis above, and it
is also the reason I once published a wrong conclusion.

nvme list-subsys shows the two controllers as optimized / non-optimized, and prints iopolicy
as well. That is a display of "there are two paths", not of "two paths are in use".

At first I varied iopolicy three ways, compared total throughput, saw that only queue-depth was
faster, and wrote that "only queue-depth uses the second path". That was wrong. I had not
counted bytes per path. A client-side total that does not increase is consistent with two different
facts: "the second path is not being used", and "it is being used but the ceiling is nearer than
the path".

Bytes per path can be read from two places

On the ONTAP side they can be read from a counter table. There is a trap here.

Table	Result in this configuration
`nvmf_lif`	Zero rows. Still empty after writing 600 GiB over NVMe/TCP
`lif`	Reports 0 bytes for the LIF in question. It does not count NVMe-oF
`nvmf_tcp_port`	Has the real data. `read_data` / `write_data` / `total_ops` per LIF

"The table exists but has no rows" and "this version does not have that table" add up to the same
appearance. Return 0 and move on, and you reach the correct conclusion, "the second path saw
0 bytes", for the wrong reason.

It can also be counted on the client side. NVMe/TCP controllers have different destination
addresses, so aggregating established sockets by destination separates the paths. That is a second
source, independent of the ONTAP side.

ss -tin state established | awk '/:4420/{...}'   # sum bytes_sent / bytes_received per destination

Counted properly, the second path was not running

I ran four workloads under queue-depth and took before/after differences.

Path	Read bytes	Write bytes	Total ops
optimized	+about 1,012 GiB	+about 730 GiB	641,908 → 120,008,866
non-optimized	±0	±0	4 → 4

Neither reads nor writes went to the second path. The ONTAP side and the client side agreed, and
it was the same on Rocky and on RHEL (the second path's increment across all workloads was about
20 KB).

Further, switching iopolicy within a single run produces no step change. With a sequential read
running I changed queue-depth → numa → queue-depth, and the values sampled at one-second
intervals stayed inside the same 1,150–1,400 MB/s range both before and after each switch.

Summary

All three kernel series returned by al2023-ami-kernel-default-x86_64 (6.1.186 / 6.12.103 / 6.18.48) had CONFIG_NVME_MULTIPATH unset. Measuring multipathing requires a different distribution. Asking AWS why AL2023 is excluded from the procedure (reply 2026-09-21) surfaced a likely reason: nvme connect-all establishes both paths and auto-discovers the alternate LIF even when only one is given, but without CONFIG_NVME_MULTIPATH the kernel never merges them into one namespace device. The procedure's later steps assume that merge already happened, so proceeding unmerged leaves the other path writable against the same namespace with no multipath software mediating it: a data-corruption risk
You can check before creating a file system. Extract the config from the kernel package in the repository and you know before billing starts. On a running instance it is the one command grep -i NVME_MULTIPATH /boot/config-$(uname -r)
Rocky Linux 9.7's figures can be read as RHEL 9.7's figures, but only for the reads. On the same namespace in the same state they agreed within 0.1%. Both write workloads had the Rocky side measured warm, so the states are not aligned. Counting the 0.3% difference on 4 KiB random write as agreement was my error, and for that shape I can say neither that it holds nor that it does not (corrected 2026-09-20). Sequential write does not agree: re-measured later in the same state, six runs each, RHEL's median was 7.0% higher and RHEL's minimum was above Rocky's maximum (spread 29.6% for Rocky against 2.1% for RHEL). When citing, say which of the two it was measured on
optimized / non-optimized in nvme list-subsys is a display of path existence. For whether they are in use, count with the nvmf_tcp_port counters. nvmf_lif returns no rows, and lif does not count NVMe-oF
In this configuration, not one byte went to the second path even under queue-depth. Two places agreed (the ONTAP-side counters and the client-side sockets), and it was the same on Rocky and on RHEL

AWS documentation notes and support inquiries

This article touches three places in the documentation. All three were submitted on 2026-09-17.

The source for all of them is
Provisioning NVMe/TCP for Linux.

Remark	State
Against "apart from installing packages, valid on other EC2 Linux AMIs too": `cat /etc/nvme/hostnqn` finds no file on AL2023	Submitted (2026-09-17). Reproduced in the same environment, confirming the wording doesn't hold as-is for the AL2023 environment checked. The two proposed fixes (scoping the wording, adding a generation step) have been shared as an improvement request. Adoption, content, and timing are unconfirmed
Against the same statement: `cat /sys/module/nvme_core/parameters/multipath` finds no file on AL2023 (kernel build option)	Submitted (2026-09-17). The observation was confirmed and filed as documentation feedback. Fix timing undecided. A follow-up question ("why is AL2023 excluded from scope?") also got a technical explanation: proceeding with the paths unmerged leaves the other path writable against the same namespace with no multipath software mediating it, a data-corruption risk, so the exclusion has a technical rationale
Step 5 directs confirming that `iopolicy` is `round-robin`, but the measurement on Rocky 9.7 / RHEL 9.7 is `queue-depth`. `nvme-cli`'s version boundary is 2.11 -> 2.12 (see below)	Submitted (2026-09-17). On 2026-09-30 it was acknowledged that a note is needed stating whether `iopolicy` resolves to `round-robin` or `queue-depth` depends on the environment, and this was fed to the responsible team (with a note that NetApp's own published documentation carries the same point). The content and timing of any change are not disclosed in advance, so they are unconfirmed

The form I submitted was this. Verbatim quotation (which line of which step), reproduction steps,
expected and measured values, and the range I confirmed across three kernel series, written as
such. Not generalising to "multipathing is not available on AL2023" is the same in the feedback: the
observation is limited to the three series returned by al2023-ami-kernel-default-x86_64.
For the second one I also included the control under which the procedure does hold (on a kernel
with CONFIG_NVME_MULTIPATH=y, Y is returned). It is not a report that the procedure itself is
wrong.

The third is not submitted as a performance problem. The difference between policies is 0.5% in
this configuration, and the report is the single point that the confirmation step returns a value
differing from the documentation.

The third changed shape before submission, after I looked at the package side. Reading
71-nvmf-netapp.rules out of four builds of the public nvme-cli package (the SRPMs for 2.11 /
2.13 / 2.16, plus the installed 2.16-1.el9) showed that 2.11 has round-robin and 2.13 changed it
to queue-depth (confirmed 2026-09-17). At that point 2.12 had not been examined, so the boundary
could only be written as "somewhere between 2.11 and 2.13".

Submission status across the three block-protocol articles and S3 Burst Part 2 is collected in
AWS documentation notes and inquiries across the block-protocol articles.

A pointer to the upstream v2.11 and v2.12 sources followed, showing the boundary is v2.12.
Fetching and checking that commit (2026-09-19, 71-nvmf-netapp.rules.in) confirmed that
queue-depth was already in place as of v2.12. So the version boundary is settled at
2.11 -> 2.12 (the original "somewhere between 2.11 and 2.13" narrowed to 2.12, from both that
pointer and my own re-check). This is agreed to be a version drift rather than an error in the
documentation.

On 2026-09-30, it was acknowledged for this point that a note is needed stating that whether
iopolicy resolves to round-robin or queue-depth depends on the environment, and it was fed to
the responsible team.
NetApp's NVMe-oF procedure for RHEL 9.x
carries the same point (from RHEL 9.7 the default is queue-depth; on 9.6 round-robin is the
default with queue-depth selectable), consistent with this understanding. The content and timing
of any change are not disclosed in advance, so its reflection in the documentation is unconfirmed.

Primary record of the figures

The primary record of the figures and their conditions is in
the protocol measurement results,
and the reproduction steps are in
the block measurement runbook.
What is unconfirmed is written as unconfirmed there, so please do not drop that when citing.

How to check whether multipathing is available, counter-table traps, and iopolicy version differences are
collected in
Considerations when measuring FSx for ONTAP's block protocols.

AWS Transform Now Supports Block Storage Migration to FSx for ONTAP — Benefits and Pitfalls from a Hands-On EC2 Test

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 28 Sep 2026 13:42:13 +0000

Introduction

This is Part 2 of the series "The Data Foundation for AWS Modernization." Part 1 set out the lens: migration is not the goal but the entry point, and putting FSx for ONTAP at the data foundation lets you re-choose the compute. This part takes that entry point through AWS Transform, the core of modernization, and verifies hands-on that the data area lands on FSx for ONTAP as part of the migration on AWS. Migration from an on-premises VMware environment will come separately; the idea here is to start modernizing from the tightly coupled EC2 × EBS configuration by decoupling the data store.

Until now, when you migrated servers with AWS Transform (MGN), every disk was placed on Amazon EBS. If the source was not ONTAP and you wanted the data area on Amazon FSx for NetApp ONTAP (FSx for ONTAP below), you needed a "two-step migration" — migrate to EBS first, then move the data — or a third-party block storage migration tool.

With the update of 30 August 2026, FSx for ONTAP can now be chosen directly as the MGN target storage type.

AWS Transform announces general availability of Amazon FSx for NetApp ONTAP support: https://aws.amazon.com/about-aws/whats-new/2026/09/aws-transform-fsx-netapp-ontap-support/

That looked useful, so I tried it hands-on straight away.
The source in this test was not an on-premises VMware environment but an EC2 instance already on AWS (Amazon Linux 2023). Using that EC2 as the source, I ran the new path that migrates the data area (EBS) directly to an FSx for ONTAP iSCSI LUN.

Running it for real turned up several pitfalls that reading the documentation alone does not reveal.
Besides sorting out what this new feature can and cannot do, this article records the traps I actually stepped on during the test — errors hidden behind a successful job, and an unexpected capacity spike.

Here are the highlights up front, especially the traps to watch for.

Only data volumes can be targeted. Boot is always Amazon EBS.
What became GA is "the MGN target storage type". It is a different thing from presenting FSx for ONTAP as a VMware datastore (the Amazon EVS side).
In MGN's phases, "SNAPSHOT" was a volume Snapshot done as a metadata operation, and "LAUNCH" was the creation of a FlexClone (about 44 seconds for 8 GiB).
[Trap 1] Finalize is not just clean-up; it is the step where physical capacity temporarily peaks (about 2x). That 2x, though, is a measured peak, not a sizing guideline: the official guidance is to provision 3x the migration data as SSD capacity (see below).
[Trap 2] Even when the job status becomes COMPLETED, a failed final snapshot (fallback) or the creation of an unbootable target can be hiding behind it.

Test Assumptions and Scope

The test was run with the following environment and scope.
(Region: ap-northeast-1 / Source: Amazon EC2 (AL2023) / ONTAP: 9.18.1P3D1)

[What was run]
The whole flow: agent installation → replication → test launch → cutover → Finalize → teardown.

[Out of scope for this article (not verified)]

Migration with VMware as the source (no vCenter was prepared, so EC2 stood in as the source)
FSx for ONTAP as an Amazon EVS datastore (a different feature)
Durations at production-scale data volumes (the test used a tiny 16 GiB environment, dominated by fixed overhead)
Windows sources, and access over SMB / NFS
Post-migration optimization with lun move start

The Shape of the Configuration

Figure 1: three EC2 instances and where each volume goes: the boot column is EBS in all three rows, and only the data column changes to FSx for ONTAP from staging on (dark theme)

Figure 2: the control path, from AWS Transform to the ONTAP management endpoint (dark theme)

The point is that the boot area and the data area go to different places. The boot column stays Amazon EBS in all three rows, and only the data column changes to FSx for ONTAP from the staging row on. The target instance boots from EBS and receives the data over iSCSI. The source in this test was EC2, but the lower two rows of the configuration are the same whether the source is a VMware datastore or on-premises block storage.

For the control path in Figure 2, the official blog says a PrivateLink connection is established automatically, but in practice a Network Load Balancer (NLB) and a VPC endpoint service are created inside your VPC. They incur hourly and LCU charges.

Layer	Normal operation (replicating)	After cutover
Source EC2	Running. The agent keeps sending blocks	Can be stopped
Replication server EC2	MGN launches it automatically	Terminated by Finalize
Amazon EBS	Boot staging	Becomes the target's root volume
FSx for ONTAP	Staging FlexVol and LUN	Target FlexVol (FlexClone)
Network Load Balancer	Running	Remains after Finalize (manual deletion needed)

Table 1: each layer after cutover: the one that needs manual deletion is the Network Load Balancer

The Most Important Distinction: What Became GA Is the MGN Target

Mix this up and you will evaluate the wrong product. It is easily confused with "FSx for ONTAP can be used as a VMware datastore".

Question	Current support
Can FSx for ONTAP be used as the MGN target storage type?	Yes (GA with this release)
Is FSx for ONTAP presented as a VMware datastore?	No. The datastore is a separate feature on the Amazon EVS side
Can the boot disk also go on FSx for ONTAP?	No. Boot is always Amazon EBS
Can EBS and FSx for ONTAP be mixed within one server?	No. All data volumes use the same type
Can you migrate agentlessly?	No. Agent-based replication only
Which protocol do clients connect with?	iSCSI. The data is placed as LUNs inside a FlexVol

Table 2: questions and current support: the agentless path is not supported

The Known limitations in the MGN documentation state Agent-based replication only. If you were planning around vCenter's agentless path, the plan has to change. The path depends on whether you leave VMs for EC2, or keep the VMs and move only the storage out.

Figure 3: the FSx for ONTAP settings in the replication template. Only the SVM ID and the Secret ARN are specified, and the note states that boot goes to EBS

Environments This Configuration Suits

Whether a migration fits depends on the shape of the workload, not on the product of the source storage.

The boot / root area and the data area are separate, and the data area is on block storage
You want capabilities such as Snapshot and thin provisioning on the data area after migration
You want to turn "migrate, then move the data" into a single step
You already run ONTAP and want the same mechanisms (Snapshot / FlexClone / SnapMirror) after migration

Most of these apply whatever the source storage is. Even when the source is ONTAP, however, permissions, quotas and Snapshot policies are not migrated and need to be set up again.

Environments Where Another Option Fits Better for Now

Boot and data sit on the same volume
You cannot install an agent (agentless is not supported)
The source is Windows and you want to move the data as SMB shares
File-level migration is enough and block-level replication is not needed (a file transfer such as AWS DataSync is sufficient)
The source is ONTAP and you want to take the volumes as they are (SnapMirror takes fewer steps)

A Mini Glossary

Term	Meaning	Where it appears in this article
SVM	A logical server inside the file system	Where MGN creates the FlexVol and LUN
FlexVol	A volume that holds data. LUNs live inside it too	MGN creates one for staging and one for the target
LUN	A unit of block storage, exposed over iSCSI	One source disk becomes one LUN
Snapshot	A point-in-time image inside a volume	MGN's SNAPSHOT phase creates it
FlexClone	A writable clone made from a Snapshot	The target FlexVol is one
Split	Detaching a FlexClone from its parent and materializing it	Finalize runs it

The Procedure and the Measured Time for Each Step

The entry point is the AWS Transform workspace. MGN runs the migration itself, but it is AWS Transform that handles everything from planning to landing zone and network as a single job.

#	Step	Measured
1	Initialize MGN	Done from the console. The CLI failed (see below)
4	Install the agent	Failed once (see below)
5	Initial full sync (16 GiB / 3 disks)	226 seconds
6	Test launch	SNAPSHOT 43 seconds, CONVERSION about 4 minutes 30 seconds
7	Cutover (correct order)	MGN job 649 seconds; 817 seconds from issuing it to confirming boot
9	Finalize	Split under 60 seconds; about 33 minutes until clean-up completed

(n=1 measurements at 16 GiB / 3 disks. Fixed overhead dominates, so they cannot be used as a basis for RTO.)

Snapshot and FlexClone: What AWS Transform's Phase Names Refer To

I lined up the ONTAP side by timestamp to see which ONTAP operation each AWS Transform phase name corresponds to.

AWS Transform phase	Corresponding ONTAP operation
SNAPSHOT	Creating a volume Snapshot of the staging FlexVol
LAUNCH	Creating a FlexClone from that Snapshot

The SNAPSHOT phase is not a plain data copy but a metadata operation (44 seconds for 8 GiB). The target volume is a clone with is_flexclone: true.

What matters most is that a FlexClone uses almost no physical capacity when it is created.
Measured just before Finalize, the target volume's logical size was 7.91 GiB while its physical consumption was only 35.5 MiB. Capacity planning has to look at physical values, not logical ones.

Finalize Is Not Clean-up; It Is Where Capacity Peaks

Running Finalize starts a split of the FlexClone (the operation that detaches it and makes it independent), which materializes the data.

Elapsed from T0	Observed change
Immediately	Lifecycle `CUTOVER`, replication `DISCONNECTED`
About 3 minutes	FlexClone split starts. Physical consumption rising from 35.5 MiB → 3.94 GiB
About 4 minutes	Split completes. Physical consumption 8.57 GiB
About 13 minutes	Staging FlexVol deleted, replication server EC2 terminated

There was a gap of about 9 minutes between the split completing and the staging volume being deleted.
During those 9 minutes, physical capacity holds twice the migrated data.

If you run Finalize while the aggregate's free space is below the migrated data size, it is likely to get stuck here. Keep in mind that Finalize is a capacity risk, not an availability risk.

The Sizing Guideline Is the Official "3x", Not the Measured "2x"

That peak in physical consumption (about 2x the migrated data) is a single value I observed in a 16 GiB environment. It shows how far physical capacity momentarily grew during Finalize, which is a different question from how much to provision the file system.

The official documentation gives 3x the migration data as the SSD capacity to provision (MGN: FSx for ONTAP configuration, Step 2, "Storage capacity"). The 3x breaks down into three consumers that hold capacity at the same time during migration:

The replicated data (staging)
The converted volumes used for launch (the target FlexClone)
The original volumes pending deletion (the staging FlexVol)

The reason the official figure is 3x even though I measured about 2x is that volume deletion is a background operation, so the freed capacity is not available immediately after deletion. Hence the guidance to provision the headroom up front. The documentation also recommends keeping SSD utilization at or below 80% throughout the migration.

If you read this article and conclude "it measured 2x, so size at 2x", you fall below the official guideline (3x) and are more likely to hit a capacity shortfall during Finalize. Base the provisioned figure on the official 3x, not on the measured peak.

Four Places I Stumbled

1. A Failed Final Snapshot Behind a Successful Job

To measure downtime, I shut down the source OS and then ran cutover. The job returned COMPLETED and the target LAUNCHED. It looked like a success, but the job log recorded this:

09:14:21  SNAPSHOT_START
09:19:22  SNAPSHOT_FAIL            timed out after 300 seconds
09:19:23  USING_PREVIOUS_SNAPSHOT

The final sync did not complete, and the job fell back to the previous snapshot. The cause was my procedure: shutting down the OS also stopped the replication agent, and taking the crash-consistent snapshot timed out.

The correct procedure is to stop only the application's writes and run cutover with the OS and the agent still running. "Stop writes on the source" in the official documentation does not mean shutting down the OS.

In this test there were no recent writes, so the data matched, but doing this in a real environment loses the latest data written after the snapshot it fell back to. What makes this risky is that it does not show up in the job status (status or launchStatus) at all. By the API's design, the job returns COMPLETED as long as processing finishes, even after a fallback.

Lesson: After cutover, always check event in describe-job-log-items. A COMPLETED job is not evidence that the final sync succeeded.

2. Two Unbootable Targets: NVMe Device Names Re-enumerated

I ran cutover in the correct order twice, and both produced instances that would not boot (a UEFI shell reboot loop). The job log showed no abnormal events, and the status was COMPLETED.

Examining the boot EBS volume of an unbootable instance, the 8 GiB boot volume held the contents of a 4 GiB data disk. The boot and data disk assignment had been swapped.

The cause: when the source was stopped and restarted as part of the test, NVMe device names were re-enumerated and the boot disk moved from /dev/nvme0n1 to /dev/nvme1n1. MGN's staging-type assignment appears to be keyed on the device name, and it is not re-evaluated after the name moves.

The official best practices say not to restart the source before cutover in the first place. If a restart does happen, check the disk assignment with a command like the one below and redo the test launch.

# Check which device is treated as the boot disk, and its assignment
aws mgn get-replication-configuration --source-server-id "$SRC" \
  --query 'replicatedDisks[?isBootDisk==`true`].[deviceName,stagingDiskType]' --output table

3. No Repair API, and Three Contradicting Errors

I called UpdateReplicationConfiguration to fix the disk assignment, but whatever parameters I passed, it returned contradicting errors, and the API could not fix it.

In the end I deleted the source server and reinstalled the agent, which turned out to be unnecessary: rerunning the installer is the documented procedure.
Rerunning the installer re-establishes the list of replicated disks and their mapping against the actual machine. To avoid losing launch template customizations, try rerunning the installer before deleting anything.

4. The Installer's Hint Did Not Match the Real Cause

When the agent installation failed, the installer displayed Are kernel linux headers installed correctly?. Reading the log closely, the real cause was lack of space in /tmp.

On Amazon Linux 2023, /tmp is tmpfs; on a t3.small it has only 955 MiB, which does not meet the installer's requirement (1 GB or more free).

sudo env TMPDIR=/var/tmp/mgn-build TEMP=/var/tmp/mgn-build TMP=/var/tmp/mgn-build \
  ./aws-replication-installer-init --region ap-northeast-1 --no-prompt

Pointing it at /var/tmp like this works around it. The installer's hint is not necessarily the real cause, so check the error code in the log.

Shrink the Leftover Capacity After Finalize to Keep Costs Down

The SSD capacity you provisioned at 3x for the migration is excessive once the migration is done. Leave it as is, and you keep paying for capacity you are not using. Here is what to keep in mind on cost.

Why the Leftover Is Billed

FSx for ONTAP storage has two tiers. The SSD tier (primary) is billed on the capacity you provision — you pay for the GiB you reserve, not for the bytes you actually use (Amazon FSx for NetApp ONTAP Pricing). The other tier, the capacity pool, is billed only for what you consume.

Right after migration, of the SSD capacity reserved at 3x the migration data, the reserved amount stays even after the staging and pending-deletion volumes are gone. Even when the real data drops back to 1, if the SSD tier is still reserved at 3, you keep paying for the 2 of unused capacity.

What to Do: On Second-Generation, Decrease SSD Capacity After Migration

Second-generation file systems (the Single-AZ 2 / Multi-AZ 2 deployment types) let you decrease SSD capacity after migration completes (File system storage capacity and IOPS). The documentation itself lists "after a large-scale data migration where temporary extra capacity was used" as a typical case for decreasing capacity. The steps and constraints:

First-generation (Single-AZ 1 / Multi-AZ 1) can increase but not decrease. If your plan is to reserve a large amount temporarily for migration, choosing second-generation is the safer choice on cost.
Each decrease is in steps of at least 9% of the current capacity, down to a size where SSD utilization stays at or below 80% after the decrease.
During the decrease operation you are billed for both the existing and the new size (for example, decreasing 10 TiB → 5 TiB is billed at 15 TiB during the operation, 5 TiB after). The decrease can take from a few hours to a few weeks.
After changing SSD capacity, IOPS, or throughput, you must wait a 6-hour cooldown before the next change.
A decrease operation pauses while a FlexClone exists. The MGN target volume is already independent after the Finalize split, so it qualifies; but with a FlexClone still present from before or during Finalize, the decrease will not progress. Run the decrease after Finalize completes.

Alternative: Stand Up a Migration-Only File System and Move to Production with SnapMirror

The decrease has constraints. First-generation cannot shrink; even second-generation has the 9% step, the 80% ceiling and the 6-hour cooldown, is billed at both sizes during the operation, and can take weeks. So another design is to separate the migration landing spot and the production home from the start.

Stand up a migration-only file system provisioned at 3x. Make it the ATX (MGN) migration target.
Run through Finalize and let the target volume become independent.
Stand up a separate production file system at the minimum size that fits the real data.
Replicate the volume from the migration-only to the production file system with NetApp SnapMirror. SnapMirror is block-level replication that carries the deduplicated and compressed state and the snapshots across intact. It supports in-Region replication between two FSx for ONTAP file systems (Replicating your data using NetApp SnapMirror).
Switch clients to the production file system and delete the migration-only file system in full.

This alternative helps when you use first-generation, or when you want to shrink substantially after migration. The decrease is an operation that shrinks a file system and is unavailable on first-generation; SnapMirror moves the contents to a separate file system and so is not bound by the generation. Delete the migration-only side and billing stops along with the whole 3x reservation. AWS also presents SnapMirror as a way to migrate from first-generation to second-generation (Migrating SQL Server FCI Storage from FSx for ONTAP Gen 1 to Gen 2).

It does add steps, though. You go through cluster peering → SVM peering → creating the SnapMirror relationship → initialize → final update → break → mount, and the migration-only and production file systems run in parallel for a while, so both are billed during that time. In this article the alternative was only checked against the documentation and not run on real hardware (not verified). Which of the decrease and SnapMirror is cheaper and faster depends on data size, generation and the shrink ratio, so estimate it for your own conditions.

Cost Measures Beyond the Decrease

Tier infrequently accessed data to the capacity pool. A volume's tiering policy moves unused data to the cheaper tier. The lower the SSD utilization, the smaller the size you can decrease to.
Reduce throughput capacity after migration too. You provision it high for speed during migration, and throughput is also billed. The documentation states you can reduce throughput after migration completes (MGN: FSx for ONTAP configuration, "Throughput capacity"). During an SSD decrease operation, however, changing SSD, IOPS, and throughput at once is constrained, so do them one at a time with the cooldown in between.
Delete the NLB and the VPC endpoint service. This is separate from capacity, but left alone they keep incurring hourly and LCU charges (see the teardown checklist below).

Teardown Checklist

Finalize does not finish the clean-up. The correct teardown order for leaving nothing behind is as follows (steps 1–7 depend on each other).

Terminate the target and source EC2 instances
Delete the MGN source server
Reject the VPC endpoint connection, then delete the endpoint service
Delete the NLB and its target group (important: charges continue until you delete them by hand)
Delete the target FlexVol with fsx delete-volume
Delete the SVM with fsx delete-storage-virtual-machine
Delete the ONTAP security login

The staging EBS volumes are deleted automatically with a delay, but the target's root EBS volume has no DeleteOnTermination and must be deleted manually.

Initializing AWS Transform from the CLI: Create the IAM Roles First

If running aws mgn initialize-service from the API or CLI fails, the cause is that the IAM roles have not been created.
Initializing from the console creates the IAM roles for you, but on the CLI path you need to create eight roles (including AWSApplicationMigrationFsxProxyRole for FSx for ONTAP) and attach their policies first.

Closing

With this GA, a migration now finishes with the data area already on FSx for ONTAP as iSCSI LUNs, and the step of moving the data afterwards is gone.

Any server whose boot and data are separate fits the same shape, whatever the original storage.
On the other hand, watch for the configuration spanning two kinds of storage, the physical capacity spike during Finalize, and MGN-specific traps such as "a job status of COMPLETED does not mean the final sync succeeded". Plan capacity as a round trip: provision the 3x SSD capacity the documentation calls for, then, once migration is done, hand back the surplus with a second-generation capacity decrease to bring the cost down.

I hope this test record helps anyone migrating servers with separate boot and data to EC2 while considering FSx for ONTAP as the home for the data area.

Resources

AWS What's New: AWS Transform announces general availability of Amazon FSx for NetApp ONTAP support
MGN: FSx for ONTAP configuration
MGN: Best practices for AWS Transform MGN
FSx for ONTAP: Using Amazon Elastic VMware Service with FSx for ONTAP
Verification report (GitHub): the test environment configuration, the measured teardown procedure, and the list of feedback sent to AWS, all omitted from this article

All test environments have been deleted. The figures are single measurements under a specific environment and conditions, and vary with data size and configuration. The configuration details are in the verification report above.

This article is Part 2 of the series "The Data Foundation for AWS Modernization." The Japanese original is on hatenablog.

Designing AWS Modernization with VMware Migration as the Entry Point — Why FSx for ONTAP as the Data Foundation

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 28 Sep 2026 13:42:08 +0000

Introduction

What comes "next" for VMware workloads? Hardly a day goes by without the question coming up.

The licensing changes that followed Broadcom's acquisition of VMware have pushed many organizations into a fundamental review of their virtualization strategy. But it is worth pausing here: "getting off VMware" is not the goal. The real destination is the modernization beyond the migration — reworking applications and data, up to and including serverless and cloud-native architectures. The AWS Storage Blog, too, frames the migration not as mere license avoidance but as an opportunity to modernize infrastructure. (Reference: AWS Storage Blog — Expedite VMware migration to Amazon EC2 and Amazon FSx for NetApp ONTAP using the BlueXP workload factory migration advisor)

The Core of Modernization on AWS — AWS Transform

What drives modernization on AWS today is AWS Transform, an agentic-AI service. Beyond rehosting VMware workloads, it can containerize source code (replatforming to Amazon ECS / Amazon EKS) and take .NET Framework applications to cross-platform .NET on Linux — capabilities that run migration and modernization in parallel. It lifts migration from "moving servers" to "reworking applications," and that is what modernization on AWS looks like now.

However you rework the compute — EC2, containers, or serverless — one thing is always needed: a place for the data. This is where Amazon FSx for NetApp ONTAP, which I work with, comes alongside.

I field questions about FSx for ONTAP every day, and one thing these migrations tend to overlook is the continuity and extensibility of the storage operations model. Keep the storage operations built up over years in VMware and ONTAP environments — Snapshot, Clone, Replication, Storage Efficiency — on AWS too, and let a multi-protocol data hub support everything from migration through to cloud-native. Keep the data layer consistent and you can re-choose the compute as many times as you like. Designing modernization with AWS migration as the entry point and the data foundation at the core — that is the lens of this series.

Five Destinations for VMware Workloads

AWS officially presents five pathways for VMware workloads. (Reference: AWS for VMware — Comprehensive Pathways)

AWS for VMware: The Pathways to Migrate and Modernize on AWS

I drew the five official pathways as five rows, combining AWS services with ONTAP:

Row 1 is Migrate to Amazon EC2: Amazon EC2 reaches Amazon FSx for NetApp ONTAP over iSCSI.
Row 2 is Modernize on AWS: Amazon Elastic Container Service, Amazon Elastic Kubernetes Service, AWS Fargate, Amazon WorkSpaces, AWS Lambda, AWS Batch and Amazon FSx for NetApp ONTAP.
Row 3 is Run VMware on AWS: Amazon Elastic VMware Service reaches Amazon FSx for NetApp ONTAP over NFS / SMB.
Row 4 is Run AWS on-premises: AWS Outposts and NetApp ONTAP.
Row 5 is Run third-party hypervisors on AWS: Red Hat OpenShift Service on AWS, Amazon FSx for NetApp ONTAP, and Nutanix Cloud Clusters on AWS.

(dark theme)

Source: AWS for VMware Partner Offerings
— "AWS offers the most comprehensive set of migration and modernization options for
VMware-based workloads - from relocating to Amazon EVS, to rehosting on Amazon EC2,
containerizing with Amazon EKS, or transitioning to running third-party hypervisors
in the cloud like ROSA and NC2 on AWS." (Summarized; rephrased for licensing constraints.)

Each Pathway in Detail

1. Rehost to Amazon EC2

The most straightforward path: migrating VMware VMs as EC2 instances. AWS Transform for VMware (an agentic-AI-based migration service) became generally available in May 2025. (Reference: AWS What's New)

Separately, on 30 August 2026, FSx for ONTAP became generally available as a target storage type for AWS Transform for migrations (MGN). The former is a service for migrating from VMware; the latter is a target-storage choice for server migration. They are different features, and Part 2 of this series covers the latter. (Reference: AWS What's New)

Storage configuration options:

EBS only: Simple. Automated with MGN / AWS Transform
EBS (OS) + FSx for ONTAP (Data): Keeps ONTAP capabilities. Supported by AWS Transform (MGN) / Shift Toolkit / Cirrus Migrate Cloud (the scope of this series)

2. Modernization (Containers / Serverless)

As the next step after rehosting to EC2, you can modernize according to the characteristics of each workload.

Target	Suitable workloads	FSx for ONTAP integration
ECS / EKS (EC2 mode)	Stateful containers (DB, middleware)	✅ iSCSI / NFS mount available
ECS / EKS (Fargate)	Stateless microservices	△ Through EFS only (no iSCSI)
AWS Lambda	Event-driven, short-running processing	△ EFS mount available, no iSCSI
AWS Batch	Batch processing / HPC	✅ iSCSI available in EC2 mode
Amazon WorkSpaces	VDI (virtual desktops)	✅ FSx for ONTAP file shares

This is not about containerizing VMs as they are. The journey is "rehost to EC2 → containerize the application → move step by step to Fargate/Lambda".

3. Amazon EVS (Keep VMware on AWS)

With Amazon Elastic VMware Service, you can deploy VMware Cloud Foundation (VCF) directly on EC2 bare metal inside your VPC. You keep your existing vSphere skill set and can connect FSx for ONTAP as an external datastore.
It suits cases with a large VMware-dependent application estate, where leaving VMware in a short time is impractical.

4. AWS Outposts (On-premises AWS + NetApp External Storage)

AWS Outposts is a fully managed service that places AWS infrastructure on premises.
Third-party block storage integration was announced in December 2024, and NetApp ONTAP and StorageGRID are available as partners validated through the AWS Service Ready Program. (Reference: AWS Blog) (Reference: NetApp)

NetApp ONTAP iSCSI LUNs can be attached directly from the AWS console as data volumes for EC2 instances, and boot volume support was added in July 2025.
This gives you separation of compute and storage: "compute is AWS-managed, storage is your existing ONTAP."

5. Partner Solutions (ROSA / Expanding Nutanix + NetApp Integration)

Red Hat OpenShift Service on AWS (ROSA): A fully managed environment based on OpenShift. With the NetApp Trident CSI driver, ROSA Pods can access FSx for ONTAP over NFS/iSCSI. (Reference)
Nutanix + NetApp ONTAP (on-premises Nutanix Cloud Platform): At .NEXT Chicago in April 2026, a partnership was announced that lets the Nutanix Cloud Platform (NCP / AHV) use NetApp ONTAP (AFF / FAS) as external storage. Over an NFS-based connection, compute (Nutanix AHV) and storage (ONTAP) scale independently. GA is planned for the second half of 2026 (NetApp targets Q3 2026, with Early Access underway). The partnership covers on-premises NetApp enterprise storage; integration with Nutanix Cloud Clusters (NC2) on AWS or with FSx for ONTAP is not announced or committed in public sources (NetApp mentions "NetApp cloud storage services for Nutanix Cloud Platform" as a future direction, with the timing and the services undetermined). (Reference: NetApp Blog, GA Q3 2026) (Reference: Nutanix press release)

The NetApp ecosystem as a whole:

Figure 2: the data layer stays the same when the compute changes. SnapMirror is what crosses the boundary (dark theme)

What this figure shows is that ONTAP is a data platform independent of the choice of hypervisor or cloud. Whether you move the compute layer from VMware to EC2 or to Nutanix, the data layer stays consistently available, and SnapMirror moves and protects the data.

Where This Series Sits

Figure 3: the phases beyond rehosting, and the alternatives to it. This series covers Phase 1 (dark theme)

This series concentrates on Phase 1 (rehost), but the EC2 + FSx for ONTAP configuration is designed not to close off the path to Phase 2 and beyond. FSx for ONTAP is reachable over NFS/iSCSI from ECS/EKS as well as from EC2, so the data layer can stay as it is after containerization.

Why "EC2 + FSx for ONTAP"

Among the five pathways, here is why this series looks at "EC2 + FSx for ONTAP" in particular.

An Entry Point for Rehosting and a Foundation for Modernization

This configuration is not just a rehost destination; it is designed not to close off future modernization.

Now (Phase 1): Rehost VMs to EC2. Data disks on FSx for ONTAP iSCSI
Next (Phase 2): Containerize the application. Keep accessing FSx for ONTAP over NFS/iSCSI from ECS/EKS
Later (Phase 3): Parts that became stateless move to Fargate/Lambda. The FSx for ONTAP data layer stays as it is

The multi-protocol access of FSx for ONTAP (NFS/SMB/iSCSI) is what supports this step-by-step migration. A volume used as an iSCSI LUN for EC2 can later be NFS-mounted by an EKS Pod.

The Value of ONTAP Is Capability, Not Capacity

Seen as just "large-capacity storage," FSx for ONTAP can look expensive next to EBS. But capacity is not where the value of ONTAP lies.

ONTAP feature	Use on AWS
Snapshot	Point-in-time copies in seconds. Instant test environments
FlexClone	Clones without copying data. Lower cost for dev/test
SnapMirror	Block-level replication. Cross-Region DR
Compression / Dedup	Lower effective capacity. Most effective for databases and logs
Thin Provisioning	Pay for what is used. Avoid over-provisioning
Multi-protocol	Access the same volume over NFS/SMB/iSCSI

FSx for ONTAP EC2 Integration Pattern

Figure 4: the boot column is Amazon EBS in all three rows; only the data column changes to FSx for ONTAP from staging on (dark theme)

This configuration has the following advantages.

Storage and compute scale independently: Change FSx for ONTAP capacity/throughput without stopping EC2
The ONTAP operations model carries over: Operate snapshots, clones and replication with the same CLI/API as on premises
The limit moves: The ceiling is no longer the per-instance Amazon EBS limit but the EC2 network bandwidth and the FSx for ONTAP throughput capacity.

Note: Performance Ceilings and Protocols

"The limit moves" above does not mean the limit goes away. This series has not measured it yet, but verification in a sibling project (F-1 iSCSI measurement and others) has shown the following.

Simple arithmetic between session count and bandwidth misses Estimating "required bandwidth ÷ 625 MBps = number of sessions" misses in both directions. One TCP connection came out at 1.82x the prediction and sixteen at 0.20x, so the ceiling is not set by the capacity of a single flow alone.
The performance gap between protocols (iSCSI vs NVMe/TCP) depends on the deployment Measured on the same file system, iSCSI and NVMe/TCP were 0.06% apart — practically the same — while on other configurations their order swapped. Quoting a past ratio as the basis for a protocol choice is unsafe.
ANA multipath is not available on Amazon Linux 2023 CONFIG_NVME_MULTIPATH is unset in all three AL2023 kernel series (details). The ANA path makes a large difference to sequential reads, so verifying it enabled needs a different distribution (RHEL 9.3 or similar).

These values come from tests against a raw device and include burst; they do not apply directly to this configuration, which puts a file system on the LUN.

Choosing a Migration Tool

Even when the destination is "EC2 + FSx for ONTAP," the tool depends on the environment.

Condition	Candidate tool	Notes
You want the data volumes on FSx for ONTAP (whatever the source storage is)	AWS Transform (MGN)	Set FSx for ONTAP as the target storage type. Generally available since 30 August 2026. Boot is always Amazon EBS. Verified hands-on in Part 2
Already on an ONTAP NFS datastore + small to medium scale	NetApp Shift Toolkit	Converts disks without copying them, using FlexClone. EC2 / FSx for ONTAP support is Early Preview. Verified hands-on in Part 4
Already on ONTAP + large scale (100+ VMs)	Cirrus Migrate Cloud	YAML-driven automation. Paid, through AWS Marketplace
Migration planning and sizing only	AWS Application Discovery Service / AWS Migration Evaluator	Inventory collection and cost estimation. AWS native

Where NetApp Shift Toolkit Fits

Shift Toolkit converts VMs on an ONTAP NFS datastore without copying their disks. Conversion is done with FlexClone and metadata operations, so the conversion itself is quick.
Migrations from VMware ESXi to Hyper-V, OpenShift Virtualization, Proxmox VE and OLVM are GA today, and support for VMware ESXi → Amazon EC2 / FSx for ONTAP is at the Early Preview stage.

⚠️ Early Preview note: The current scope is configurations that place data disks on FSx for ONTAP. Specifications and constraints may change.

What This Series Covers

Starting from migration as the entry point, the series runs four parts through to the modernization beyond it. Parts 2 and 4 are the two migration paths; Part 3 is the modernization that follows.

Part	Theme	Status	Source storage
Part 2	AWS Transform Now Supports Block Storage Migration to FSx for ONTAP — Benefits and Pitfalls from a Hands-On EC2 Test	GA (30 August 2026) / verified hands-on	Any
Part 3	Beyond the AWS migration of a VMware environment: serverless, cloud-native, and the data foundation (ECS / EKS, S3 Access Points, DR)	Upcoming	—
Part 4	NetApp Shift Toolkit	EC2 / FSx for ONTAP support is Early Preview / verified hands-on	An ONTAP NFS datastore

Of the two migration paths, AWS Transform is already GA and applies more widely, so the one a reader can try today (Part 2) comes first. Shift Toolkit (Part 4), which builds on existing ONTAP assets, comes later because its EC2 support is Early Preview. Part 3 sits between them and covers how to provide the data foundation when migrated workloads are reshaped into containers and serverless.

Across the parts, we plan to cover:

Environment setup: Design and build of VPC + FSx for ONTAP + VPN
Migration: Migrating Linux / Windows VMs to EC2
Modernization: Containerization on ECS / EKS, Fargate with S3 Access Points, DR and cyber resiliency
Operational continuity: Snapshot / Clone / SnapMirror behavior after migration
Cost comparison: TCO against an EBS-only configuration

Summary

Migrating from VMware is not only about "where to go" but about "how to carry the storage operations model forward".
For ONTAP users, FSx for ONTAP is a strong option for keeping, and making full use of, the value of ONTAP on AWS.
Choose the tool by source storage and scale. If the source storage does not matter, AWS Transform (MGN); for ONTAP NFS datastores at small to medium scale, Shift Toolkit.
Shift Toolkit's EC2 support is at the Early Preview stage, so those verification results may change at GA.

The next article (Part 2) covers the AWS Transform (MGN) path that sets FSx for ONTAP as the target storage, verified hands-on after GA. Part 3 covers the modernization beyond migration (serverless and cloud-native), and Part 4 explains how Shift Toolkit works — how FlexClone changes VM migration.

Reference Links

Verification repository (GitHub): VMware-Migration-EC2-ONTAP — procedures, configuration and figures for this series
Modernization journey map (FSx for ONTAP Adoption Playbook) — which material and implementation pattern to read at each stage beyond migration
AWS for VMware — Comprehensive Pathways
AWS Transform for VMware
AWS What's New: AWS Transform announces general availability of Amazon FSx for NetApp ONTAP support
MGN: FSx for ONTAP configuration
Nutanix Blog: NetApp and Nutanix Announce Technical Partnership
AWS for VMware Partner Offerings (ROSA, NC2)
NetApp Shift Toolkit Overview
AWS Storage Blog: Seamless migration from VMware to FSx for ONTAP and EC2
AWS Storage Blog: Expedite VMware migration to Amazon EC2 and Amazon FSx for NetApp ONTAP using the BlueXP workload factory migration advisor
Amazon FSx for NetApp ONTAP
Amazon Elastic VMware Service (EVS)
Red Hat OpenShift Service on AWS (ROSA)
Nutanix Cloud Clusters on AWS (NC2)
AWS VMware Migration Accelerator
NetApp Blog: Simplify VM migration with Shift Toolkit

This article is Part 1 of the series "The Data Foundation for AWS Modernization." The Japanese original is on hatenablog. Shift Toolkit's EC2 support is Early Preview, and its specifications are subject to change.

	Article	Reader
A	Sessions and queues	People about to mount
B	Confirming whether multipathing is usable	People who followed the procedure and got stuck
C	What moves the numbers (this article)	People who measure the numbers, and people who cite them

	Article	Reader
A	Session and queue counts (this article)	Anyone about to mount
B	Checking whether ANA is available	Anyone who followed the procedure and got stuck
C	What moves the numbers	Anyone measuring or citing figures

	Article	Reader
A	Session and queue counts	Anyone about to mount
B	Checking whether multipathing is available (this article)	Anyone who followed the procedure and got stuck
C	What moves the numbers	Anyone measuring or citing figures

Client	Min	Median	Max	Spread
Rocky 9.7	842.70	1,040.26	1,092.50	29.6%
RHEL 9.7	1,105.75	1,112.65	1,129.30	2.1%

DEV Community: Yoshiki Fujiwara(藤原 善基)@AWS Community Builder

Turning a Step Functions workflow into AI agent tools through AgentCore Gateway

The update this post is about

What this post covers

What this post does not cover

Why I did this

Where this fits

What became possible

The pattern I added

How it differs from the existing AgentCore integration

It completes in IaC

Design points that are easy to trip over

On Japanese input

Running it on real infrastructure

On completeness

Wrapping up

Modernizing an App on Amazon FSx for NetApp ONTAP: Four .NET / Java / Python Candidates, Plus Containers and AI — Six Paths Measured (Once per Case)

Changing the runtime and the access path without moving the data: how authentication, authorization, and failures differ

Intended readers, and what this article does not answer

Reading order

What this article covers / does not cover

Terms used in this article

The delta from the previous article

The paths at a glance

Verification environment and method

Configuration (A, B, C, C-REST)

Configuration (D, E-1, E-2)

Method

Candidate A (.NET / NFS): the previous article in summary

Candidate B (Java / NFS): FileLock behavior is invisible without two processes

Candidate C (Python Lambda / FSx for ONTAP S3 Access Points): IAM authorizes

Candidate C-REST (Python Lambda / ONTAP REST API): ONTAP authorizes, plus the self-signed certificate and multipart traps

Path D (EKS and Trident): using FSx for ONTAP as a container's persistent volume

Path E (services that read through S3 Access Points): analytics and RAG

E-1 Analytics: Athena reads Parquet

E-2 RAG: narrowing search results by permission

How failures return per path, and what was confirmed for D and E

How to choose a path

Summary and the path options

Resources

Keeping the Data Where It Is, Modernizing Only the App From .NET Framework to Lambda — What Changes and What Does Not (A 1-Environment Test)

Keeping the data you already have, modernizing only the app — what changes across SMB, NFS, and S3 Access Points

When this post fits and when it does not

Reading by purpose

What this post covers and what it does not

Terms used in this post

The modernization work so far, and the new step this time

Differences from the previous article

Staged design and the single-volume constraint

Handing out working volumes with FlexClone (not done in this verification)

How this project used it

Use cases the documentation describes

Relationship to the single fixed volume

How to choose

The value of S3 Access Points and its four prerequisites

Test environment and method

Configuration

Method

Evidence labels and run ids

Stages 0 and 1: unchanged UUID and seed, changed root handling

Stage 2: atx up to net10.0, AIMF up to ADRs and a ledger

Stage 3: Lambda reaching the same volume through S3 Access Points

Behavior by access path

Eight failures and the checks to put in place before measuring

The eight failures

Five preventive checks

Choosing between atx and AIMF

Cost estimates and observations

Hourly-rate estimates against observed values

Identified cost of the whole verification

Costs that could not be attributed, and the teardown check

Costs that could not be attributed, and why

Teardown check results

What has not been measured yet

Summary and choosing an access path

Resources

Can the QoS burst defaults on FSx for ONTAP be changed from ONTAP 9.19.1, the same as on-premises? I checked on a live file system

What this post covers, and what it does not

Who benefits from this setting

What changed from the previous post

DEV Community: Yoshiki Fujiwara(藤原善基)@AWS Community Builder