Changing the runtime and the access path without moving the data: how authentication, authorization, and failures differ
A long-lived business application often keeps the data it reads and writes on a file share. For rebuilding such an application on a newer language or runtime, Microsoft Learn lists dependencies reaching end of support, new security or compliance requirements, and performance or scalability limits as the signals (Upgrade .NET apps overview). Its examples include moving a Windows-only .NET Framework app to cross-platform modern .NET, and beyond that, containerizing the app or replacing components with managed services.
When you rebuild an application, what usually stalls the effort is not the application itself but what to do with the data it has been reading and writing. Moving the data to a new store brings along downtime during the migration, redesigning permissions, re-pointing reference paths, and re-validating on the destination. This bundle is what stops modernization.
Not moving the data removes that whole bundle, because it decouples rebuilding the app from migrating the data. You get three things:
- You avoid the migration work itself. No downtime, no permission redesign, no path re-pointing, no destination re-validation. You can focus only on changing the app's language or runtime.
- The data stays a single source of truth that several runtimes share at once. No extra copies to keep in sync. (Keeping the data in one place so copies and synchronization are unnecessary is the single-source-of-truth, SSOT, idea.) This article measured six paths reaching the same single volume, with the data unchanged.
- You can choose the language and runtime independently of where the data lives. The same data is reachable from .NET, Java, or Python, and from EC2, Lambda, or EKS.
This matters when the existing data is large and costly to move, when the permission design is complex and you would rather not rebuild it on the destination, when several users or services use the same data at once, or when you cannot take the downtime a migration needs.
Amazon FSx for NetApp ONTAP (FSx for ONTAP below; NetApp's ONTAP as an AWS-managed shared storage service) makes this possible, because it lets you use one volume through several doors: NFS, SMB, and the S3 API. AWS documents that S3 Access Points let you read and write file data with S3 object operations, without application changes and without moving the data out of file storage, and that you can keep using NFS and SMB alongside (Accessing your data via Amazon S3 access points). On the container side, the Amazon EKS User Guide describes using NetApp Trident so that FSx for ONTAP serves as the storage behind Kubernetes persistent volumes (Use high-performance app storage with FSx for NetApp ONTAP).
But when you add access paths without moving the data, who authenticates, where authorization happens, and how failures return change per path. Add a path without knowing this, and you find the difference afterward. So this article measures reaching the same data through six paths and how authentication, authorization, and the shape of failures swap per path.
Last time I took one .NET Framework app and, keeping its business logic unchanged while renewing only the language and runtime — the kind of migration AWS calls Refactor — rebuilt it step by step down to AWS Lambda without moving its data, and measured how SMB, NFS, and S3 Access Points differ when opening the same volume. This article continues from there by adding the runtimes and paths that reach the data. I measured SMB last time, so I did not measure it this time.
Even after the app migration, the access path changes with how you run the app in the cloud: whether the same data is mounted from EC2, read with the S3 API from Lambda, or used from a container on EKS. This time I measure that next step after the migration — choosing the access path — by adding runtimes.
I measured the following six paths. I call E-1 and E-2 together E.
| Path | What uses the volume, and how |
|---|---|
| A | A .NET 10 app on a Linux EC2 instance mounts the volume over NFS and reads and writes it |
| B | A Java (Spring) app on a Linux EC2 instance mounts the volume over NFS and reads and writes it |
| C | A Python Lambda reads and writes the volume through S3 Access Points (the S3 API) |
| C-REST | A Python Lambda reads the volume through the ONTAP REST API |
| D | Containers on EKS write to persistent volumes that Trident provisions |
| E-1 | Athena (an analytics service) reads Parquet files with SQL through S3 Access Points |
| E-2 | Amazon Bedrock Knowledge Bases (a RAG service) ingests documents through S3 Access Points, and a filter inside a web app narrows the search results to the user's permissions |
- Four of them (A, B, C, C-REST) ran the same file-processing job, reimplemented in .NET, Java, and Python, against the same single volume. Because the job is the same, these four can be compared with the difference attributed to the path. I call each reimplementation a "candidate" in this article.
- D and E are not the same job. I tested different workloads, in other repositories, in other disposable environments. I do not compare them with A through C-REST; I report each path's behavior on its own.
I did not run the same job over all six paths.
First, the premises. All six paths ran on the same single file system (Single-AZ, ap-northeast-1; an existing verification file system, separate from the one I measured in stages 0 to 3 of the previous article), using disposable SVMs, volumes, and other resources I created. I read the ONTAP version in the measurements for candidates A, B, C-REST, and path D (across several days): 9.18.1 (NetApp Release 9.18.1P6). The C and E measurements did not read it and carried the value over. A different version can change a result, so read the results for C and E as results whose version I did not confirm at the time. A "case" here is a check fixed beforehand for each path (that it runs as expected, that it fails as expected on a deny or an unreachable target, and so on), and I ran each case once. I did not repeat runs to confirm reproducibility. None of these values is a guarantee for production.
For A through C-REST, the job I measured is the core of DocIntake, the same sample app as last time. It is not a real business app or a production app. It has no screen (UI) and no web layer: a minimal batch job that reads files on the volume, processes them, and writes a result, reimplemented per language. The results do not carry over to an app with a UI.
You may be wondering about these three points.
- When the same file-processing job runs in languages other than .NET (Java, Python) over NFS, S3 Access Points, and ONTAP REST API, what changes in authentication, authorization, and how failures return?
- When containers on EKS, or analytics and RAG services, use the same volume, what could I confirm, and what is outside this article?
- Which of the six paths should I choose, and on what basis?
I found three things. I answer the three questions above in this order.
- The answer to the first question is that the authentication principal, the network boundary, the authorization principal, and how failures return all change per path. For example, when access was denied, NFS (A and B) returned an OS permission error, S3 Access Points (C) returned HTTP 403 (AccessDenied), and ONTAP REST API (C-REST) returned HTTP 403 with an ONTAP error code. The four candidates in .NET, Java, and Python ran their designed cases, without moving the data, against disposable volumes on the same FSx for ONTAP file system.
- The answer to the second question has three parts. On D, I wrote data from a verification pod on EKS to iSCSI and NFS persistent volumes that I provisioned with Trident, and the same data was readable after a pod restart and after a reschedule to another node. On E-1, Athena read Parquet through S3 Access Points and the aggregation matched the values I computed from the source data. On E-2, a filter inside the web app removed documents the user was not permitted to see from the Bedrock Knowledge Bases search results, according to the user's permissions. All three are small-scale results for workloads other than A through C-REST.
- The answer to the third question is that authorization happens in a different place per path, so choose by the kind of client you use and where you want authorization to happen. The evidence is the tables "The paths at a glance" and "How failures return per path, and what was confirmed for D and E". I do not rank them.
What you can take back to your own environment is this. Each time you add a runtime or a path to the same volume, re-measure on the new path who authenticates (the OS, IAM, an ONTAP credential, the app), where the network goes (an NFS mount, the public S3 endpoint, a management LIF inside the VPC), where authorization is decided (OS permissions, IAM, ONTAP, a filter in the app), and how a failure returns (a permission error, an HTTP status, an ONTAP error code, removal from the search results).
Intended readers, and what this article does not answer
The intended readers are SAs, application developers, storage administrators, and Partner/SI engineers who are modernizing applications on AWS, or considering it. Not only those already using FSx for ONTAP, but also those still choosing a destination. I picture the moment of being stuck on "I want to change the language or runtime, but what do I do with the data?", or of choosing between NFS, S3 Access Points, ONTAP REST API, and Trident.
This article does not answer the following, because I did not measure them or measured them under different conditions.
- Performance (I ran no measurement with defined performance conditions).
- Guarantees for production (every value is one observation per case).
- Multi-AZ behavior (I built only Single-AZ).
- Behavior of an app with a UI (the A through C-REST sample has no UI).
- Large data or long-running operation (I used only small datasets).
Reading order
- To get the conclusion only, "I found three things" in this section is enough.
- To see the differences between paths at a glance, read "The paths at a glance" and "How failures return per path, and what was confirmed for D and E".
- To decide which path to choose, read "How to choose a path".
- Per-path detail is in the sections for candidates B, C, C-REST, path D, and path E.
- The axes this article does not cover are collected in "What this article covers / does not cover".
What this article covers / does not cover
This article reports six paths, each measured once per case in its own environment. It is not an exhaustive comparison.
What I cover
- How authentication, network, authorization, and failure behavior differ when six paths use the same volume. A, B, C, and C-REST ran the same job over different paths, so I can attribute the difference to the path. D and E are different workloads, so I report each path's behavior on its own.
- How the failure cases (deny, unreachable, invalid identifier) returned on each path. What returns differs per path (a permission error, HTTP 403, an ONTAP error code, a connection timeout).
- Each case was measured once. Where I got through by hand, and where I did not retrieve a value, I say so on the spot.
What I do not cover
- Performance. The purpose was to measure behavior differences per path, and I ran no measurement with defined performance conditions. Performance and scale are tracked in #38.
- Multi-AZ and other ONTAP versions. I built only a Single-AZ configuration, and the measurements that read the ONTAP version read 9.18.1P6. Path D's Multi-AZ switchover and same-file concurrent-write consistency are tracked in #17 in Container-Datastore.
- Response time. I did not measure path E-2's response time or the natural expiry of the Lambda cache (waiting five minutes); they are tracked in #209. Path E-1's behavior when the domain controller is unreachable is #155 in Lakehouse.
- SMB. I measured it in the previous article. I also do not cover containers on ECS or Windows containers that mount over SMB. For path D, I tested only one path, EKS with Trident.
- Reconciling with invoices. Costs are estimates for approved disposable environments, pre-bill values.
- Whether this applies to real data. E-1 and E-2 were measured with non-confidential synthetic data and placeholder SIDs. Data classification, retention policy, and any legal or compliance judgment before applying it to production business data or regulated data are outside this article.
- Whether to use the RAG's generated output. E-2 measured only the permission-based narrowing of search results; whether a generated answer built from the retrieved documents may be used directly for a decision (clinical or otherwise) needs separate consideration.
Terms used in this article
| Term | One-line meaning |
|---|---|
| Path | The route by which a runtime reaches a volume on FSx for ONTAP. In this article there are six: A, B, C, C-REST, D, and E |
| Job | The core processing of the sample app DocIntake: a minimal batch unit that reads a document from an intake folder, processes it, and writes a result report |
| Candidate | A different implementation of the same job. Used only for A, B, C, and C-REST. D and E are not the same job, so I do not call them candidates |
| The same job | Running the job above as the same processing even when the language or the path changes |
| NFS | The protocol Linux uses for file sharing. This article uses NFS 4.1 with AUTH_SYS (a simple authentication where the client claims a uid) |
| S3 Access Points | A door that reads and writes an FSx for ONTAP volume through the S3 API. AWS IAM, the access-point policy, and a file-system user authorize requests. This is an S3-compatible door, not an Amazon S3 bucket itself. The backing store is the same ONTAP volume as NFS/SMB, so reaching one volume through several protocols shares its throughput |
| ONTAP REST API | A REST door for managing and operating ONTAP. Authentication is an ONTAP credential and ONTAP authorizes |
| Management LIF | ONTAP's network interface for management. It is reachable only from inside the VPC |
| SVM | ONTAP's virtual storage server. In this article I created a disposable SVM per verification |
| Trident | NetApp's CSI driver, which provisions Kubernetes persistent volumes (PVs) as FSx for ONTAP volumes and LUNs |
| PV and PVC | A PV is a persistent volume Kubernetes manages; a PVC is a pod's claim for one |
| RAG | An approach in which a generative model answers using documents retrieved by search |
| SID | A value that identifies a Windows user or group. In the RAG verification, the SIDs that permit each document are matched |
| fail-closed | Erring toward the safe (closed) side when information is missing. In this article it refers to E-2's runtime behavior: a user with no permission information gets all search results denied |
The delta from the previous article
The previous article (the previous article in this series) rebuilt one .NET Framework app from stage 0 to stage 3 and measured how file handling (case sensitivity, exclusive locking, write visibility, and so on) differs as the ways to open the same volume increased across SMB, NFS, and S3 Access Points. It had one implementation (.NET), and its axis was how file handling changes when the protocol changes.
This time three axes change. One is more languages (Java and Python in addition to .NET). Another is a new access path, the ONTAP REST API. With these two, what I look at moves from file handling to authentication, network, authorization, and how failures return. The third is workloads other than the job itself that use the same volume: containers on EKS (D) and analytics and RAG services (E), which I tested in other repositories.
Candidate A (.NET / NFS) is the same lineage as stages 2 and 3 last time, so I keep it to a summary and point to the previous article and the issue that records its runtime verification (#42). What I newly measured is B, C, C-REST, D, and E.
The paths at a glance
Here are the six paths in one table. Only the four with "Yes" in the "Same job?" column can be compared with each other.
| Path | Same job? | Runtime | How it reaches the volume | Authentication and authorization | Repository where tested |
|---|---|---|---|---|---|
| A | Yes (DocIntake) | .NET 10 / Linux EC2 | NFS 4.1 mount | The OS (uid) authenticates; OS permissions authorize | This repository |
| B | Yes (DocIntake) | Java / Spring worker / Linux EC2 | NFS 4.1 mount | The OS (uid) authenticates; OS permissions authorize | This repository |
| C | Yes (DocIntake) | Python 3.14 Lambda (outside the VPC) | S3 Access Points (S3 API) | AWS IAM authenticates; IAM and the access-point policy authorize | This repository |
| C-REST | Yes (DocIntake) | Python 3.14 Lambda (inside the VPC) | ONTAP REST API (management LIF) | An ONTAP credential (basic auth); ONTAP authorizes | This repository |
| D | No (a verification pod using Trident PVs) | EKS 1.35 / EC2 nodes | PVs Trident provisions (iSCSI, NFS) | Trident authenticates to the management LIF with a vsadmin credential and provisions the volumes | Container-Datastore |
| E-1 | No (an Athena SQL query) | Athena (managed) | S3 Access Points (S3 API) | AWS IAM authenticates; IAM and a file-system user authorize | Lakehouse |
| E-2 | No (a Bedrock Knowledge Bases search) | Bedrock Knowledge Bases and a web app | S3 Access Points as the ingestion source | Users authenticate with Cognito; a SID match inside the app decides per-document access | Agentic-RAG |
As a diagram: the top tier is the four candidates for the same job (A and B on Amazon EC2, C and C-REST on AWS Lambda), the bottom tier is the two paths for different workloads (D on Amazon Elastic Kubernetes Service, E on Amazon Athena and Amazon Bedrock), and both reach one Amazon FSx for NetApp ONTAP volume.
Figure 1: Six paths reaching one FSx for ONTAP volume. The top tier is the same job's candidates; the bottom tier is different workloads (dark theme version).
Verification environment and method
Configuration (A, B, C, C-REST)
- Region ap-northeast-1, Single-AZ. The ONTAP version is 9.18.1 (NetApp Release 9.18.1P6), read in the measurements for candidates A, B, C-REST, and path D. For C and E I carried the value over from the same file system and did not read it again in those measurements.
- I used the existing file system (an existing verification file system that I keep, separate from the one I measured in stages 0 to 3 of the previous article) as the host, and created a disposable SVM and volume per candidate. I did not touch the existing 5 SVMs and 31 volumes. After each verification I deleted the disposable resources and confirmed that the existing 5 SVMs and 31 volumes were still present.
- Candidates A and B mounted over NFS 4.1 from a Linux Amazon EC2 instance (Graviton arm64, AL2023). Candidate C reached FSx for ONTAP S3 Access Points from a Lambda outside the VPC, over the public S3 endpoint. Candidate C-REST reached the ONTAP management-LIF REST API from a Lambda inside the VPC.
Configuration (D, E-1, E-2)
I measured D and E in other repositories, on other days. In each case I added disposable resources to the same file system as A through C-REST, and deleted them after verification.
| Path | What was disposable | Confirmation that existing resources were untouched |
|---|---|---|
| D | A dedicated SVM (with its own vsadmin), an EKS cluster (Kubernetes v1.35), two m5.large nodes (Amazon Linux 2023), and the Trident add-on v26.6.2 | Before and after teardown, this file system's 5 backups, 5 SVMs, 31 volumes, and 72 ENIs in the VPC were identical |
| E-1 | One volume, one internet-origin S3 Access Point, a Glue database and table, and an Athena results bucket | After teardown, the existing file system, its 5 SVMs, and its 8 access points were unchanged |
| E-2 | One volume, one internet-origin S3 Access Point, a Bedrock Knowledge Base, an S3 Vectors index, and a CDK stack including the web app | After teardown, the existing file system, SVMs, access points, and volumes were unchanged |
I read the ONTAP version in the path D measurement (9.18.1P6) and did not read it in the E measurements. The SVM used for E-1 was joined to Active Directory, and the SVM used for E-2 was not.
Method
For each path I fixed beforehand, as the success criteria, both the cases that run as expected and the failure cases (that fail as expected on a deny, an unreachable target, or an invalid identifier), and recorded the real responses. The failure cases are there to confirm not only "it ran" but also "the failure returns in the expected shape." If you only watch the happy path, you miss a request that should be denied slipping through, or an unreachable target quietly returning empty. Because the shape of a failure swaps per path (a deny as an OS permission error vs. HTTP 403; an unreachable target as an exception vs. a timeout), I measured the failure cases per path as much as the normal ones. Completion was judged not by a build or a call succeeding, but by each normal and failure case returning the expected real response. The real responses for A through C-REST are in a private record. For D and E they are in the public records of each repository, linked in "Resources".
Each case was measured once, without repeating runs to confirm reproducibility. Where I got through by hand, and where I did not retrieve a value, I note it on the spot.
Candidate A (.NET / NFS): the previous article in summary
Candidate A is a .NET 10 BackgroundService that processes files on a volume mounted over NFS 4.1. It is the same lineage as stages 2 and 3 last time; I confirmed four cases.
Two findings, briefly. One is the systemd unit type: Type=notify stalls startup because the worker sends no sd_notify, and the correct type is Type=simple. The other is that the local promotion gate (NfsPromotionGate) validated source_revision with a 64-hex (SHA-256) regex, rejecting a 40-hex Git SHA-1 commit id. Details are in the previous article and #42.
Candidate B (Java / NFS): FileLock behavior is invisible without two processes
Candidate B runs a Java/Spring background worker on Linux, processing a volume mounted over NFS 4.1. I confirmed file processing, lock acquire/release, cross-process lock contention, same-JVM lock overlap, a platform I/O error, and wait/retry, using two hosts.
- File processing processed a document on the NFS mount and wrote a report with the expected length and first line.
- File lock acquire/release recorded
acquired_released, report-write and claim-release for each item, with a clean Spring start, item success, normal stop, and exit 0. - Cross-process lock contention ran two JVMs on the NFS mount. While the holder kept the lock, the contender got
tryLock=nulland was classifiedlock_contention. - Same-JVM lock overlap threw
OverlappingFileLockExceptionwhen opened twice within one JVM, classifiedsame_jvm_contention. - A platform I/O error was recorded as
platform_io_errorwith operation, platform, and exception. - Wait/retry: a bounded wait was measured on two hosts over NFS (below). Past the limit, a
nullis stilllock_contention, and nothing is retried silently.
Wait/retry was measured once each on two hosts (ONTAP 9.18.1P6, NFS 4.1, Amazon Linux 2023 arm64; the SVM's lease is 30 s, grace 60 s). The waiter took the claim 198 ms after the holder released it (250 ms retry interval); a 10 s limit ended as lock_contention after 41 attempts; with no wait it was lock_contention after one attempt. After kill -9 of the holder JVM the claim was taken 162 ms later; after a forced stop of the holder's host it was taken about 75 s after the stop request (a single observation that does not separate stop latency from lease expiry). During the measurement, a start-signal file placed on the NFS volume was not seen by the second host for about 30 s; the cause is unconfirmed, and the two-host start signal was switched to the process's standard output.
A guide to setting the wait limit in production. When the holder releases normally, the claim was taken in a few hundred milliseconds, so a short limit is enough. But if you wait for recovery from a holder host failure, the lock only clears after the lease expires, so the limit must be longer than the lease (30 s) plus grace (60 s), about 90 s, or you give up before recovery. Decide by whether you prioritize responsiveness or not dropping work on a failure.
One behavior stood out. A null java.nio.channels.FileLock (cross-process lock contention) is observable only with two processes. Within one JVM the platform throws OverlappingFileLockException (same_jvm_contention). The NFS mount enforced cross-process exclusion as expected.
There was also a build-environment trap: Maven's default JDK on AL2023 did not accept release 21, and JAVA_HOME had to point explicitly at Corretto 21, otherwise mvn failed with "release version 21 not supported."
Candidate C (Python Lambda / FSx for ONTAP S3 Access Points): IAM authorizes
Candidate C is a Python 3.14 Lambda that reads and writes volume objects through FSx for ONTAP S3 Access Points. On a disposable UNIX SVM and volume not joined to Active Directory, I created an internet-origin access point and reached it from a Lambda outside the VPC over the public S3 endpoint. I confirmed five cases; file locking was out of scope.
- PutObject wrote an object.
- GetObject returned a read-back hash equal to the write input hash (roundtrip hash match).
- ListObjectsV2 returned a listing.
- Policy-condition deny: with an explicit Deny variant, a GetObject through the access point returned
AccessDenied/ HTTP 403. A before/after control (writable before the Deny, denied after) isolated the policy effect. - Invalid parser input classified invalid bytes locally. It is not a managed-Lambda result.
- File locking was out of scope because this operation set does not express file locking.
The finding is that FSx for ONTAP S3 Access Point object operations (Get/Put/List) ran from a Lambda outside the VPC over an internet-origin access point on a UNIX SVM not joined to Active Directory, with no dependency on reachability to an AD domain controller. A deny returned through the access point as the exact AccessDenied / HTTP 403 class, and the before/after control isolated the policy effect.
Candidate C-REST (Python Lambda / ONTAP REST API): ONTAP authorizes, plus the self-signed certificate and multipart traps
Candidate C-REST is a Python 3.14 Lambda that reads files through the ONTAP REST API. It is the same Python Lambda as candidate C, but the access path is entirely different. Candidate C uses the public S3 endpoint and AWS IAM authorizes. C-REST has a Lambda inside the VPC reach the management-LIF REST API, authenticate with an ONTAP credential (basic auth), and let ONTAP authorize. I confirmed a read, recording the result, a write, a multi-call read, and four failure cases (ONTAP 9.18.1P6).
- File listing read: a GET on the files resource returned a listing, HTTP 200, Accept
application/json. - Recording the read result: the read result was recorded to a separate bucket, and its hash was independently confirmed to match the read's output hash.
- Write: 2.5 MB written in three calls. The ONTAP REST API creates a file with a multipart POST and then writes with PATCH at a byte offset (a JSON POST to create first returned HTTP 405). The read-back hash matched the write hash — but the read-back uses the same REST API, so it is not independent of the write path.
- Multi-call read: the size came from the parent directory listing, then four ranged reads reassembled the whole file and the hash matched.
- Single-call size limit: 1,048,576 bytes returned HTTP 200 and 1,048,577 returned HTTP 413 (no body), confirming the documented 1 MB per-request limit.
- Auth failure: a wrong credential returned HTTP 401, ONTAP error code 6691623, and no bytes were served.
- Authorization deny: with a read-only user, GET was allowed and a write returned HTTP 403, ONTAP error code 6, "not authorized for that command."
- Network unreachable: toward an unused address, a connection timeout was classified endpoint-unreachable, and no bytes were served.
- Invalid identifier: a valid volume identifier with a non-existent path returned HTTP 404. A malformed volume identifier instead returned HTTP 200 with an empty listing, so I measured the invalid identifier on the path.
There are four findings about the shape of the API when you call the ONTAP REST API from a client. A file is created with a multipart POST and then written with PATCH at a byte offset (a JSON POST returns HTTP 405). A read response returns two entries (bytes_read and the data), so only the data is hashed. A JSON listing of a file path returns HTTP 400 "Not a directory", so the size comes from the parent directory listing. Exceeding the single-call read limit returns HTTP 413, which maps to its own class. These are recorded in #73.
There is also a TLS finding, and it is not about S3 Access Points (Candidate C). When you reach FSx for ONTAP through S3 Access Points, an AWS-managed endpoint terminates TLS, so the client does not deal with a certificate (putting a certificate on a custom domain is AWS Certificate Manager territory, separate from any ONTAP-side certificate). The ONTAP REST API (Candidate C-REST), by contrast, calls the ONTAP management endpoint (the cluster-management LIF), where ONTAP itself — not an AWS-managed TLS endpoint — presents the certificate. ONTAP presents a self-signed certificate by default (ONTAP management protocols and the TLS handshake, generating a self-signed certificate for FSx for ONTAP), so by default a certificate-verifying call fails. This failure must be classified as a TLS-trust failure, not "unreachable." Python's urllib wraps the SSL error inside the URLError reason, so you unwrap the reason before classifying. The option to disable certificate validation is used only as an explicitly recorded diagnostic, never as a silent bypass. In production, install a certificate signed by a trusted CA on ONTAP and have the client trust that CA.
The Accept header needs care too. An ONTAP REST API listing does not accept multipart/form-data (HTTP 406, ONTAP error code 262269). A directory listing must use Accept application/json, and a byte read branches to multipart with a length.
There is also a finding on FSx for ONTAP role scope. At the file-system level the roles are fixed (fsxadmin, fsxadmin-readonly); custom roles are definable only at the SVM level. An SVM-scoped user gets HTTP 401 on the management LIF (there is no SVM REST management LIF). The read-only role passes all GETs, and a write returns HTTP 403 / ONTAP error code 6.
Path D (EKS and Trident): using FSx for ONTAP as a container's persistent volume
Path D is not the same job. I tested, in the Container-Datastore repository, a path in which containers on EKS use FSx for ONTAP volumes as persistent volumes through Trident. Unlike A through C-REST, it does not read the data of an existing volume: Trident provisions new volumes and LUNs for the containers.
What I confirmed is whether data written by a container remains after a pod restart and after a reschedule to another node. I created two StorageClasses. ontap-san is iSCSI, with RWO, which only one pod writes to. ontap-nas is NFS, with RWX, which several pods share. I provisioned one 1 GiB PVC from each.
| Type | Check | Result |
|---|---|---|
| iSCSI / RWO | PVC provisioning |
Bound in about 10 seconds |
| iSCSI / RWO | After writing a marker, after a pod restart (same node), and after cordoning the node and deleting the pod (other node, other AZ) | The marker matched all three times. The new pod started only after the old pod had finished deleting. Both nodes mounted the same multipath device (ext4) |
| NFS / RWX | PVC provisioning |
Bound in about 10 seconds |
| NFS / RWX | Pod B on another node reads what pod A wrote | Matched (NFS v4.2) |
| NFS / RWX | After restarting the Deployment so every pod was new | Matched. An append made after the restart was readable from another pod |
With the corrected template, no manual step was needed (the Trident node pods restarted 0 times and both nodes could use iSCSI) and the persistence results in the table held. I also checked importing an existing volume: I wrote a file to a volume created with the FSx API, then imported it with an annotated PVC. It bound in about 10 seconds and a pod read the pre-existing file. ONTAP renamed the volume to trident_pvc_…, keeping its UUID and junction path. With reclaim policy Retain, deleting the PVC and the PV left the ONTAP volume in place (though Trident kept its own record, so backend deletion waited until tridentctl delete volume removed it). As a limit, for RWX I confirmed shared visibility, not consistency when several pods write the same file at once.
I also measured backend failure shapes. A backend with a wrong credential turned Failed (HTTP 401) and one pointed at an unreachable management LIF turned Failed (no route to host); both left the PVC Pending with a "no available backends" event. Changing a working backend's credential to a wrong value kept it running on the old credential until it was reloaded: existing pods' reads and writes did not stop, only new PVC provisioning did. This failure test locked the disposable SVM's vsadmin, which also stopped another backend using the same user, so I unlocked it as the file system administrator. Failure tests are safer with a separate SVM user, or run after the other checks.
During verification I found three defects in the CloudFormation template I used. I recorded them in an issue (#26) and fixed them in #27.
- The default Kubernetes version was 1.31, which was in extended support in ap-northeast-1 on the day I tested. I removed the default so the user chooses a version in standard support.
- The extra security group the template created attached to the control-plane network interfaces, not to the managed nodes (both nodes, one observation). I changed the source of the FSx-side ingress rules to the cluster security group, which the nodes carry.
- The Amazon Linux 2023 EKS AMI did not include the iSCSI tools. I made a launch template install them, and with the changed template I confirmed that first-boot nodes had the tools and that Trident's node pods detected iSCSI without a restart (two nodes, one observation).
Teardown showed one more behavior. The FSx API (describe-volumes) reflects the volumes Trident creates on the ONTAP side with a delay. It did not list them at 23 seconds and at 11 minutes after creation, and listed them at 26 minutes. After I deleted the volumes on ONTAP, it still listed them at 41 minutes and no longer did at 44 minutes (one observation). While the FSx API showed a non-root volume, deleting the SVM was rejected. After teardown, check for leftovers with the ONTAP REST API as well as the FSx API. Deleting the PVCs removed the LUN and the volumes from ONTAP, and the file system's backup count was the same before and after (environments with automatic backups enabled are unchecked).
The results and the procedure are in the Container-Datastore record (section 4.3 of the storage-verification memo).
Path E (services that read through S3 Access Points): analytics and RAG
Path E is AWS services that read FSx for ONTAP data through the S3 API. It goes through the same S3 Access Points as candidate C, but the caller is not a Lambda I wrote; it is a service such as Athena or Bedrock Knowledge Bases. I tested analytics (E-1) in the Lakehouse repository and RAG (E-2) in the Agentic-RAG repository.
E-1 Analytics: Athena reads Parquet
I put a 10,000-row Parquet file (Snappy compression, about 150 KB, microsecond timestamps) on a disposable volume, and read a table registered in the Glue Data Catalog with Athena (engine v3) through an internet-origin access point (UNIX user).
| Check | Result |
|---|---|
| PutObject, ListObjectsV2, and HeadObject on the access point | All succeeded. The listing showed storage class FSX_ONTAP and size 150,902 bytes |
SELECT COUNT(*) |
10,000 rows, matching the source row count |
Conditional aggregation (WHERE, GROUP BY, ORDER BY) |
The counts (1,667, 1,667, 1,666) and totals for the three regions matched the values computed from the source data exactly |
| Run time (small data, cache-resident) |
COUNT took 821 ms in total and scanned 0 bytes. The aggregation took 942 ms in total and scanned 12,384 bytes |
The run times are dominated by planning on small data. They are not production-scale values.
Two premises apply. Athena needed an internet-origin access point; a VPC-origin one did not work. The SVM I used was joined to Active Directory. The test succeeded, but I did not measure behavior when the domain controller is unreachable. Query results are written to a separate S3 bucket, not to FSx. I did not retrieve the ONTAP version. Other engines (Glue, EMR Serverless, Redshift Spectrum, and so on) are in other records in the Lakehouse repository, and I did not re-measure them here.
The record is the Lakehouse evidence-record.yaml.
E-2 RAG: narrowing search results by permission
I took documents on a disposable volume (on an SVM not joined to Active Directory), ingested them from an internet-origin access point into a Bedrock Knowledge Base (S3 Vectors as the vector store), and checked on a deployed environment a path in which a filter inside the web app matches SIDs that permit each document and narrows the search results. I measured four cases.
| Case | Principal and search | Retrieved | Allowed | Denied | Method |
|---|---|---|---|---|---|
| 1 | An administrator (holding the SIDs of the administrator user, Domain Admins, and Everyone) searches "company sales" | 6 | 6 | 0 | SID match |
| 2 | A general user (the individual user's SID and Everyone only) runs the same query as case 1 | 6 | 0 | 6 | SID match |
| 3 | A user with no registered SID runs the same query as case 1 | 6 | 0 | 6 | Deny all (fail-closed) |
| 4 | The administrator searches "product overview" (over Everyone documents) | 8 | 8 | 0 | SID match |
| 4 | The general user runs the same query | 8 | 3 | 5 | SID match |
In cases 2 and 3, the same query as case 1 retrieved 6 documents, and all 6 were denied. That shows the filter removed the documents after retrieval, not that the user's forbidden documents were never found. For the user with no registered SID, all 6 retrieved documents were denied, so the path was fail-closed (deny everything when there is no permission information).
The four cases above were measured on the inline path (filter inside the web app), which is the production default. That path has no cache and reads DynamoDB each time, so adding the confidential group's SID to a general user made the same query ALLOW immediately, and removing it returned it to DENY. As an alternative, with the filter moved to a Lambda (usePermissionFilterLambda=true) I measured the same four cases plus group add/remove, confirmed filterLog.source reads lambda, and got the same verdicts. The Lambda path has a five-minute cache: after removing the group, a cached entry returned the old verdict until it was cleared.
A CDK deploy plus one setup script get ingestion (11 documents) and the four cases through with no manual step, except one: creating the Cognito users, whose password does not meet the pool policy. Code defects found during the deploy are recorded in #228 and fixed. The set was 11 documents, and each case is one observation. I did not measure an SVM joined to Active Directory.
The record is section 3.3 of the Agentic-RAG evaluation.md.
How failures return per path, and what was confirmed for D and E
For A, B, C, and C-REST, which ran the same job, authentication, network, authorization, and the way failures return changed as follows.
| Aspect | A (.NET / NFS) | B (Java / NFS) | C (Python / S3 Access Points) | C-REST (Python / ONTAP REST API) |
|---|---|---|---|---|
| Auth principal | OS (uid, AUTH_SYS) | OS (uid, AUTH_SYS) | AWS IAM | ONTAP credential (basic) |
| Authorization principal | OS (NFS permissions) | OS (NFS permissions) | AWS IAM + access-point policy | ONTAP |
| Network | NFS mount (in-VPC) | NFS mount (in-VPC) | public S3 endpoint (outside the VPC) | management LIF (in-VPC) |
| Auth failure returns | at the NFS mount stage | at the NFS mount stage | (credential is IAM-side) | HTTP 401 / ONTAP 6691623 |
| Authorization deny returns | NFS permission error | NFS permission error | AccessDenied / HTTP 403 | HTTP 403 / ONTAP code 6 |
| Unreachable returns | mount failure | mount failure | (public endpoint) | connection timeout → unreachable |
| Invalid identifier returns | file not found | file not found | (per key) | HTTP 404 / ONTAP 6684674 |
| TLS consideration | none (NFS) | none (NFS) | none (AWS terminates TLS; no client-side certificate) | ONTAP self-signed certificate → must classify as trust failure |
| Dependency on AD DC reachability | none (measured on an SVM not joined to AD) | none | none | none |
The same "deny" returns as an OS permission error over NFS, as AccessDenied / HTTP 403 over S3 Access Points, and as HTTP 403 with an ONTAP error code over ONTAP REST API. When you implement failure classification, you have to fold in this per-path difference.
D and E are different workloads, and I did not measure their failure cases in the same form. I separate what I confirmed from what I did not into another table.
| Path | What I confirmed | What I did not confirm |
|---|---|---|
| D (Trident PVs) | iSCSI and NFS volumes were provisioned in about 10 seconds, and the data remained after a pod restart and a reschedule to another node. Deleting the PVCs removed the LUN and the volumes. An existing volume was imported as a PV and its pre-existing data was readable. A backend with a wrong credential or an unreachable management LIF returned Failed, leaving the PVC Pending
|
Multi-AZ switchover. Performance. Same-file concurrent-write consistency |
| E-1 (Athena) | Reading and aggregating Parquet through S3 Access Points matched the source data. It succeeded on an SVM joined to AD | Behavior when the domain controller is unreachable. Production-scale data. Write-back |
| E-2 (Bedrock Knowledge Bases) | Documents outside the user's permissions were removed from the search results, and a user with no permission information was denied everything. A deny is not an error; it returns as removal from the results. Group add/remove was reflected in the results. The inline and Lambda paths gave the same verdicts. CDK and one script nearly completed (only a password step by hand) | An SVM joined to Active Directory. Response time. Natural expiry of the Lambda cache |
How to choose a path
Not by winner, but by the client you use and where you want authorization to happen. I state trade-offs symmetrically, including for the option I lean toward.
| Choice | When | Trade-off / consideration |
|---|---|---|
| NFS mount (A, B) | When a Linux server process needs POSIX file semantics (such as exclusive locking) directly | Auth is uid-based AUTH_SYS and depends on OS permissions. It assumes an in-VPC mount and is hard to use directly from serverless. File-lock behavior is only observable with two processes. |
| Your own code calling S3 Access Points (C) | When an S3 API client such as Lambda should be authorized by AWS IAM | Authorization is IAM plus the access-point policy; it holds no ONTAP credential. It suits jobs that fit the S3 API operation set (it does not express POSIX semantics such as exclusive locking). The network origin is fixed at creation. It uses the public S3 endpoint, so an in-VPC Lambda is not required. |
| ONTAP REST API (C-REST) | When you want fine control with an ONTAP credential and ONTAP-side authorization for management or file operations | The management LIF is reachable only from inside the VPC, so you need an in-VPC Lambda (mind cold starts and the ENI). You must handle the self-signed certificate (provide a trusted CA in production; disable verification only as an explicitly recorded diagnostic). File-system-level roles are fixed; custom roles are SVM-level only. Large files hit the 1 MB-per-call limit, so split them across calls. |
| Trident persistent volumes (D) | When stateful containers on EKS should use FSx for ONTAP volumes as PVs; also when you want to import existing NFS/SMB data as a PV | You need EC2 worker nodes (it does not run on Fargate), and iSCSI needs the tools and a multipath configuration on the nodes. Node preparation has no guaranteed start order, so restart the DaemonSet if the order breaks. The source of the FSx SG ingress is the cluster SG, not the node SG. Trident's objects are applied with kubectl, outside CloudFormation. The EKS cluster is billed. I confirmed both provisioning new volumes and importing an existing one. |
| An AWS service that takes S3 as input (E) | When a service such as Athena or Bedrock Knowledge Bases should use data on FSx for ONTAP | You are bound to the access-point settings (Athena needs an internet-origin one). Choose internet-origin when the service comes from outside the VPC, VPC-origin when it stays inside. Per-document permissions must be designed outside the service (for example, a filter in the app). It does not express all of S3 (for example, conditional writes are unavailable). |
All of them are paths to the same FSx for ONTAP data. The basis for choosing is what the client is (a Linux process, an S3 API client, something holding an ONTAP credential, a Kubernetes container, or an AWS managed service) and where you want authorization to happen (the OS, IAM, ONTAP, or the app).
Reaching one volume through several paths at once (NFS, SMB, S3 Access Points) shares that volume's throughput. When you design for several paths at the same time, size the combined load against the volume's throughput.
This table does not decide which path is better. It lays out a fact: authentication, authorization, and the shape of failures swap per path. Which path fits your system, and whether you can keep operating it, is for someone who knows the app and the data to judge from the facts laid out here.
Summary and the path options
The result is the three findings from the start.
- The same file-processing job ran as four implementations (.NET, Java, and Python) against disposable volumes on the same FSx for ONTAP file system, without moving the data. Changing the path swapped the authentication principal, the network boundary, the authorization principal, and how failures return.
- Containers on EKS read the data in volumes Trident provisioned after a pod restart and after a reschedule to another node (SAN and NAS, once each). Athena read Parquet through S3 Access Points and the aggregation matched the source data (10,000 rows). In Bedrock Knowledge Bases searches, documents the user was not permitted to see were removed from the results (four cases, 11 documents).
- The basis for choosing is the client you use and where you want authorization to happen.
All costs are estimates before billing, for disposable environments that lived minutes to a few hours, not production always-on costs. Each A through C-REST measurement was under $1 in an approved disposable environment. Path D's stack lived about 36 to 70 minutes; from the unit prices (ap-northeast-1, Pricing API, 2026-10) of an EKS cluster at $0.10/hour plus two nodes at $0.124/hour each, that is about $0.2–0.4, excluding NAT data processing. Path E-2 is a WAF Web ACL ($5/month, $1/month per rule, prorated) plus small usage-based charges for CloudFront, Lambda, and Bedrock, on the order of under $3 for a disposable few hours. For E-1 and E-2 I did not compute a cost for the actual time the resources lived, and I did not check the billed amounts.
Each case was measured once, and I said on the spot where I got through by hand and where I did not retrieve a value. Path E-2 is nearly completed by CDK and one script (only the password step is by hand), and the SAN and NAS data persistence on path D was observed on the corrected template with no manual step.
Repository:
https://github.com/Yoshiki0705/FSx-for-ONTAP-App-Modernization-Patterns
The previous article is "Keeping the Data Where It Is, Modernizing Only the App From .NET Framework to Lambda — What Changes and What Does Not (A 1-Environment Test)."
To try this in your own environment, start with the premises in reproduce.md.
Modernization does not have to replace everything at once. With the data in one place, you can add access paths a little at a time — first an EC2 instance over NFS, then Lambda, and later EKS or analytics and AI. You choose each path to fit the runtimes your team can operate and where you want authorization to happen. Choosing FSx for ONTAP as the destination lets you take this "keep the data in place, add access paths in stages" approach in combination with the various AWS services. This article measured those six paths actually reaching the same data, and how authentication, authorization, and the shape of failures change per path.
Whether you are about to modernize on AWS or already use FSx for ONTAP, I hope this gives you something to decide with as you go step by step.
See you next time.
Bye now!!
Yoshiki Fujiwara
Resources
- Candidate A runtime verification issue (#42)
- Candidate B spec / runtime issue (#46)
- Candidate C spec / runtime issue (#47)
- Candidate C-REST spec / runtime issue (#48)
- Runtime verification tracking (#38). A, B, C, and C-REST are complete; their deliverables are in the issues and records above. #38 now tracks the remaining deep-dives such as performance and scale. The letters A to C in #38 differ from this article's B, C, and C-REST; the mapping is in #45
- Path D record (Container-Datastore, section 4.3 of the storage-verification memo), and the issue for the template defects (#26)
- Path E-1 record (Lakehouse, evidence-record.yaml)
- Path E-2 record (Agentic-RAG, section 3.3 of evaluation.md)
- FSx for ONTAP S3 Access Points user guide
- Amazon EKS User Guide: use FSx for NetApp ONTAP
- Microsoft Learn: Upgrade .NET apps overview
- ONTAP REST API reference
Top comments (0)