S3 Burst on ONTAP Files — Collect via S3 API, Consume via FlexCache NFS/SMB

Introduction

You have an S3 ingestion pipeline. But the consumers only see an NFS mount. Sound familiar?

You want to deliver driving logs to a HiL test bench, stage design jobs for an EDA toolchain, push rendering assets to production nodes, or feed genomic sequencer output to an HPC cluster. In each case the writer speaks S3 API in the cloud while the reader speaks NFS/SMB on physical hardware. Without a way to connect these directly, you end up adding a sync job in between — and with it comes latency, cost, and one more thing to monitor.

This structure spans industries: automotive, semiconductor, media/VFX, oil and gas, life sciences, manufacturing, remote work, and IoT (see the full table in When This Pattern Fits below).

This post introduces a pattern that removes that sync job. Write to an Amazon FSx for NetApp ONTAP (hereafter FSx for ONTAP) S3 Access Point, and the data appears on a FlexCache NFS mount. No copy job, no configuration between the two. I measured all four directions to see how fast the visibility propagates.

Here's the conclusion up front:

• S3 PutObject → FlexCache NFS read visibility: p50 8 ms. No copy job, no configuration
• FlexCache overhead is +5 ms (vs. reading origin directly) — nearly transparent for same-region VPC peering
• Reverse direction (NFS write → S3 AP read) is p50 44 ms, dominated by S3 API overhead. Not the main path in this design
• FlexCache duality (NAS bucket S3 reads on FlexCache volumes) returns AccessDenied on FSx for ONTAP 9.18.1P3D1. Works on regular volumes. FlexCache-specific constraint

Repository: Yoshiki0705/s3-burst-on-ontap-files

The Architecture

[S3 Client] --PutObject--> [S3 Access Point] --> [Origin Volume]
                                                       |
                                                  FlexCache
                                                       |
                                                [NFS/SMB Client]

Layer	What	Protocol
Collect (write)	S3 Access Point on FSx for ONTAP	S3 API
Source of truth	Origin volume	—
Distribution	FlexCache	Cluster/SVM peering
Consume (read)	Cache volumes	NFS/SMB only

The S3 Access Point attaches to the origin only. Cache volumes serve NFS/SMB. This single decision simplifies the design considerably:

• One write path. The origin is authoritative; writes always go through the S3 AP
• No S3 implementation differences pushed to the edge
• Cache side requires only FlexCache + NFS/SMB — fewer platform constraints to hit

Test Environment

Item	Value
Date	2026-08-09
ONTAP version	9.18.1P3D1 (both clusters)
Configuration	SINGLE_AZ_1, 128 MBps × 2 clusters
Connectivity	VPC peering (same region, same account)
Mount	NFSv3, actimeo=0 (client cache disabled)
Object size	64 bytes
Concurrency	1
Method	boto3 persistent session, same host, 30 iterations

Why NFS First

The initial measurements used NFS. Three reasons:

1. Client cache control. NFS actimeo=0 fully disables kernel attribute caching, isolating storage-side visibility latency. SMB oplocks/leases let the client cache independently, making it harder to measure the same thing at the same granularity.
2. S3 Access Point requirement. The FSx for ONTAP S3 AP attaches to UNIX security-style volumes. SMB access on a UNIX volume is possible (via name-mapping + permission mapping), but NFS is the simplest consumption path for this architecture.
3. Primary use-case distribution. HiL test benches, rendering farms, and IoT analysis appliances are predominantly Linux + NFS. When the read side is SMB-dominant (Windows workstations in a production studio), the origin security style changes to NTFS — a different design path (see First Decisions).

SMB Shows Identical Results

I ran a separate verification with SMB (mount -t cifs, cache=none) on the same architecture:

Protocol	Mount method	p50	p90	max	n
SMB	mount -t cifs, cache=none	7 ms	8 ms	9 ms	30
NFS	mount -t nfs, actimeo=0	7 ms	8 ms	15 ms	30

With persistent mounts, SMB and NFS have identical visibility latency. No protocol difference.

Note: smbclient (establishing a new session per request) showed p50 43 ms. This is SMB session setup overhead — the same structural problem as the AWS CLI cold-start that inflated our first NFS→S3 measurement to 873 ms. Production environments with persistent connections won't see this.

SMB Considerations

FlexCache serves both NFS and SMB. This architecture doesn't exclude SMB — but note:

• With a UNIX security-style origin, SMB clients receive UNIX permission-based access control, not NTFS ACLs
• For SMB-primary workloads, NTFS security style is more natural, but S3 AP attaches to UNIX-style volumes. This is a genuine tradeoff for "collect via S3, consume via SMB" scenarios
• mixed security style is available in the API but officially not recommended by AWS — it's labeled "advanced users only." Permission type is determined by the last client that wrote, making troubleshooting difficult. This architecture doesn't use it
• SMB requires Active Directory (FSx for ONTAP does not support workgroup mode)

Recommendation for this architecture: Origin uses UNIX security style (S3 AP requirement). SMB readers use name-mapping and accept UNIX permission-based access. If your workload requires NTFS ACL granularity for Windows clients, this architecture isn't the best fit.

actimeo=0 measures the minimum visibility latency by disabling client-side caching. Production deployments should use appropriate actimeo values for their workload — the default (~60 s) means subsequent reads hit kernel cache at ~0.05 ms, but changes aren't visible during that window.

Results — All Four Directions

#	Direction	p50	p90	p99	max
1	S3 AP PutObject → FlexCache NFS read	8 ms	9 ms	19 ms	19 ms
2	S3 AP PutObject → Origin NFS read (direct)	3 ms	5 ms	8 ms	8 ms
3	NFS write (Origin) → FlexCache NFS read	6 ms	7 ms	25 ms	25 ms
4	NFS write (Origin) → S3 AP GetObject	44 ms	49 ms	328 ms	328 ms

What the Numbers Say

Direction 1 is the main path. S3 write, FlexCache NFS read: p50 8 ms.

The +5 ms gap between directions 1 and 2 is FlexCache overhead. For same-region VPC peering, FlexCache is nearly transparent.

Direction 3 is faster than 1 (6 ms < 8 ms). NFS writes commit directly to the origin — no S3 API overhead — and FlexCache propagation alone is faster than the full S3-to-cache path.

Direction 4 (reverse) is slowest at 44 ms. S3 read-side processing dominates. This design keeps reads on NFS/SMB, so direction 4 isn't the main path.

NFS Client Cache Effect

Condition	p50
actimeo=0 (cache disabled)	7 ms
actimeo=60 (subsequent access)	0.05 ms (kernel cache hit)

With defaults, reads within 60 seconds of the last attribute check hit the kernel cache. The tradeoff: you won't see changes during that window. Tune per workload.

FlexCache Duality — It Works (With One Extra Step)

ONTAP 9.14.1 introduced NAS buckets on FlexCache volumes (FlexCache duality) — S3 reads from the cache side. I tested this on FSx for ONTAP 9.18.1P3D1.

Result: Works After Enabling -is-s3-enabled true

My initial test returned AccessDenied and I concluded it didn't work. After additional investigation, I found missing configuration step: S3 access must be explicitly enabled on the FlexCache volume:

set -privilege advanced
flexcache config modify -vserver <svm> -volume <fcache_vol> -is-s3-enabled true

After applying this setting:

Operation	Before (missing setting)	After (-is-s3-enabled true)
HeadBucket	✅	✅
ListObjectsV2	❌ AccessDenied	✅ KeyCount=1
GetObject	❌ AccessDenied	✅ Content verified

The fsxadmin role on FSx for ONTAP has access to advanced privilege commands including flexcache config modify. Documented at NetApp: Enable FlexCache duality.

Design Implication

FlexCache duality works, which means S3 reads from the cache side are possible. However, this architecture still recommends NFS/SMB on the cache side because:

• ONTAP native S3 (NAS buckets) and AWS-managed S3 Access Points are different mechanisms
• NAS buckets are read-only (no PutObject)
• Requires additional configuration (advanced privilege + S3 user management)
• No IAM integration or access point policy governance like S3 AP provides

FlexCache duality becomes an option when "S3 reads at the cache site" is a hard requirement. For AWS service integration (Lambda, Bedrock, etc.), the origin-side S3 AP remains the better fit.

When This Pattern Fits

"Collect via S3 API in the cloud, consume via NFS/SMB at the edge" — this structure exists across industries.

Industry	Collect side	Consume side	Example
Automotive (AV/ADAS)	Driving logs and sensor data ingested to S3	HiL test benches replay via NFS	AWS + NetApp: Hybrid Cloud HiL
Semiconductor (EDA)	Design job I/O staged via S3	Toolchains (Synopsys, Cadence) run on NFS	EDA Scale with FSx for ONTAP
Media and VFX	Rendering assets collected to S3	Artist workstations mount SMB/NFS	FlexCache: distributed product development
Oil and Gas	Seismic survey data uploaded to S3	Interpretation workstations mount NFS	VDI for Subsurface O&G
Life Sciences	Genome sequencer output stored in S3	Bioinformatics HPC processes via NFS	Sequencer → S3 → NFS pipeline
Manufacturing / QA	Inspection camera images collected to S3	Line-side inspection software reads via NFS	Image → judgment → archive flow
Remote Work	Central design data updated via S3	Remote-site WorkSpaces access FlexCache NFS/SMB	FlexCache in AWS WorkSpaces
IoT / Edge	Sensor data streamed to S3	On-site analysis appliances read via NFS	Factory gateway → cloud → shopfloor

Common structure: Few write sites (often one), multiple read sites. Writes are bursty; reads touch only what's needed.

When It Doesn't Fit

• Need full S3 semantics (versioning, event notifications, lifecycle): Use S3 natively
• Object names aren't NAS-friendly (flat namespace, millions of keys with no directory separators): Performance degrades as root directory grows
• Consumers need S3 reads at the cache site: Not supported on FlexCache today
• Need write-back from the cache: This pattern keeps cache read-centric
• Need conditional writes (If-None-Match): Returns 501 NotImplemented. Handle exclusion at the application layer

S3 Access Point Design Notes

A few things to know when using the S3 AP as the collect layer.

S3 AP ≠ Amazon S3

The FSx for ONTAP S3 AP supports a subset of S3 operations. GetObject, PutObject, ListObjectsV2, HeadObject, DeleteObject, and MultipartUpload work. The following do not:

• S3 Event Notifications (use FPolicy + EventBridge instead)
• Lifecycle rules (use FabricPool)
• Versioning (use ONTAP Snapshots)
• Conditional writes If-None-Match (returns 501)
• S3 Select, SSE-S3/KMS, Cross-AP Copy

Throughput is shared with NFS/SMB

S3 AP, NFS, and SMB all consume the same FSx for ONTAP provisioned throughput. In this architecture, origin and cache are separate clusters so this rarely matters — but if NFS clients also access the origin directly, account for the shared bandwidth. Rule of thumb: 128 MBps supports 2–5 concurrent S3 requests, 512 MBps supports 10–20.

Directory design matters

S3 PutObject keys map directly to directory structure. Beyond 100K files per directory, ListObjectsV2 slows down. Partition by date, tenant, or hash prefix.

For full details, see the S3 AP Design Guide in the repository.

Deploy

CloudFormation and Terraform templates:

• GitHub: Yoshiki0705/s3-burst-on-ontap-files

Conclusion

"Collect via S3, consume via NFS" — one volume, no copy jobs. FSx for ONTAP S3 Access Point + FlexCache. Main path p50 8 ms. FlexCache adds ~5 ms for same-region — nearly transparent.

This verification was done entirely on AWS (FSx for ONTAP to FSx for ONTAP over VPC peering), but the cache side isn't limited to AWS. On-premises NetApp AFF/FAS, ONTAP Select, Cloud Volumes ONTAP, Azure NetApp Files, Google Cloud NetApp Volumes cache volumes— any ONTAP-based environment can potentially serve as a FlexCache target for this same architecture. The repository's Portability and Support Matrix pages track what's confirmed and what's next. Cross-platform verification is on the roadmap.

FlexCache duality (S3 reads on the cache side) is an ONTAP 9.14.1 feature, but on FSx for ONTAP 9.18.1P3D1 data operations return AccessDenied on FlexCache volumes. Works on regular volumes — FlexCache-specific. The architecture works around this by keeping cache-side access NFS/SMB-only. Feature requests are filed.

---

All test resources torn down. Numbers are from a specific test environment and vary by workload and configuration.