Introduction
In July 2026, AWS released KNFSD File Cache as Preview — an open-source (Apache 2.0) NFS caching proxy built on the Linux kernel's NFS re-export and FS-Cache subsystems.
This is the first time an NFS read-acceleration layer has shipped as an official AWS solution. Working daily with FSx for ONTAP NFS performance for EDA and VFX workloads, my reaction was immediate: spin up a verification environment and see what happens.
What I tested was a Dual-Path architecture: KNFSD caching NFS reads from the same FSx for ONTAP volume that S3 Access Points expose to serverless Lambda processing. The question: can compute fleets read at local-NVMe speed while Lambda runs AI/ML workloads on the same data, without bandwidth contention?
Here's what I found:
- Dual-Path E2E test passed: A file written via S3 AP was immediately readable through the KNFSD NFS cache
- Cache effect: first read 55ms → subsequent reads 2ms (28x improvement), proxy cache serves 422-619 MB/s
- NFSv4.1 is mandatory: NFSv3 re-export causes Stale file handle on file creation (kernel-level filehandle size constraint)
- FSID_MODE=local + SQLite on FSx for ONTAP: FSID persistence at zero additional cost
This post covers every pitfall I hit during deployment, and why this architecture still makes sense despite those issues.
When to Consider This — and When Not To
This architecture suits workloads that:
- Need NFS read bandwidth beyond FSx for ONTAP's provisioned throughput (EDA DRC/LVS, VFX rendering, genomics)
- Want both NFS (high-speed reads) and S3 API (Lambda AI/ML post-processing) on the same data
- Use Spot instances for burst compute and need the cache to stay warm through reclamation
- Unify multiple NFS sources (FSx for ONTAP + on-premises NAS + other clouds) into one cache layer
Consider alternatives when:
- FSx for ONTAP throughput scaling alone covers your bandwidth needs (steady-state workloads)
- Write caching is needed (FlexCache offers configurable write-back/write-around)
- You want fully managed operations with no EC2 to run (FlexCache or Amazon File Cache)
- Production SLA is non-negotiable (KNFSD is Preview — wait for GA)
On vendor neutrality: This post evaluates KNFSD File Cache on technical merits only. FlexCache, Amazon File Cache, EFS, and other options each suit different contexts. The goal here is to document what actually works, not to rank solutions.
What is KNFSD File Cache?
KNFSD File Cache is an open-source (Apache 2.0) NFS caching proxy built on the Linux kernel's NFS re-export and FS-Cache subsystems. NFS re-export means mounting a remote NFS share and publishing it as a new NFS export to downstream clients — the proxy sits transparently between source and consumers. Wētā FX (Avatar: The Way of Water) and ILM used the predecessor project in production.
NFS Source (FSx for ONTAP)
│ NFS mount (NFSv4.1, fsc)
▼
KNFSD Proxy (EC2, NVMe L2 cache)
│ NFS re-export (NFSv4.1)
▼
Compute Fleet (Spot × hundreds)
- L1 cache: Linux Page Cache (RAM — 5-9 GB/s on 16 GB in this test)
- L2 cache: FS-Cache (local NVMe — survives reboot, 106 MB/s measured)
- Writes: write-through (immediately forwarded to source)
- Scaling: ASG + DNS round-robin or NLB
- Prerequisite: source must be an NFS v3/4.x server (parallel filesystems like Lustre/GPFS/WekaFS are not supported)
The key difference from FlexCache: KNFSD is a protocol-level transparent proxy that doesn't require the source to be ONTAP. It can unify FSx for OpenZFS, on-premises NetApp/Pure/Dell/DDN (A3I), and other-cloud NFS servers (via Direct Connect / Cloud Interconnect) into a single cache layer.
Test Environment
| Component | Details |
|---|---|
| Region | ap-northeast-1 |
| Source | FSx for ONTAP (128 MBps, Single-AZ, ONTAP 9.17.1) |
| KNFSD Proxy | m6gd.xlarge (arm64 Graviton, 16 GB RAM, 237 GB NVMe) |
| Kernel | 7.1.3-knfsd (Packer custom build) |
| FSID Mode | local (SQLite on FSx for ONTAP NFS mount) |
| NFS Version | NFSv4.1 (source → proxy → client) |
| Test Client | t4g.micro (AL2023, 916 MB RAM) |
| S3 AP | Internet-origin, UNIX identity (root) |
Cost note: KNFSD-specific incremental resources only (Packer Spot build ~$0.30 + m6gd.xlarge 1hr ~$0.29). Prerequisites: a running FSx for ONTAP (~$194/mo minimum) + VPC infrastructure. If you already operate FSx for ONTAP, the incremental evaluation cost is effectively < $1.
Dual-Path E2E Test Results (Verified)
Architecture
[Lambda / CLI] [Compute Fleet]
│ │
│ S3 PutObject │ NFS read (NFSv4.1)
▼ ▼
┌──────────────────┐ ┌──────────────────────┐
│ S3 Access Point │ │ KNFSD File Cache │
│ (Internet-origin)│ │ (m6gd.xlarge, NVMe) │
└────────┬─────────┘ └──────────┬───────────┘
│ │
│ ← same volume → │ NFS mount (NFSv4.1)
▼ ▼
┌──────────────────────────────────────────────────┐
│ FSx for ONTAP Volume (/vol1, UNIX security style)│
└──────────────────────────────────────────────────┘
Results
| Test | Result | Details |
|---|---|---|
| S3 AP write → KNFSD NFS read | ✅ | Immediate reflection, content match |
| NFS write → S3 AP read (MD5) | ✅ |
c092ef65... match both directions |
| S3 AP batch 50 files → NFS bulk read | ✅ | 50 files in 107ms (2.1ms/file) |
| S3 AP multipart 50MB → NFS read | ✅ | MD5 verified |
| NFSv4.1 write via KNFSD | ✅ | Write-through reflects immediately |
| fsidd + SQLite persistence | ✅ | SQLite created on FSx for ONTAP |
| Proxy restart → client continues | ✅ | NFSv4.1 grace period, no remount needed |
# Actual test log
$ aws s3 cp test.txt s3://arn:aws:s3:ap-northeast-1:XXXX:accesspoint/knfsd-dualpath-test/s3ap-e2e-1784803253.txt
# → upload: success
$ cat /mnt/knfsd/vol1/s3ap-e2e-1784803253.txt # NFSv4.1 via KNFSD
dual-path-e2e-test-1784803253
# → read success ✅
Performance Numbers
Test environment note: These numbers are from m6gd.xlarge (237 GB NVMe, single drive). Production-recommended instances (im4gn.16xlarge: 30 TB NVMe RAID, i3en.24xlarge: 60 TB) are expected to deliver significantly higher L2 NVMe throughput (estimated 2-8 GB/s sequential).
3-Layer Cache Hierarchy
| Layer | Throughput | Use Case |
|---|---|---|
| L1: Proxy RAM (page cache) | 5.0-9.1 GB/s | Hot data, repeated reads |
| L2: NVMe (FS-Cache) | 106 MB/s | Warm-up shortcut after proxy restart |
| Source: FSx for ONTAP (NFS fetch) | 18-19 MB/s | Cache miss, initial fetch |
Important note on source performance: The 18-19 MB/s figure reflects test-time NFS mount constraints (NFSv3 single-stream, rsize=64K) — not FSx for ONTAP's throughput limit (128 MBps). Switching the source mount to NFSv4.1 + rsize=1M is expected to significantly improve cache miss performance (proposed upstream in Issue #42).
L2 NVMe is 5.6x faster than source fetch. After a proxy reboot with empty RAM, data is served from NVMe cache immediately.
Throughput
| Operation | Throughput |
|---|---|
| Sequential read (proxy L1 → client) | 422-619 MB/s |
| Large write (write-through) | 157-218 MB/s |
| S3 AP multipart upload (50 MB) | 36.6 MiB/s |
Latency
| Operation | Cold | Cached | Improvement |
|---|---|---|---|
| 10 MB file read | 55 ms | 2 ms | 28x |
| 4 KB small file | 1.75 ms | 1.5 ms | 14% |
| S3 AP batch → NFS | — | 2.1 ms/file | — |
Working Set > Client RAM
With client RAM at 916 MB and a 500 MB dataset, reading after drop_caches still achieves 422-428 MB/s — served from the KNFSD proxy's 16 GB RAM cache. This is the core value proposition: small compute nodes can access datasets larger than their local memory at proxy-cache speed.
nconnect Caveat
| nconnect | 100 MB cold read |
|---|---|
| 1 (default) | 619 MB/s |
| 16 | 184 MB/s |
On t4g.micro (5 Gbps NIC), nconnect=16 is counterproductive. Only effective on 100 Gbps instances (c5n.18xlarge, hpc7g).
Cache Coherency
S3 AP write visibility through KNFSD depends on NFS attribute cache timeouts:
| Parameter | Default | Effect |
|---|---|---|
acregmin |
3s | Minimum file attribute cache |
acdirmax |
60s | Maximum directory attribute cache |
In our tests, S3 AP-written files were immediately visible through KNFSD (new files bypass directory attribute cache). For frequently-overwritten files, set acregmax=5-10 to reduce stale-data windows.
Every Pitfall We Hit (Documenting All of Them)
NFSv3 Breaks Writes (Most Important Finding)
This consumed the most debugging time. When mounting the KNFSD proxy via NFSv3, existing file reads succeed but new file creation returns Stale file handle.
The cause is documented in the Linux Kernel NFS re-export documentation:
reexport server's filehandle for the reexported object will be X+22 bytes
FSx for ONTAP filehandles are ~40+ bytes. After +22 bytes = ~62+ bytes. New file creation (which adds subdirectory inode information) pushes past NFSv3's 64-byte limit.
NFSv4.1 supports 128 bytes — ample headroom. Our Terraform validates against nfs_version="3" with a clear error message linking to the upstream issue.
Filed upstream as Issue #40.
Other Pitfalls
| Problem | Cause | Fix |
|---|---|---|
| proxy-startup.sh immediate failure | No EC2 API access (no Public IP) | assign_public_ip = true |
SSM GetParametersByPath denied |
IAM missing path-level ARN | Include both .../* and ... (Issue #41) |
| "No storage devices found" | CACHEFILESD_DISK_TYPE=local_nvme |
Correct value: local-nvme (hyphen) |
FSID_MODE=static breaks all writes |
fsidd disabled, filehandle unresolvable | Switch to FSID_MODE=local
|
| S3 AP create BadRequest |
--type One_Zone is for OpenZFS |
Use --type ONTAP
|
exportfs: No fsid found |
fsidd first-time allocation warning | No operational impact (ignore) |
FSID_MODE=local: SQLite on FSx for ONTAP
KNFSD's NFS re-export requires stable FSID management. FSID_MODE=static (fsidd disabled) is fundamentally broken — all write operations fail with Stale file handle.
The fix: use the Linux kernel's fsidd daemon with SQLite backend, placing the database file on the FSx for ONTAP NFS mount:
/srv/nfs/vol1/.knfsd/fsids.sqlite ← on FSx for ONTAP (persistent, 99.99% SLA)
| Aspect | local (SQLite on FSx) | external (RDS) |
|---|---|---|
| Additional cost | $0 | $15-45/mo |
| Persistence | ✅ (FSx for ONTAP SLA) | ✅ (RDS SLA) |
| Multi-node | △ (single-node only) | ✅ |
| Complexity | Low | Medium-High |
| Best for | PoC / single-node | Production multi-node |
FlexCache and KNFSD: Different Tools for Different Contexts
| Criterion | FlexCache | KNFSD |
|---|---|---|
| Source must be ONTAP | ✅ Native integration | ○ Any NFS-compatible filer |
| Multiple source unification | Not possible | ✅ |
| Write-back/write-around | ✅ (configurable) | △ (write-through only) |
| Burst read scaling | △ (within FSx throughput) | ✅ (NVMe cache scales with ASG) |
| Managed operations | ✅ | △ (EC2 operations) |
| Protocol | NFSv3/4.x | NFSv4.1 (required for re-export) |
| Data protection integration | ✅ (SnapMirror/SnapLock) | None (pass-through) |
| Node elasticity | △ (manual) | ✅ (Auto Scaling) |
FlexCache note: FlexCache also doesn't consume origin bandwidth on cache hits (same structure as KNFSD). However, in FSx for ONTAP's managed environment, auto-scaling FlexCache node count is not possible. For burst read bandwidth exceeding FSx max throughput, KNFSD offers more flexibility.
Amazon File Cache / FSx for Lustre: Amazon File Cache is a managed Lustre-compatible cache requiring the Lustre kernel client module on every node. KNFSD uses standard NFS clients (built into Linux/macOS), requiring no additional modules. However, Lustre's parallel I/O bandwidth is inherently higher than NFS re-export — if raw bandwidth is the priority and you can deploy Lustre clients on all nodes, File Cache / FSx for Lustre is the better fit.
Cost break-even guideline: KNFSD becomes economical when you need burst bandwidth beyond FSx for ONTAP's maximum throughput tier (4,096 MBps at ~$2,000/mo). For steady workloads where FSx throughput scaling suffices, increasing FSx throughput is simpler.
Target Workloads
| Industry | Scenario | KNFSD Role | S3 AP Role |
|---|---|---|---|
| Semiconductor EDA | DRC/LVS burst | Tech file repeated reads | Verification result AI classification |
| VFX | Rendering | Texture/asset Fanout | Render quality AI check |
| Genomics | Reference genome + dbSNP | Hundreds of nodes on same reference | Variant AI classification |
| Financial Risk | Monte Carlo VaR | Market data sub-ms delivery | Anomaly scenario AI detection |
| Automotive CAE | Structural sim | Shared mesh data reads | Result comparison AI |
| Weather | NWP ensemble | GFS initial condition cache | Extreme weather AI detection |
Step-by-Step Deployment
1. Build AMI (~25 min)
# Find your subnet (same AZ as FSx for ONTAP recommended)
aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-YOUR-VPC" \
--query 'Subnets[].{Id:SubnetId,AZ:AvailabilityZone}' --output table
git clone https://github.com/awslabs/knfsd-file-cache.git /tmp/knfsd-file-cache
cd /tmp/knfsd-file-cache/image
packer build -var 'REGION=ap-northeast-1' -var 'ARCH=["arm64"]' \
-var 'ASSOCIATE_PUBLIC_IP_ADDRESS=true' -var 'SUBNET=subnet-YOUR-SUBNET' .
# → Note the AMI ID (ami-0xxxxxxxxxx)
The AMI is reusable across deploy/destroy cycles. Rebuilds are only needed for kernel updates (~monthly). Normal deploys just pass the existing AMI ID to Terraform.
2. Terraform Deploy (~3 min)
cd infrastructure/knfsd-file-cache/terraform
cp terraform.tfvars.example terraform.tfvars
# Edit: vpc_id, subnet_ids, knfsd_ami_id, source_mounts
./scripts/preflight-check.sh # Validates environment before deploy
terraform init && terraform apply
3. Mount from Client
sudo mkdir -p /mnt/knfsd
sudo mount -t nfs -o vers=4.1 <KNFSD_IP>:/vol1 /mnt/knfsd
echo test > /mnt/knfsd/hello.txt && cat /mnt/knfsd/hello.txt
⚠️ Always specify
vers=4.1. Usingvers=3will cause Stale file handle on file creation.
4. Verify Cache Effect
dd if=/dev/urandom of=/mnt/knfsd/bench.dat bs=1M count=10; sync
sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time cat /mnt/knfsd/bench.dat > /dev/null # ~55ms (first read — fetched from source)
time cat /mnt/knfsd/bench.dat > /dev/null # ~2ms (subsequent — served from cache, 28x)
5. Dual-Path Test (if you have an S3 AP)
# Write via S3 AP
aws s3 cp test.txt "s3://arn:aws:s3:REGION:ACCOUNT:accesspoint/AP-NAME/file.txt"
# Read via KNFSD NFS (immediate reflection)
cat /mnt/knfsd/file.txt
Security note: The above test uses
root(UID 0) as the S3 AP FileSystemIdentity. This is for testing only. In production, use a dedicated non-root user following least-privilege principles.
6. Cleanup
terraform destroy
aws ec2 deregister-image --image-id ami-xxx
Upstream Feedback
Five issues filed during this verification:
| # | Summary |
|---|---|
| #38 | Verification report + minimal example for existing source filers |
| #39 | Private subnet support (graceful degradation without EC2 API) |
| #40 | NFSv3 re-export filehandle overflow documentation |
| #41 | IAM policy missing path-level ARN (bug fix) |
| #42 | NFSv4.1 source mount support (end-to-end v4.1) |
Filing upstream issues for things you find broken (or under-documented) is one of the most impactful things you can do with a Preview-stage OSS project. It ensures the maintainers have real-world data to prioritize GA fixes.
Summary and Next Steps
Dual-Path E2E test passed — S3 AP write → KNFSD NFS read works.
Three key findings:
- NFSv4.1 mandatory (NFSv3 filehandle size limit breaks writes)
- FSID_MODE=local + SQLite on FSx for ONTAP provides FSID persistence at $0
- S3 AP writes reflect immediately through KNFSD NFS
What's next:
- Wait for GA (SLA terms, support scope confirmation)
- Multi-node (ASG + NLB) verification
- EDA/VFX-specific file access pattern benchmarks
- im4gn.16xlarge (30 TB NVMe) L2 bandwidth at production scale
KNFSD File Cache is Preview, but it clearly functions as a practical read-acceleration layer for burst compute workloads.
Small-scale starting point: m6gd.xlarge × 1 running 24/7 = ~$215/mo. Daytime-only (10h) = ~$65/mo.
Resources
- KNFSD File Cache GitHub
- AWS Launch Blog
- Dual-Path Architecture Guide (JA) — 7 industry deep dives + cost estimates + observability
- Terraform + Demo Guide
- FSID Backend Options — SQLite/RDS/Aurora comparison
- Linux Kernel NFS Re-export Docs
- Verification Issues #38-#42
Preview notice: KNFSD File Cache is in Preview as of July 2026. The code is OSS (Apache 2.0) but AWS Preview Service Terms apply. Evaluate for non-production or supplementary use until GA. AWS Support coverage may not extend to KNFSD-specific issues — GitHub Issues is the primary channel. NVMe instance store cache data is protected by AES-256 hardware encryption and auto-erased on instance termination.
Top comments (0)