Introduction
Part 4 described putting 182 ONTAP operations on a file portal: the design of delegation and record, and the treatment of buttons that cannot be pressed and buttons that were never built.
This part is what the running cluster refused while those 182 were being built. None of it was visible while reading the documentation; each one arrived as an error code. That is the difference from last time: Part 4 was about design, this one is about measurement.
The readers I have in mind are people writing code against the FSx for ONTAP ONTAP REST API. You do not need to be building a portal — if you call the same API, you stop in the same places.
What this part covers: FlexGroup creation, FlexGroup capacity rebalance, how to read the capacity numbers, deletes that refuse, and four pitfalls in the S3-compatible API. Each with the response the system actually returned.
What this part does not cover:
- ONTAP internals (I have no way to confirm them)
- Absolute throughput figures (the subject here is whether something is possible and under what condition — but the defaults that decide whether a measured figure can be trusted are covered in a section below)
- Running SnapLock compliance (irreversible, and it would make a test file system undeletable for months, so it was deliberately not done)
- Actual tiering into a capacity pool (needs a different aggregate configuration, which was not on this path)
Up front:
- Creating a FlexGroup on FSx for ONTAP always fails if you let ONTAP choose the aggregate. FSx for ONTAP aggregates have tiering enabled and are excluded from automatic placement
- The same root cause is documented on the FlexCache side as a flag (
use_tiered_aggregate), and it had not reached the volume-creation side -
The capacity rebalance
max_runtimehas a lower and an upper bound that are not in the API reference. With ONTAP's defaults it never starts - Starting a rebalance enables an irreversible setting. Stopping it does not undo it
-
space.usedmeans a different quantity depending on the volume. Snapshots inside the reserve are not counted - A clone that refuses a parent delete is invisible to the API. There are two ways out, and both were measured
- Two places the documentation and the running system disagreed, and both changed the screen's wording
- Five combinations of defaults return a plausible figure while measuring something other than what you intended. Three of them come from reads being served out of cache
Test environment
| Item | Value |
|---|---|
| Dates | mid-August 2026 to 2026-09-02 (JST) |
| Region | ap-northeast-1 |
| ONTAP version | 9.18.1P3D1 |
| Deployment type | SINGLE_AZ_1 |
| Caller | ONTAP REST API from a VPC Lambda (Python 3.13) |
| Privilege | fsxadmin |
| Cleanup | All test volumes deleted |
The distinction that matters most — ONTAP's words and AWS's words
Mix these up and searching for the error text will not find the answer.
| Term | Where it appears |
|---|---|
FabricPool, use_tiered_aggregate
|
ONTAP error text and REST field names |
| tiering, capacity pool tier | AWS documentation and console |
They refer to the same mechanism. But the error arrives in ONTAP's vocabulary while the documentation explains it in AWS's, so searching the AWS documentation for the error string returns nothing. The portal's wording follows the AWS side; only API strings are quoted as-is.
FlexGroup creation and the failure of automatic placement
A FlexGroup presents several "constituents" — internally split real volumes — as one namespace. ONTAP normally picks the aggregates if you do not name them. On FSx for ONTAP that always fails.
Aggregates not matching FabricPool requirements: aggr1
An FSx for ONTAP file system has a primary storage tier (SSD) and a capacity pool storage tier, and infrequently accessed data is tiered to the latter (AWS documentation). Aggregates in that configuration fall outside ONTAP's automatic placement.
The conditions came out one failure at a time.
| Step | Result |
|---|---|
| Create with default parameters | Failed. Volumes of this type must be at least 50GB
|
| Retry at 50 GiB | Failed. Aggregates not matching FabricPool requirements: aggr1
|
| Retry naming the aggregate | Failed. Minimum size is "400GB" (4 constituents × 100 GiB) |
400 GiB, aggregate named, tiering.policy=none, thin |
Succeeded |
Naming the aggregate works. But that aggregate name appears in neither the AWS console nor the FSx API. It has to be fetched with GET /storage/aggregates. Nobody hands the tenant an aggregate name, so that extra call is mandatory.
The same root cause was already known on the FlexCache side. There is a flag, use_tiered_aggregate, defaulting to false. It had not reached the volume-creation side. The same cause surfacing elsewhere with a different face is a familiar shape in a managed service that wraps upstream software.
The screen carries this note:
When creating a FlexGroup from this screen, the aggregate is named explicitly rather than placed automatically. FSx for ONTAP aggregates have tiering enabled and are excluded from automatic placement (measured: without it, creation fails with "Aggregates not matching FabricPool requirements"). With the default four constituents the minimum is 400 GB.
The capacity rebalance that never starts with the defaults
A FlexGroup places files across constituents by hash, so they drift apart over time. When one fills, the whole volume returns "no space" even though the others have room. A capacity rebalance is what corrects that.
Running it against the real cluster produced two constraints on max_runtime that are not in the API reference.
| Constraint | Detail | Error code |
|---|---|---|
| Lower bound | Under 30 minutes is refused | 144182221 |
| Upper bound | Must be shorter than the time remaining until the next snapshot | 13107433 |
Together, the window to start is:
30 min <= max_runtime < (time from start until the next snapshot)
ONTAP's default max_runtime is 6 hours. On a volume with the default snapshot policy (hourly at :05), 6 hours always violates the upper bound. With the value the portal offered as its default, it never started once. With the default policy the only window is the 30 minutes from :05 to :35 each hour. On a frequent schedule such as 5min, nothing starts until the policy is detached.
The boundary was pinned with a one-second A/B.
| Time | maxRuntime |
Until next snapshot | Result |
|---|---|---|---|
| 16:06:12 |
PT1H (60 min) |
58m47s | Refused (13107433) |
| 16:06:13 |
PT30M (30 min) |
58m46s | Succeeded |
60 minutes > 58m47s refused; 30 minutes < 58m46s succeeded. Because the two calls are one second apart, the deciding factor is the max_runtime value and not elapsed time. ONTAP's own message names both remedies — "reduce the -max-runtime or disable the snapshot policy" — because of this structure.
The full observation log is in the FlexGroup capacity rebalance record. State values absent from the reference's volume-side list turned up there too: idle means running with nothing to move, scheduled means a reservation exists. The portal had been displaying "running" as "unknown".
A note on irreversibility
Starting a rebalance enables
granular dataon that volume. It is required for the rebalance, and it creates two multi-part inodes per moved file (ONTAP documentation). Stopping does not undo it. The vendor's documentation lists no way back other than deleting the volume or restoring from a snapshot taken before it was enabled.Stopping is not a rollback either: files already moved stay where they are. Both were measured (
granular_dataremainstrueafter a stop). The screen's confirmation says both.
The space that does not come back after a delete
Read the usage figure in a volume list naively and you will get it wrong, because space.used means a different quantity depending on the volume.
- Snapshots that fit inside the snapshot reserve (5% by default) are not counted in
space.used - Anything past the reserve consumes the same space as live data and is counted
The measurements. A 100 GiB volume reported used of 18.1 MiB while holding 77.3 MiB of snapshots. That sits inside the 5% reserve (5 GiB), so the usage bar does not move. On a different volume with a 0% reserve, used was 83,677 MiB: 81,934 MiB of live data and 1,743 MiB of snapshots. On 8 of 11 volumes, snapshots exceeded live data.
So the list was split from one usage figure into three: live data, snapshots, and reserve spill. Whether the space is not coming back because of snapshots or because live data fills it can now be read off the screen.
Sometimes the cause is on another volume and its own row does not show it. That is FlexClone. A clone is based on one of the parent's snapshots, and that snapshot stays locked while the clone exists. Deleting files on the parent frees nothing while the locked snapshot still references the blocks. Reading the parent's row alone cannot tell you that, so it is written on the FlexClone panel instead.
The exits from a refused delete
I reproduced the state where deleting a parent volume is refused with has one or more clones. The awkward part is that the API cannot see that clone. It is not in the clone listing, fetching it directly returns entry doesn't exist, and yet the parent refuses, saying a clone exists.
The cause is ONTAP's volume recovery queue, which retains deleted volumes for 12 hours by default. An A/B in the same environment, varying only whether the clone was split first:
| Steps | Parent delete |
|---|---|
| Delete an unsplit clone, then delete the parent | Failed (still failing after 7 and 15 minutes) |
| Split the clone, delete it, then delete the parent | Succeeded within seconds |
The exit was measured too. GET /api/private/cli/volume/recovery-queue reads the queue. POST .../purge goes through with fsxadmin privilege, the entry left the queue in about 20 seconds, and the parent delete succeeded immediately afterwards.
This is a place where I published something wrong and have corrected it. The earlier text said purge needs diag privilege, is unreachable as fsxadmin, and therefore the only option is to wait 12 hours. That was wrong. I blamed privilege without trying it.
The screen's error carries the causation and both exits (split first, or purge). Purge cannot be undone, so it also says to confirm the volume was yours before running it.
Four pitfalls in the S3-compatible API
Behaviours worth knowing before you call an FSx for ONTAP S3 access point from a browser or an SDK. Knowing them changes how long the investigation takes.
The 501 failure of a "write only if absent" request
PutObject with if-none-match: * returns 501 NotImplemented, with the body A header you provided implies functionality that is not implemented.
| Request | Result |
|---|---|
PUT + if-none-match: *
|
501 NotImplemented |
PUT + x-amz-checksum-crc32 (no if-none-match) |
200 OK |
GET / ListObjectsV2
|
200 (neither header is sent) |
CRC32 flexible checksums pass. Only if-none-match fails. Reads are unaffected, so the symptom presents as "the listing works but every write fails."
This is not somebody else's problem. Amplify's Storage Browser translates its "do not overwrite" option into that header, so dropping it in unchanged breaks every upload and folder creation. If you need overwrite protection, look the key up before writing instead. That does not give you if-none-match's atomicity, and two concurrent writes to the same key can both decide it is absent.
Two delete-protection modes, indistinguishable from the error
Deleting an object written with one day of retention:
| Operation | GOVERNANCE | COMPLIANCE |
|---|---|---|
| Delete without bypass | AccessDenied ... object protected by object lock |
Identical text |
Delete with BypassGovernanceRetention
|
Succeeds | Refused with identical text |
Because the text is identical, the response does not tell you why the bypass did not work. The first things to read are get-object-lock-configuration and head-object's ObjectLockMode, not the IAM policy. Get that backwards and you lose time digging into permissions.
The lock indicator that does not change under automatic locking
Assigning a policy whose schedules carry a retentionPeriod leaves the locking feature flag at false; only the assigned policy name changes. Reading the state alone suggests nothing will be locked, while every snapshot that policy takes is in fact locked. The judgement needs the assigned policy's retentionPeriod as well.
The usable endpoints, derived from the API rather than a config file
An alias copied into a config file looks equally correct for an access point that has been deleted and for one that is MISCONFIGURED. fsx describe-s3-access-point-attachments returns Lifecycle and the Internet or VPC origin, so derive the inventory from there. Always follow the pagination: read only the first page and an access point that exists goes missing in a way indistinguishable from one that does not.
Two places the documentation and the running system disagreed
Two things behaved differently from what I read before implementing them. Both changed the screen's wording.
| Subject | Expected from the documentation | Measured (ONTAP 9.18.1P3D1) |
|---|---|---|
| Deleting a QoS policy while assigned | The CLI reference says it is refused without -force
|
REST accepts it, and the volume's assignment is silently removed (every ceiling becomes 0, meaning unlimited) |
| Enforcement after deleting a quota rule | The REST reference says it stays in effect until enforcement is cycled off and on | The deleted rule's ceiling disappeared from the usage report immediately |
The QoS confirmation text was rewritten. The correct warning is not "it cannot be deleted" but "deleting it removes the ceilings from every volume using that policy". A note was added that releasing a single volume means assigning none, not deleting the policy.
The five defaults that can invalidate a measurement
Up to here the defaults decided whether an operation goes through. Measurement has defaults of the same shape. None of them returns an error: each returns a plausible figure while measuring something other than what you intended.
Where these figures come from: a different measurement from the environment above. 2026-09-01 to 02, ap-northeast-1, ONTAP 9.18.1P3D1, SINGLE_AZ_1, throughput tiers 128 MBps and 2048 MBps, client c5n.9xlarge. Repeatability across repeat runs of the same measurement was 0.02–0.14%.
| Default | What happens | How to notice | What to do |
|---|---|---|---|
| Linux NFS uses one TCP connection | Stops around 590 MB/s and does not respond to stream count | 1 / 4 / 8 give the same figure | nconnect=16 |
dd if=/dev/zero |
Zero blocks skip disk and return at 4x the published ceiling | Exceeds the published ceiling | Use incompressible data |
| Volume inline efficiency is on | Compressible or identical payloads collapse (4x on reads, under 5% on writes) |
space_savings.dedupe_percent is high |
Turn it off before writing |
DiskIopsConfiguration: AUTOMATIC |
3 IOPS/GiB, so IOPS becomes the limit | MB/s divided by IOPS is implausibly small | USER_PROVISIONED |
| A read only slightly over the cache size | The cache-served share dominates and the disk path is not measured |
DiskReadBytes divided by DataReadBytes is a few per cent or less |
Read at least twice the cache in one pass |
How the fifth one surfaced. I assumed 280 GiB would not fit in cache. Read through Amazon CloudWatch's AWS/FSx namespace, DiskReadBytes divided by DataReadBytes came to 1.4 / 0.9 / 1.5%. 98.5-99.9% of the bytes read never touched disk.
The 2048 MBps tier's in-memory cache is 256 GB, that is 238 GiB (FSx for ONTAP performance; ap-northeast-1 falls under the first-generation Single-AZ "all other Regions" table). 280 GiB exceeds it by 18%, not by a factor of two or three. "Past the cache" and "not served from the cache" are different things.
Inline efficiency can be restored, provided you wait.
# Rejected unless efficiency.op_state is idle
PATCH /api/storage/volumes/{uuid} {"efficiency": {"compression": "inline"}}
PATCH /api/storage/volumes/{uuid} {"efficiency": {"dedupe": "both"}}
But only turn it off on a volume you can discard. Immediately after writing, a background efficiency operation is running and the wait is not predictable.
The range tested on real hardware, and the point where I stopped
"The tests pass" is not "it works in production." Unit tests exercise handler logic; they do not see the real shape of an ONTAP REST response, AppSync authorization, VPC reachability, or Cognito group propagation.
So the verification results are split four ways: real-hardware end to end, real-hardware read-only, automated tests only, and DemoMode only. Keeping the distinction lets me ask myself later whether I actually saw a thing run. The one correction in this article — the recovery queue purge privilege — came out of a place where that distinction had been vague.
Putting it in a runbook
| # | What to write down |
|---|---|
| 1 |
For FlexGroup creation, fetch the aggregate with GET /storage/aggregates and name it. A runbook that relies on automatic placement always fails on FSx for ONTAP |
| 2 |
Set the rebalance max_runtime to at least 30 minutes and below the time to the next snapshot. Do not pass the default through |
| 3 |
Do not treat starting a rebalance as an operation you can press at any time. granular data is enabled irreversibly |
| 4 | Do not show usage as a single number. Separate live data from snapshots, or nobody can reach the cause of "deleting files did not free space" |
| 5 | A teardown procedure for an environment with clones must include the split, or the recovery queue purge. "The clone was deleted, so the parent can go" does not hold |
| 6 | Errors arrive in ONTAP's vocabulary. Keep the AWS-to-ONTAP term mapping at hand before searching |
| 7 |
A runbook for measuring performance states how to drop the defaults first. Leave any one of zero-filled data, inline efficiency, an unspecified nconnect or AUTOMATIC IOPS in place, and the figure produced is measuring something else |
What I did not measure
"Unverified" is not "impossible". As last time, the boundary stated as a boundary:
| Item | State |
|---|---|
| The actual effect of a rebalance (how far the skew is evened out) | Unmeasured. Only the conditions to start were measured |
Performance impact after granular data is enabled |
Unmeasured |
| A recovery queue with a changed retention | Unverified. Measured at the 12-hour default |
| Actual tiering into the capacity pool | Unverified. Needs a different aggregate configuration |
| Running SnapLock compliance | Deliberately not done. Irreversible, up to the file system |
| An alternative path for copies over 5 GiB (NFS or SMB) | Unmeasured. The portal stops at refusing |
Part 6 covers the other side: what was left off the screen. How much of ONTAP stays usable alongside the S3 access point path, and the operational work that was handed to a schedule instead of a button.
Conclusion
Nothing in this part was visible while reading the documentation. Creating a FlexGroup always fails if placement is left to ONTAP, a capacity rebalance never starts with the defaults, space.used means a different quantity per volume, and the clone that refuses a delete cannot be seen through the API.
What they share is that the error text does not name the cause. That is why the causation and the exits live on the screen. As with the one-second A/B on the rebalance, another explanation looked just as correct until one control was added.
There is also one correction. I published "purge needs diag privilege, so wait 12 hours" and later found it goes through. I blamed privilege without trying it.
The figures come from one specific environment and configuration and will differ with workload and setup. The error codes were observed on ONTAP 9.18.1P3D1.
Top comments (0)