Conclusion first: on Amazon FSx for NetApp ONTAP you can change the QoS burst defaults. Not only is the configuration accepted — a brief overshoot above the ceiling (the burst) happened exactly as configured. The catch: you cannot change this setting from the AWS Management Console or the AWS CLI (the AWS API). FSx for ONTAP has two ways in — the AWS-side consoles and APIs, and a separate path that drives the storage engine (ONTAP) directly through its own REST API / CLI. The QoS burst can be set only from that second path.
What this post covers, and what it does not
A colleague at NetApp asked me, "since ONTAP 9.19.1 you can change the QoS burst defaults (burst-percent / burst-duration / burst-iops), but is that possible on FSx for ONTAP too?" — so I tried it.
This post records what I measured on a verification FSx for ONTAP file system in ap-northeast-1 (ONTAP 9.19.1P2): setting the options, and confirming that a burst above the ceiling occurs as configured.
Who benefits from this setting
When several workloads or tenants share one file system, one workload can eat the bandwidth and slow the others — the "noisy neighbor" problem. A QoS ceiling holds that down, and burst is the buffer that lets a workload briefly exceed the ceiling at startup or cutover. Being able to change the burst defaults on 9.19.1 means you can tune how the ceiling is exceeded per use case.
On AWS, here is who it helps. If you run databases like Oracle / SQL Server / SAP on Amazon EC2 with their data on FSx for ONTAP, you can cap the nightly batch and reporting side to leave bandwidth for daytime transaction processing (OLTP), while absorbing the database-startup spike with burst. If you operate virtual desktops (VDI, where employees' Windows sessions run on servers rather than their own PCs), you can absorb the morning login storm with burst and keep fairness with a per-group ceiling. If you run multi-tenant SaaS on one file system, you can set ceilings at each level of the storage hierarchy (SVM → volume → qtree, a set of nested units) and vary the burst per tenant to express different service levels (SLA tiers). Each is the same move: keep the ceiling itself fixed, change only how it is exceeded, per tenant or workload.
The use-case mappings and how to choose between QoS and the alternatives (application-layer rate limiting, separate file systems, separate throughput capacity) are organized in the companion note on GitHub (Which workloads the QoS ceiling and burst help). What follows is the premise behind all of that: whether FSx for ONTAP even lets you change the defaults.
What it does not cover: whether a downgrade (revert) is operationally possible on FSx for ONTAP (unverified); the effect of burst values on overall workload cost; interaction with other features such as SnapMirror.
What changed from the previous post
Earlier I measured the speed and monthly cost of using S3 as files versus FSx for ONTAP (I measured the speed and monthly cost of Amazon S3 Files). That time I measured the one-client speed limit and how it scales with client count. This time it is the same FSx for ONTAP, but the question is whether the defaults for how a QoS ceiling is exceeded (burst) can be changed on 9.19.1. Not the ceiling itself, but the design lever for how it is exceeded.
What QoS burst is
A QoS throughput ceiling caps a workload's throughput at a maximum (in IOPS — operations per second — or MB/s). Even when other workloads are competing for the same storage, it guarantees the workload does not exceed the cap. It is used to hold down a noisy neighbor.
But the ceiling is not a hard wall: to respond to short spikes, by default a workload may exceed the ceiling by 50% for 1 second. This absorbs temporary resource demands — starting a database, booting many VMs — without permanently adding capacity.
Since ONTAP 9.19.1, you can change this "50% / 1 second" default. There are three options.
When I measured it, with the ceiling at 2000 IOPS, the default (burst-percent 50 / duration 1) reached about 3000 IOPS (+50%) for the first second only and settled to 2000 by the second second. Setting burst-percent to 25 shrank the first peak to about 2500 (+25%), and setting burst-duration to 5 held about 3000 for 5 seconds. The width, duration, and absolute magnitude of the excess moved exactly as configured.
xychart-beta
title "Burst behavior at ceiling=2000 IOPS (measured, 4k random read)"
x-axis "seconds" [1, 2, 3, 4, 5, 6]
y-axis "IOPS" 0 --> 3500
line "default 50% / 1s" [3017, 2000, 2000, 2000, 2000, 2000]
line "25% / 1s" [2507, 2000, 2000, 2000, 2000, 2000]
line "50% / 5s" [3038, 2970, 3032, 2997, 2970, 2000]| Option | Default | Meaning |
|---|---|---|
| burst-percent | 50% | the percentage over the ceiling (the peak excess width) |
| burst-duration | 1 second | how many seconds the excess is sustained |
| burst-iops | 0 | the absolute IOPS cap during the excess (0 means unused) |
burst-percent and burst-iops are mutually exclusive: you set the excess either as a percentage or as an absolute IOPS value.
Checking on a live file system
I reached a verification FSx for ONTAP file system in ap-northeast-1 (ONTAP 9.19.1P2, single-AZ) from a Linux client inside the VPC, using the commands and API that drive the storage engine (ONTAP) directly. That management connection is only reachable from inside the VPC, so I went through a bastion host placed in the VPC.
First, does the configuration go through? I created a QoS policy group with a 2000 IOPS ceiling, set burst-percent to 25, set burst-duration to 5. Both create and modify succeeded, and reading the values back confirmed them.
Then the real question: does changing the setting actually change the read/write behavior? With a load-testing tool (fio) I drove 4 KB random reads well past the 2000 IOPS ceiling (without QoS it does 24,400 IOPS) and observed IOPS at one-second resolution. Burst only kicks in once enough "credit" has built up, so I left an idle period before the load to accumulate it.
| Setting | t=1s peak | Steady | What happened |
|---|---|---|---|
| default (percent=50, duration=1) | 3017 | 2000 | exceeded ceiling by +50% for 1 second, settled to 2000 by the 2nd |
| percent=25, duration=1 | 2507 | 2000 | the peak excess shrank to +25% |
| percent=50, duration=5 | ~3000 held for 5s | 2000 | the excess lasted 5 seconds |
| burst-iops=3000, duration=5 | ~3000 held for 5s | 2000 | the excess IOPS pinned to the absolute value 3000 |
| disabled (0/0/0) | 2020 (~+1%) | 2000 | the excess all but disappeared |
burst-percent governed the peak excess width, burst-duration the number of seconds the excess lasted, and burst-iops the absolute IOPS during the excess. Setting 0/0/0 removes the burst entirely. Which aspect of the behavior moves is cleanly separated by which option you set.
The NetApp documentation states that a ceiling may be exceeded by up to 10%, and by up to 50% to handle bursts. The default +50% peak and the ~+1% when disabled match that.
The errors along the way were not FSx constraints
Two errors came up, and both are ONTAP's own constraints, not something FSx for ONTAP blocks.
- burst-iops and burst-percent cannot be specified together — they are mutually exclusive. Reset one to 0, then set the other, and it goes through.
- burst-iops must be larger than the ceiling IOPS. Against a 2000 IOPS ceiling, burst-iops=500 was rejected and 3000 was accepted. The command reference also says that if burst-iops is below max-throughput, max-throughput is used.
Separating "FSx for ONTAP cannot do this" from "this is how ONTAP behaves" matters. A managed service can hide or restrict settings the underlying product exposes, so when something is rejected, unless you isolate which layer rejected it, you misattribute the cause.
Which way in lets you change it, and which does not
I only checked via the storage engine's own commands and API (ONTAP REST / CLI). On that path the burst defaults can be changed, and the change shows up in real read/write behavior.
The AWS Management Console and the AWS CLI (the AWS API) have no item for these burst options. As noted at the top, FSx for ONTAP has two ways in — the AWS-side consoles and APIs, and the path that drives the storage engine (ONTAP) directly — and QoS settings live on the second one. So the practical takeaway: looking in the usual AWS console will not find it; you set it on the storage-engine side.
Architecture and operational caveats
The intro covered who benefits. Here is the same setting as one picture of how you actually run it on AWS, with the points that trip you up in operation. The figure shows the flow from the workload to FSx for ONTAP, plus monitoring and automated configuration.
As the figure shows, setting QoS itself rides the path that drives the storage engine (ONTAP) — in the diagram, called from AWS Lambda / Amazon EventBridge. Setting the ceiling and burst by hand every time you add a tenant is not realistic, so you automate it as part of provisioning (the process that stands up each new environment). Per the two-ways-in split from earlier, this path is the storage-engine side; it is not on the AWS API side.
Three caveats I hit, or expect to hit, with this shape:
1. There is no burst-specific metric to monitor. What Amazon CloudWatch gives you is the throughput time series; there is no value that directly says "bursting now." Whether you are pinned at the ceiling or briefly exceeded it is something you read off the per-second series yourself. In this verification, the burst peak was invisible at anything coarser than one-second resolution — it averages away.
The finest unit the standard CloudWatch metrics can target is the volume (VolumeId), with the file system (FileSystemId) above it; there is no breakdown (dimension) that identifies a QoS policy group or workload, so from the standard metrics you cannot tell which policy group a number belongs to. If the unit you want to monitor is the volume or the file system, the standard metrics are enough — but when you want it per policy group or per workload, you close that gap yourself. ONTAP has a facility that exposes performance figures (its counter manager), reachable over the REST API, and a small Lambda can read it on a schedule and write per-workload FSxONTAP/QoS metrics into CloudWatch. I added exactly that collector to the Observability-integrations repo and verified the collection mechanism against the same live 9.19.1P2 file system. Two honest limits stay. First, this facility returns running totals (numbers that only ever grow since boot), so a single read does not tell you "how many per second"; the collector reads twice in one run and divides the difference by the elapsed time to get a per-second value (an average over a short window, not a long one). Second, there is still no burst-specific figure, so these metrics show effective throughput and IOPS and "pinned at the ceiling" — "currently bursting" remains inferred, not measured.
2. When ceilings are nested, the real overshoot is bounded by the tightest one. For multi-tenant SaaS where you set ceilings at each level of the storage hierarchy (SVM → volume → qtree), wherever you set burst, it only works up to the strictest ceiling. A generous burst on a lower-level qtree still tops out at a low SVM ceiling above it. Unless you manage which value sits at which layer consistently, you do not get the service-level (SLA) tiers you intended.
3. Ingest paths (AWS DataSync / SnapMirror) can crowd out production reads. Capping the migration/replication ingest keeps it from eating production reads at peak. One pattern is to relax it with burst only during catch-up windows. This is a design expectation, though — I did not measure specific burst values for ingest workloads.
4. ONTAP QoS burst and FSx-for-ONTAP-side burst are different layers. FSx for ONTAP itself also bursts above a baseline for a time (performance documentation). QoS burst governs "how far a workload may exceed its ceiling"; FSx-side burst governs "how much the file system itself can deliver" — different layers. In this measurement the peak was ~3,000 IOPS (~12 MB/s) against ~24,400 IOPS available without QoS, a large margin, so FSx-side bursting did not affect the result (this part I confirmed by measurement). As a design caution, though: if you size so the sum of per-workload (ceiling + burst) exceeds the file system's baseline (sustained) performance, a workload might not even reach its ceiling once FSx-side credits are exhausted, and the same QoS settings can behave differently depending on credit state (I did not measure this saturated case). Size the sum of ceilings to the baseline performance, not the burst.
Caveats 2 and 3, and the per-workload mappings from the intro, are cases where each service's role is real but the combination with QoS is my design expectation. What I measured is the first half: that the defaults can be changed and are reflected in real I/O. The detailed use-case mappings are in the GitHub note.
QoS is not the only way to cap throughput
Isolating workloads and capping throughput is not QoS alone. You can rate-limit at the application layer, separate the file systems themselves, or allocate throughput capacity separately — each with its own fit.
Roughly, it splits on "is the storage layer enough, or do you need to see the meaning of the requests?" If you want to control by request type or priority, choose the application layer; if you only want to suppress storage contention, choose QoS. If you need full physical isolation between tenants, separate file systems; if you need guaranteed bandwidth, separate capacity. QoS is not mutually exclusive with these — they can be combined. None is always right; you pick by requirement.
Summary
- On FSx for ONTAP 9.19.1P2 the QoS burst defaults can be changed. Confirmed in both configuration and real read/write behavior.
- You change it from the storage-engine (ONTAP) side. The AWS Management Console and AWS CLI (the AWS API) have no matching item.
- burst-percent governs the peak excess width, burst-duration the sustained seconds, burst-iops the absolute IOPS during the excess.
- The mutual exclusion and the value floor are ONTAP's own constraints, not ones FSx for ONTAP adds.
All the measured values are on GitHub with their environment (the FSx for ONTAP adoption playbook, performance domain). I hope this helps anyone who has set a QoS ceiling and wants to shape how it is exceeded to their own requirements.
Until next time.
Yoshiki Fujiwara
Top comments (0)