Disclaimer
- Opinions are my own.
- Cover image is for Amazon S3 Intelligent-Tiering storage class Automatic Access tiers, but main topic is AWS File Storage Tiering.
- If you have any questions or concerns after reading this article, please let us know.
- Based on the features and contents as of December 31, 2024. If there are any discrepancies, please check the latest AWS official information at the time you read the article.
Table of contents
- Why it's a good time to take a look at AWS File Storage Tiering now?
- File storage services on AWS
- What is Tiering?
- Tierings of AWS file storage services
- Conclusion
Why it's a good time to take a look at AWS File Storage Tiering now?
In conclusion, "AWS file storage services and features are so diversified that it is difficult to understand them as a whole". In such a situation, Tiering is an effective function for optimizing the file storage environment. But, it's difficult to understand because there are no materials for cross-sectional understanding.
I would like to understand it from a bird's-eye view rather than comparing services. If I could not find such materials, I'll write it myself.
The direct trigger of this blog post was the update during AWS re:Invent 2024, which introduced the new FSx storage class Amazon FSx Intelligent-Tiering. You can check the AWS release note titled "Announcing Amazon FSx Intelligent-Tiering, a new storage class for FSx"
This is a announcement related storage during AWS re:Invent 2024 and is an exciting update that is attracting attention.
It was also covered at Storage-JAWS#6, the re:Cap community based webinar of AWS re:Invent 2024 of "Storage-JAWS", a storage specialized branch of the Japanese AWS User Group, or JAWS-UG.
You can check the YouTube video linked above for details. (Since this is a local event in Japan, the introduction was in Japanese)
- If you haven't seen the Storage-JAWS video above, please check it out as the speakers summarize the updates in an easy-to-understand manner, and there are great sessions and LTs that are explained with live demos and screenshots of management console. Please feel free to fill out the survey after watching (this is a guide for me as a Storage-JAWS management member).
...But,
Contrary to the title and content of the release notes above, "FSx Intelligent-Tiering," the reality is that this is a feature only available for Amazon FSx for OpenZFS, and is not available for the entire Amazon FSx series. Other services that follow are not available at the time of writing this blog.
Personally, I felt that this notation was confusing, so I decided to take this opportunity to clarify "What is File Storage Tiering on AWS and how it works?"
Below are some excerpts from the AWS release note. I think there are some expressions that can lead to misunderstandings as to whether this is about the FSx series or FSx for OpenZFS feature. I will try to write this blog in a way that avoids any misunderstandings.
Today, AWS announces the general availability of Amazon FSx Intelligent-Tiering, a new storage class for Amazon FSx that costs up to 85% less than the FSx SSD storage class and up to 20% less than traditional HDD-based NAS storage on premises, and that brings full elasticity and intelligent tiering to network-attached storage (NAS). The new storage class is available today on Amazon FSx for OpenZFS.
Using Amazon FSx, customers can launch and run fully managed cloud file systems that have familiar NAS capabilities such as point-in-time snapshots, data clones, and user quotas. Before today, customers have been moving NAS data sets for mission-critical and performance-intensive workloads to FSx for OpenZFS, using the existing SSD storage class for predictable high performance. With the new FSx Intelligent-Tiering storage class, customers can now bring to FSx for OpenZFS a broad range of general-purpose data sets, including those with a large proportion of infrequently accessed data stored on low-cost HDD on premises. FSx Intelligent-Tiering delivers low-cost storage and costs up to 85% less than the FSx SSD storage class and up to 20% less than traditional HDD-based NAS storage on premises...
Now, let's get into the details.
File storage services on AWS
Items in this chapter
- Storage services on AWS
- File Storage services on AWS
Storage services on AWS
First, let's take a look at storage services on AWS.
As far as we can see from the following AWS Webpage "Cloud Storage on AWS", there are 11 "categories" as shown in the figure below, and each category is further divided into services and features.
Quote: Cloud Storage on AWS
File Storage services on AWS
There are three types of file storage in the diagram above: Amazon Elastic File System (EFS), Amazon FSx, and Amazon File Cache. In this article, we will take a look at the EFS and FSx series other than Amazon File Cache, which is a cache service that does not have a tiering feature.
The characteristics of File Storage and how to choose one are summarized in an easy-to-understand manner on the AWS blog "How to choose an AWS file storage service".
It's written in Japanese, but you can easy to understand the contents of it. Or you can check AWS re:Invent sessions of AWS Files Storage Team like AWS re:Invent 2024 - Network-attached storage in the cloud with Amazon FSx (STG202)
For example, in the AWS blog mentioned above, in the "Storage Types" section below, it confirmed the understanding of the three types of "block storage", "object storage", and "file storage" that will be explained this time,
In the "File Storage Protocols" section below, it discusses the two protocols NFS (Network File System) and SMB (Server Message Block), and in the "AWS File Storage Service" section, it explains Amazon FSx for Luster's unique protocol. Let's check the three protocols including.
In addition, in the "Comparison of AWS File Storage Services" section, the figure below shows a list of services excluding FSx for Luster at the time of writing, and points to consider when making a selection. In "Protocols supported by each service," Amazon FSx for NetApp ONTAP is characterized by being multi-protocol, not only supporting both NFS and SMB protocols, but also supporting iSCSI. It also touches on some unique aspects.
Next, in the section "How to choose an AWS file storage service", the part "Consider what you are looking for" touches on Tiering, the main theme of this blog, and the EFS lifecycle policy. In this blog, I will update and supplement the above table based on the AWS re:Invent 2024 Update.
What is Tiering?
What do you think of when you hear the word "Tiering"?
Let's take a look at Amazon S3, AWS's representative storage. In the feature description of "Amazon S3 Intelligent-Tiering storage class" whose name includes Tiering, This feature is introduced from the perspective of cost optimization as follows.
The Amazon S3 Intelligent-Tiering storage class is designed to optimize storage costs by automatically moving data to the most cost-effective access tier when access patterns change.
However, I guess that the scope of implementation and use cases are gradually expanding now.
For example, on an AWS blog, "Optimizing your AWS Infrastructure for Sustainability, Part II: Storage", the section "Analyze data access patterns and use storage tiers", with the following two explanations, it suggests to make S3 lifecycle management sustainable by using automated tiering.
- Choosing the right storage tier after analyzing data access patterns gives you more sustainable storage options in the cloud.
- For data with unknown or changing access patterns, use Amazon S3 Intelligent-Tiering to monitor access patterns and move objects among tiers automatically.
In this way, we can see that storage tiering is important not only from a cost optimization perspective but also from a sustainability perspective.
I often use not only S3 but also the EFS and FSx series of file storage services from the perspective of optimizing cost and performance, and provide design support. I will take a deep dive into file storage, which has a wide variety of types and features.
Tierings of AWS file storage services
Items in this chapter
- Tierings of AWS file storage services
- Relevant information
Tierings of AWS file storage services
| Description/Service | EFS | FSx for OpenZFS | FSx for ONTAP | FSx for Lustre | FSx for Windows |
|---|---|---|---|---|---|
| Tiering | Available | Available | Available | N/A | N/A |
| Tiering granularity | File Level | Data Block Level | Data Block Level | N/A | N/A |
| Tiering configuration unit | File System | File System | Volume | N/A | N/A |
| Tiering Class/Pool | 1.Standard 2.Infrequent Access (IA) 3.Archive |
1.Frequent Access 2.Infrequent Access 3.Archive |
1.Primary Storage(SSD) 2.Capacity Pool (HDD) |
N/A | N/A |
I purposely included "Tiering granularity". This is because it's one of pitfalls of tiering. Let's say S3 and EFS implement file/object level tiering, so when the file/object is read, it is determined that it has been accessed and the tiering is applied even if the data blocks in the file have hardly been read and tiering won't be triggered the rest of data block has never been accessed.
On the other hand, FSx for OpenZFS and FSx for ONTAP have data block level tiering, so the data blocks that can potentially be optimized for cost/performance through tiering may be wider.
- As an example, if you open this blog and leave after just seeing the title, EFS and S3 will not be subject to Tiering who this file/object, but FSx for OpenZFS and FSx for ONTAP will execute Tiering for unread data blocks other than the title. As described in the ONTAP Knowledge Base and Technical Report, FSx for ONTAP judges data blocks in 4K units and performs tiering in 4M units. Regarding FSx for OpenZFS, I have not yet been able to find any documentation that shows the specific behavior of Tiering at the data block level, so if anyone knows about it, I would appreciate it if you could let me know.
Cautions and TIPs:
- The FSx Intelligent-Tiering storage class of FSx for OpenZFS can only be used in multi-AZ configurations on a file system basis.
- Screenshot of AWS Management Console for creating FSx for OpenZFS file system. When you choose Intelligent-Tiering (elastic) Storage class, you cannot choose Single-AZ 2 (HA) nor Single-AZ 2 (non-HA).
- Screenshot of AWS Management Console for creating FSx for OpenZFS file system. When you choose Intelligent-Tiering (elastic) Storage class, you cannot choose Single-AZ 2 (HA) nor Single-AZ 2 (non-HA).
- Tiering in FSx for ONTAP allows you to change and tune the Tiering Policy even after the file system and volume are created. Both single-AZ and multi-AZ configurations are possible.
- Screenshots of AWS Management Console for creating FSx for ONTAP file system and Updating volume.
- Screenshots of AWS Management Console for creating FSx for ONTAP file system and Updating volume.
- If you want to run EFS Tiering when creating a file system, select "Customize" and set it in "Lifecycle Management".
Relevant information
EFS Tiering：
- Amazon EFS Pricing: > Amazon EFS offers three storage classes: EFS Standard, SSD-based storage which delivers sub-millisecond latencies for actively-used data; EFS Infrequent Access (EFS IA), cost-optimized storage which delivers milliseconds latencies for data accessed only a few times a quarter; and EFS Archive, cost-optimized storage which delivers milliseconds latencies for long-lived data accessed a few times a year or less.
Amazon EFS Infrequent Access：「Amazon EFS will automatically and transparently move your files to the lower cost regional EFS IA storage class based on the last time they were accessed. 」
Amazon EFS Archive：「Amazon EFS will automatically and transparently move your files to the lower cost EFS IA and Archive storage classes based on the last time they were accessed.」
FSx for OpenZFS Tiering：
Intelligent-Tiering delivers automatic storage cost savings when data access patterns change, without performance impact or operational overhead. The Amazon FSx for OpenZFS Intelligent-Tiering storage class is designed to optimize storage costs using elasticity to automatically move data to the most cost-effective access tier when access patterns change. Amazon FSx Intelligent-Tiering is up to 85% lower cost than the FSx SSD storage class, and up to 20% lower cost compared to traditional on-premises HDD deployments.
FSx for ONTAP：
Each Amazon FSx for NetApp ONTAP file system has two storage tiers: primary storage and capacity pool storage. Primary storage is provisioned, scalable, high-performance SSD storage that’s purpose-built for the active portion of your data set. Capacity pool storage is a fully elastic storage tier that can scale to petabytes in size and is cost-optimized for infrequently-accessed data. Amazon FSx for NetApp ONTAP automatically tiers data from SSD storage to capacity pool storage based on your access patterns, allowing you to achieve SSD levels of performance for your workload while only paying for SSD storage for a small fraction of your data. Capacity pool storage automatically grows and shrinks as you tier data to it, providing elastic storage for the portion of your data set that grows over time without the need to plan or provision capacity for this data.
- ONTAP Tiering logic: Technical Report FabricPool best practices ONTAP 9.14.1
Data movement > Tiering data to an object store
After a block has been identified as cold, it is marked for tiering. During this time, a background tiering scan looks for cold blocks. When enough 4KB blocks from the same volume have been collected, they are concatenated into a 4MB object and moved to the cloud tier based on the volume tiering policy.
Conclusion
What do you think of this summary of AWS file storage Tiering?
When selecting a file system, you will most likely choose one that you are familiar with. However, from the perspective of cost/performance optimization, data integration, application configuration, security, etc., why not consider equally the features and benefits of other services?
Among the options discussed this time, there are differences in the tiering methods for EFS, FSx for OpenZFS, and FSx for ONTAP, and you will need to make a choice based on usage and user experience.
Regarding prices, the actual amount and validity vary greatly depending on the method of use and purpose, and it often changes with the release of new features, so I intentionally did not include it in the list this time so as not to make the comparison stand alone.
I am sure that Amazon FSx Intelligent-Tiering will become even more powerful in the future. Let's keep an eye on AWS file storage and Tiering, which will continue to evolve!!
I hope this blog will be helpful to someone.
Bye now!!
Socials:
Top comments (0)