The goal of this article is to explain the different ways provided by AWS to detect incomplete multipart uploads.
What is s3 multipart upload?
For objects bigger than 100MB, AWS provides a feature to chunk an object before uploading it to s3.
What is s3 incomplete multipart upload?
An incomplete multipart upload occurs when a user starts uploading a large object using multipart upload but does not finish the process.
The problem?
We need to pay for the incomplete parts stored in our s3 buckets.
AWS provides two ways to detect it:
- Storage Lens
- CUDOS dashboard
CUDOS
AWS provides CUDOS, a QuickSight dashboard part of the CID Framework(Cost Intelligence Dashboards) to dive deep into your cost usage based on the CUR(Cost & Usage Report) data.
One of the tabs of CUDOS is "Amazon s3". At the bottom of it, we have a view named "Buckets with Incomplete MultiPart Uploads Last 30 Days".
The source of this dashboard is the CUR (Cost Usage Report). When I saw this view, I wondered:
How can we retrieve this information from a cost perspective without access to S3 metrics?
Let's deep dive into the CUDOS template and what we have behind the view above.
From the analysis, if you click on the view, we will see the fields used in it.
Let's check what we have behind the "MPU Request Delta" calculated field.
In the CUR we have two operations:
- InitiateMultipartUpload
- CompleteMultipartUpload
The people who built this view must have thought like this:
"We cannot obtain the exact number of gigabytes of lost storage from the CUR. Nevertheless, we can still get an idea if we take the delta between 'Initiate' and 'Complete' requests we will know if some multipart uploads failed."
To know exactly the amount of gigabytes, we need to go through Storage Lens.
Storage Lens
For each of my articles, I tried to not duplicate existing content.
For s3 multipart upload with Storage Lens, you can check this blog post written by AWS employees.
Key take Aways from it:
- Delegate Storage Lens administration to a member account in your AWS organization (Audit or Security account).
- Create an organizational dashboard from the member account.
Observation
The default configuration for s3 buckets should include a lifecycle rule to automatically abort old incomplete multipart uploads. It would be good to just have a checkbox to activate it.
Top comments (0)