DEV Community: Yuichi Sato

S3 Presigned URLs SSE-KMS: Common Pitfalls and How to Avoid Them

Yuichi Sato — Tue, 16 Jun 2026 10:44:48 +0000

This article was originally written in Japanese and published on Qiita. It has been translated with the help of AI.
Original article: https://qiita.com/sassssan68/items/9f37fac0f4e2210deb0c

You switched S3 encryption from SSE-S3 to SSE-KMS, and suddenly your presigned URLs started returning errors.
Sound familiar?
It happened to me.

Presigned URLs and SSE-KMS are both common features when used on their own, but combining them comes with some tricky permission requirements.

This article covers:

Why presigned URLs fail when you switch to SSE-KMS
Easy-to-miss permission pitfalls (IAM policies and KMS key policies)
Different gotchas for downloads (GET) and uploads (PUT)

Background

What Is a Presigned URL?

A presigned URL grants temporary access to an S3 object.
Even users without IAM credentials can use the URL to download or upload S3 objects.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html

Common use cases:

Letting a user download a file temporarily
Letting a user upload a file (e.g., form submission)

# Generate a download presigned URL (valid for 3600 seconds)
aws s3 presign s3://your-bucket/your-file.pdf --expires-in 3600

The important thing to keep in mind is that a presigned URL borrows the permissions of the user who generated it.
If the generator doesn't have the required permissions, the URL can still be issued, but actual access will fail.

What Is SSE-KMS?

SSE-KMS is one of S3's server-side encryption methods. It encrypts objects using a customer managed key in AWS KMS.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html

Comparing S3 encryption methods:

Method	Key Manager	Compatibility with Presigned URLs
SSE-S3	Fully managed by AWS	Works without issues
SSE-KMS (AWS managed key)	Managed by AWS (`aws/s3`)	Has constraints
SSE-KMS (customer managed key)	Managed by user	Has constraints

Why It Works with SSE-S3 but Fails with SSE-KMS

With SSE-S3, S3 manages the keys internally, so no extra permission setup is needed.
With SSE-KMS, on the other hand, separate permissions to access the KMS key are required. This is the part you need to watch out for.

Worth noting: this applies even when you use the AWS managed key (aws/s3) for SSE-KMS.
You might think, "AWS manages this key, so it should work like SSE-S3 without extra permissions, right?" But as long as it's SSE-KMS, KMS permissions are required regardless of who manages the key.

When you access an object via a presigned URL, here's what happens internally:

With SSE-S3

Request comes in via the presigned URL
S3 decrypts the data with its internal key
Data is returned

With SSE-KMS

Request comes in via the presigned URL
S3 asks KMS to decrypt the data key ← Without KMS key permissions, this fails
KMS decrypts and returns the data key
S3 decrypts the data with the data key
Data is returned

With SSE-KMS, you need permissions for both S3 and the KMS key.
If the user generating the presigned URL doesn't have permission to use the KMS key, the URL can still be issued, but the actual request will return an error.

Common Pitfalls

1. IAM Policies Need KMS Permissions

The IAM user or role generating the presigned URL needs to have KMS-related permissions added to its policy.

For Downloads (GET)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowS3GetObject",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-bucket/*"
        },
        {
            "Sid": "AllowKMSDecrypt",
            "Effect": "Allow",
            "Action": "kms:Decrypt",
            "Resource": "arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id"
        }
    ]
}

s3:GetObject alone isn't enough — without kms:Decrypt, you'll get an error.

For Uploads (PUT)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowS3PutObject",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::my-bucket/*"
        },
        {
            "Sid": "AllowKMSForUpload",
            "Effect": "Allow",
            "Action": "kms:GenerateDataKey",
            "Resource": "arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id"
        }
    ]
}

For uploads, kms:GenerateDataKey is required.
S3 needs to ask KMS to generate a data key in order to encrypt the object.

⚠️ Note
For multipart uploads, kms:Decrypt is also required.
Each part is encrypted using the same data key, which means the encrypted data key needs to be decrypted on subsequent parts.

2. Explicitly Specify Signature Version 4 in Boto3

When using SSE-KMS, Signature Version 4 (SigV4) is mandatory.
This is documented in the official AWS docs.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html#aws-signature-version-4-sse-kms

Recent versions of Boto3 use SigV4 by default, but depending on your environment or configuration, it may fall back to SigV2.
In that case, you'll get an InvalidArgument error like this:

<Error>
  <Code>InvalidArgument</Code>
  <Message>
    Requests specifying Server Side Encryption with AWS KMS managed keys
    require AWS Signature Version 4.
  </Message>
  <ArgumentName>Authorization</ArgumentName>
</Error>

To make sure it works reliably, explicitly specify SigV4 when creating the client.

from botocore.config import Config

s3_client = boto3.client("s3", config=Config(signature_version="s3v4"))

I actually got tripped up by this myself.
Everything worked fine with SSE-S3, but the moment I switched to SSE-KMS, presigned URLs stopped working — and the cause was that I forgot to explicitly specify SigV4.

⚠️ Note
SigV4 may not be the default in environments like:

Older versions of Boto3 / Botocore

When signature_version is explicitly set in ~/.aws/config

Legacy region settings

When using SSE-KMS, always specify Config(signature_version="s3v4") to be safe.

3. KMS Key Policy Permissions

If you created the KMS key with default settings, the root delegation policy means IAM policies alone are sufficient.
If you've modified the key policy, however, the KMS key policy must also grant permission, or you'll get an error.

KMS keys have a key policy that controls "who can use this key."
Permission must be granted in both the IAM policy and the key policy.

Example Key Policy

{
    "Sid": "AllowPresignedUrlRole",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/PresignedUrlGeneratorRole"
    },
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
    ],
    "Resource": "*"
}

Note
Using * for Resource in a KMS key policy is fine.
Since the key policy is attached to the key itself, * refers to that very key.

KMS keys created with default settings include a policy with root as the Principal.
Specifying root delegates the decision to the account's IAM policies, which means IAM policy alone is enough to use the KMS key.
However, if you've removed or modified this statement for security reasons, you'll need to explicitly grant the presigned URL generator's principal in the key policy.

4. PUT Presigned URLs Need SSE Parameters in the Signature

This applies when you don't use bucket default encryption and instead specify the KMS key explicitly in the request.
If the bucket's default encryption is set to SSE-KMS, S3 handles encryption automatically server-side, so this step isn't needed.

When generating a presigned URL for upload, you need to include the SSE-KMS parameters in the signature.

With AWS CLI

The AWS CLI's presign command is designed for GET (download) use cases.
For PUT use cases that need SSE-KMS parameters, it's more reliable to use an SDK.

With AWS SDK (Python / Boto3)

import boto3
from botocore.config import Config

s3_client = boto3.client("s3", config=Config(signature_version="s3v4"))

# For downloads (GET)
download_url = s3_client.generate_presigned_url(
    "get_object",
    Params={
        "Bucket": "my-bucket",
        "Key": "my-file.pdf",
    },
    ExpiresIn=3600,
)

# For uploads (PUT)
upload_url = s3_client.generate_presigned_url(
    "put_object",
    Params={
        "Bucket": "my-bucket",
        "Key": "upload/new-file.pdf",
        "ServerSideEncryption": "aws:kms",
        "SSEKMSKeyId": "arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id",
    },
    ExpiresIn=3600,
)

For PUT presigned URLs, include ServerSideEncryption and SSEKMSKeyId in Params.

⚠️ Note
The headers in the PUT request must match the parameters included in the signature.
When making the request from the client side (curl, fetch, etc.), include the following headers:
curl -X PUT \
  -H "x-amz-server-side-encryption: aws:kms" \
  -H "x-amz-server-side-encryption-aws-kms-key-id: arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id" \
  --upload-file ./new-file.pdf \
  "<presigned-url>"
Missing headers will result in a SignatureDoesNotMatch error.

Note
For multipart uploads (upload_part), SSE parameters are specified at the create_multipart_upload step.
The presigned URLs for individual parts don't need ServerSideEncryption or SSEKMSKeyId.

Troubleshooting Checklist

Here's a checklist for when you hit errors with presigned URLs and SSE-KMS:

#	Check	Target
1	Does the IAM policy include `kms:Decrypt`?	Presigned URL generator
2	Does the IAM policy include `kms:GenerateDataKey` (for PUT)?	Presigned URL generator
3	Is `Config(signature_version="s3v4")` set in Boto3?	Application
4	Does the KMS key policy permit the generator's principal?	KMS key
5	Are SSE parameters included in the signature for PUT (when not using default encryption)?	Application
6	Do the PUT request headers match the signature (when not using default encryption)?	Client
7	Is the presigned URL still within its expiration window?	URL
8	Is the KMS key in an enabled state (not disabled or pending deletion)?	KMS key

Summary

The biggest reason presigned URLs and SSE-KMS trip people up is that switching to SSE-KMS introduces a separate requirement: permissions on the KMS key itself.
On top of that, you also need to be careful about SigV4 and how SSE parameters are handled.

SSE Method	Additional Permission Setup Required
SSE-S3	None
SSE-KMS	IAM policy + KMS key policy

Key takeaways:

When you switch to SSE-KMS, add KMS permissions — kms:Decrypt, plus kms:GenerateDataKey for PUT
In Boto3, explicitly set Config(signature_version="s3v4") — without SigV4, you'll get InvalidArgument
Check the KMS key policy too — IAM policies alone may not be enough
Include SSE parameters in PUT presigned URLs — header mismatches lead to SignatureDoesNotMatch

You're most likely to hit these errors when migrating from SSE-S3 to SSE-KMS, so use this checklist when you do!

Stop Paying Too Much for CloudWatch Logs — Auto-Archive to S3 via Firehose

Yuichi Sato — Tue, 21 Apr 2026 11:58:09 +0000

This article was originally written in Japanese and published on Qiita. It has been translated with the help of AI.
Original article: https://qiita.com/sassssan68/items/da2aa98bba12748daca7

Have you ever calculated how much it actually costs to keep CloudWatch Logs long-term?

TL;DR
Keeping logs in CloudWatch Logs long-term is expensive.
Subscription → Firehose → S3 (Deep Archive) is more stable and cost-effective.

I recently had an audit requirement to retain logs for 18 months. When I estimated the CloudWatch Logs cost…

18 months = $1,069.20
— That's way too much!

So I followed the AWS-recommended architecture — Subscription → Firehose → S3 — and combined it with a lifecycle policy to transition to Deep Archive. The result:

~85% cost reduction
Plus fully automated, stable operations

This article covers:

Why CreateExportTask is not recommended (per AWS)
How much cost you can actually save (with formulas)
How to set up Subscription → Firehose → S3 (Deep Archive)

Why You Shouldn't Keep Logs in CloudWatch Logs Long-Term

Audit and regulatory requirements often mandate log retention for years. However, storing large volumes of logs in CloudWatch Logs gets expensive fast:

High storage cost — CloudWatch Logs storage pricing is heavy
Scales linearly — The more data you store, the worse it gets
Not designed for long-term archival — It's a monitoring tool, not a storage solution

This raises the question: What's the right way to handle long-term log retention?

AWS Says Export Task Is "Not Recommended" — Here's Why

From the CloudWatch console, you can manually export logs to S3. To automate this, you'd use the CreateExportTask API:

https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateExportTask.html

You could call this periodically from Lambda or EventBridge Scheduler. However, the AWS documentation explicitly discourages this:

Note
We recommend that you don't regularly export to Amazon S3 as a way to continuously archive your logs. For that use case, we instead recommend that you use subscriptions. For more information about subscriptions, see Real-time processing of log data with subscriptions.

On top of that, there's a concurrency limit of 1 export task at a time. If you're exporting from multiple log groups or across multiple time ranges, tasks will queue up, causing failures and delays.

Given these limitations, CreateExportTask is unreliable for audit-grade long-term retention. As the AWS docs say, subscriptions are the way to go.

The AWS-Recommended Architecture: Subscription → Firehose → S3

AWS recommends using CloudWatch Logs subscription filters:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html

With Firehose, logs are transferred in near real-time — no manual operations required, stable and suitable for long-term archival.

But when I first saw this architecture, I thought: "This looks expensive. Is it actually cheaper than just leaving logs in CloudWatch?"

So I ran the numbers.

Cost Comparison: CloudWatch Logs vs. Subscription → Firehose → S3 (Deep Archive)

Bottom line:
Over 18 months, Subscription → Firehose → S3 (Deep Archive) is approximately 85% cheaper.

⚠️ Note:
If your retention period is 2 months or less, CloudWatch Logs may actually be cheaper.
The conclusion in this article assumes long-term retention.

Assumptions

Retention period: 18 months
Monthly log volume: 100 GB
Tokyo region pricing (as of April 2026):
- CloudWatch Logs storage: $0.033/GB/month
- Firehose delivery: $0.036/GB
- S3 Standard: $0.025/GB/month
- Glacier Deep Archive: $0.002/GB/month
Case 1: Keep all logs in CloudWatch Logs for the full 18 months
Case 2: Keep logs in CloudWatch for 2 weeks (for analysis), simultaneously stream via Firehose → S3 Standard → Glacier Deep Archive after 1 day

Comparison Table

Item	Case 1	Formula	Case 2	Formula
CloudWatch Logs storage	$1,069.20	0.033 × (100 × 18) × 18	$27.72	0.033 × (100 × 14/30) × 18
Firehose delivery	—	—	$64.80	0.036 × 100 × 18
S3 Standard (1 day)	—	—	$1.50	0.025 × (100 × 1/30) × 18
Glacier Deep Archive	—	—	$64.80	0.002 × (100 × 18) × 18
Total	$1,069.20	—	$158.82	—
Savings	—	—	~$910.38 (~85% reduction)	—

Note:
- Average stored volume for 2-week retention: monthly log volume × (14/30)

Conclusion

By keeping logs in CloudWatch for just 2 weeks (for analysis) and archiving older logs in Deep Archive, you can achieve approximately 85% cost savings over 18 months.

How to Set It Up

The setup involves five steps:

Create an S3 bucket with a lifecycle rule (transition to Glacier Deep Archive after 1 day)
Create a Firehose stream (source: Direct PUT, destination: S3)
Create an IAM role for the subscription filter
Create a CloudWatch Logs subscription filter
Verify logs are flowing to S3

For steps 3 and 4, the AWS documentation provides a complete walkthrough including the IAM policy:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample

Summary

Export Task is not recommended (per AWS documentation)
Firehose is the most operationally practical solution
S3 lifecycle rules enable cost-optimized long-term archival

For short-term log retention, CloudWatch Logs works just fine. But if you need to retain logs for months to years, Subscription → Firehose → S3 (Deep Archive) is the practical solution.

When long-term retention becomes a requirement, it's worth revisiting your architecture.

I hope this helps anyone else dealing with the same log retention cost challenges.

Cryptographic Erasure: A Data Deletion Strategy Using AWS KMS

Yuichi Sato — Tue, 17 Mar 2026 11:57:10 +0000

This article was originally written in Japanese and published on Qiita. It has been translated with the help of AI.
Original article: https://qiita.com/sassssan68/items/a406e971217c1523026b

Are you really sure your data is deleted?

With a shift in mindset and AWS KMS, you can effectively achieve data deletion.

This article covers:

Challenges of traditional data deletion
The concept of Cryptographic Erasure
How to implement Cryptographic Erasure with AWS KMS

Background and Challenges

Scenario 1: User Account Deletion

Imagine you're running a SaaS service and a user requests to close their account. You need to delete all of that user's data.

You might think, "Just run a DELETE statement on the database, right?" But in reality, data exists in many places:

Production database ← You can delete here
Database backups ← But there are 30 days' worth...
Log files ← Stored in S3...
Analytics data warehouse ← Copies in Redshift too...
Dev/staging environments ← Using masked production data...

Deleting everything without missing anything is practically very difficult.

Scenario 2: System Decommissioning / Service Shutdown

When a SaaS service is no longer profitable and you decide to shut it down, you still need to delete all the data you've been handling.

Just like Scenario 1, data is scattered across databases, backups, logs, and more. On top of that, cloud environments automatically replicate data for availability and fault tolerance:

Multi-AZ RDS and DynamoDB
Automated backups and snapshots stored in S3
Cross-region replicas
Data forwarded via CloudWatch Logs or Kinesis to downstream systems

On-premises, you could physically destroy the disks as a last resort, but in the cloud, you can't touch physical storage (Shared Responsibility Model). This makes it even harder to confidently say, "Everything has been deleted."

The Risks of "Thinking You Deleted It"

Lingering data poses risks such as:

Data breaches
- Data you thought was deleted gets restored from backups and leaked
Regulatory violations
- Failure to comply with GDPR's "Right to Be Forgotten" can result in fines
Audit findings
- Being flagged for "no evidence of data deletion"

What Is Cryptographic Erasure?

Destroy the Key, and the Data Becomes Garbage

Cryptographic Erasure is a sanitization method defined in NIST SP 800-88.

https://csrc.nist.gov/pubs/sp/800/88/r2/final

It's essentially a shift in mindset:

Traditional approach
- Deleting data = erasing the data itself
Cryptographic Erasure approach
- Deleting data = making the data impossible to decrypt (by destroying the key)

Encrypted data without its key is nothing more than meaningless bytes.

In other words, if you delete the key, the data is effectively deleted — even if it physically remains.

Traditional Deletion vs. Cryptographic Erasure

Item	Traditional Deletion	Cryptographic Erasure
Database	✅	↓ (Cannot be decrypted)
Backups	❌ (Cannot delete during retention period)	↓ (Cannot be decrypted)
Logs	❌ (Mixed with other logs)	↓ (Cannot be decrypted)
Analytics	△ (Requires manual work)	↓ (Cannot be decrypted)
Encryption key	—	✅

Traditional Deletion
- Complete deletion is difficult
Cryptographic Erasure
- Deleting the key renders all data unrecoverable

Benefits of Cryptographic Erasure

Invalidate all distributed data at once
- Wherever the data exists, destroying the key invalidates it all
Safe even if backups are restored
- Restored backups are unreadable without the key
Auditable evidence
- A record of "when the key was deleted" is preserved

What Is AWS KMS?

AWS KMS (Key Management Service) is a key management service provided by AWS. It allows you to securely create, store, and delete encryption keys.

https://docs.aws.amazon.com/kms/latest/developerguide/overview.html

Key features:

Fully managed
- AWS handles key storage and protection
Audit-ready
- Key usage history is recorded in CloudTrail
Integrated with other AWS services
- Can be used to encrypt S3, RDS, EBS, and more

Types of KMS Keys

AWS KMS offers three types of keys:

Type	Description	Can Be Deleted?
AWS owned keys	Used internally by AWS; invisible to users	❌
AWS managed keys	Automatically created, e.g., `aws/s3`, `aws/rds`	❌
Customer managed keys	Created and managed by the user	✅

Two Types of Customer Managed Keys

Customer managed keys are further divided into two types based on the origin of the key material (the actual cryptographic key data):

Type	Key Material	Immediate Deletion	Characteristics
KMS-generated key	Generated and managed by AWS	❌ (7–30 day waiting period)	Easy to operate
Imported key	Brought in from outside	✅	Useful when immediate deletion is required

Features Required for Cryptographic Erasure

Customer managed keys in AWS KMS have all the features needed for Cryptographic Erasure:

Key creation
- Keys can be created per user or per system
Key management
- Keys can be given aliases for easier identification
Key deletion
- Keys that are no longer needed can be deleted

Envelope Encryption

AWS KMS uses a method called "Envelope Encryption." This is the technical mechanism that makes Cryptographic Erasure work.

Here's how it works:

A KMS key is used to generate a "data key"
The data key is used to encrypt the actual data
The data key itself is encrypted with the KMS key and stored
The plaintext data key is immediately discarded

So only two things are stored: the "encrypted data key" and the "encrypted data." Neither can be decrypted without the original KMS key.

This is the key point:
When a KMS key is deleted, the data key can no longer be decrypted, and consequently, the data itself becomes unrecoverable. This is the mechanism behind "destroy the key, invalidate all the data."

Practical Design

Scenario 1: User Account Deletion

Design Approach: Issue a KMS Key per User

At user registration:
- User A registers → KMS Key A is created
- User B registers → KMS Key B is created
- User C registers → KMS Key C is created
When User A closes their account:
- Schedule deletion of KMS Key A
- After the waiting period (7–30 days), the key is deleted
- User A's data becomes unrecoverable

User B and User C's data remain unaffected.

Benefits of This Design

Only the departing user's data is invalidated
No impact on other users
Easier to comply with GDPR's "Right to Be Forgotten"

Scenario 2: System Decommissioning / Service Shutdown

Design Approach: Issue a KMS Key per System (or Tenant)

At system build:
- System X → KMS Key X is created
- All data for System X is encrypted with KMS Key X
At system decommissioning:
- Schedule deletion of KMS Key X
- After the waiting period (7–30 days), the key is deleted
- All data for System X becomes unrecoverable

Even if backups remain, the data is unreadable after restoration.

Benefits of This Design

All system data can be invalidated at once
Covers tape backups and DR site data as well
Auditable evidence of "complete deletion" is preserved

Comparing the Two Scenarios

Item	User Account Deletion	System Decommissioning
Key granularity	Per user	Per system
Number of keys	Many (one per user)	Few (one per system)
Scope of deletion impact	Single user	Entire system
Cost	High	Low

Implementation Considerations

Cost

AWS KMS incurs the following charges (as of February 2026):

Item	Cost
KMS key storage	~$1/month/key/region
API requests	~$0.03/10,000 requests

If you create a key per user, costs grow with the number of users.

Example: 100,000 users

Key storage alone: ~$100,000/month

Deletion Is Not Immediate (With Exceptions)

KMS key deletion has a mandatory waiting period:

Minimum: 7 days
Maximum: 30 days
Default: 30 days

During this period, deletion can be cancelled. This is a safety feature to prevent accidental deletion, but it cannot meet requirements for "delete everything right now."

Exception: Use Imported Keys When Immediate Deletion Is Required

KMS keys with externally imported key material are the exception.

Running the following command immediately deletes the key material. Unless it's re-imported, the data is permanently unrecoverable:

# Immediately delete the key material
aws kms delete-imported-key-material --key-id <your-key-id>

⚠️ Warning
Imported keys require generating, managing, and importing key material externally, which adds operational complexity. Carefully evaluate whether immediate deletion is truly necessary.

Design Early

Retrofitting Cryptographic Erasure into an existing system is difficult. Applying it to existing systems may require re-encrypting all data.

Use Customer Managed Keys

When you enable encryption for S3 or RDS, AWS managed keys may be used by default. AWS managed keys (aws/s3, aws/rds, etc.) cannot be deleted. Since you can't delete them, you can't perform Cryptographic Erasure — so always specify a customer managed key.

Here are the default encryption settings for major services and what's needed for Cryptographic Erasure:

Service	Default Encryption	For Cryptographic Erasure
S3	SSE-S3 or aws/s3	Configure SSE-KMS with a customer managed key (also enable S3 Bucket Key to reduce KMS API costs)
RDS	aws/rds	Create with a customer managed key specified
EBS	aws/ebs	Create with a customer managed key specified

Summary

Here are the key takeaways about Cryptographic Erasure:

Shift your mindset from "delete the data" to "delete the key"
Invalidate distributed data all at once
AWS KMS makes it achievable in a fully managed way
Applicable to both user account deletions and system decommissioning
Cost and design require upfront planning
Cryptographic Erasure requires customer managed keys

Lingering data poses risks such as data breaches and regulatory violations. Let's address them properly with Cryptographic Erasure!