DEV Community: Scott Burgholzer

The InvalidCiphertextException Mystery: Decrypting Cognito's Encrypted OTP Codes

Scott Burgholzer — Thu, 01 Jan 2026 05:05:01 +0000

The Problem

You're building a custom email sender for Cognito because the default emails are pretty basic. In this case you want branded HTML emails for a One Time Password code that actually explain what the code is for and when it expires. Sounds straightforward, right?

Then you hit the encryption wall. Cognito encrypts those OTP codes for security (which is great), but when you try to decrypt them in your Lambda function using standard KMS operations, you get this error:

InvalidCiphertextException: An error occurred (InvalidCiphertextException) when calling the Decrypt operation:

That's it. No helpful details, no explanation of what went wrong. The error message makes it seem like there's something fundamentally wrong with the ciphertext, but even after triple-checking your KMS permissions, it still doesn't work. Here's why, and more importantly, how to fix it.

TL;DR - The Solution

Cognito doesn't use regular KMS encryption. It uses the AWS Encryption SDK, which creates a completely different data structure. You need to use the same AWS Encryption SDK (v4 with Material Providers Library) to decrypt the codes. Jump to the working code if you just want the fix.

What Everyone Tries First, Including Myself (And Why It Fails)

When you think AWS encryption, you think KMS. So naturally, you try the standard approaches:

Attempt 1: Basic KMS Decryption

# This seems logical but doesn't work
kms_client = boto3.client('kms')
decrypt_response = kms_client.decrypt(CiphertextBlob=encrypted_blob)

Result: InvalidCiphertextException

Attempt 2: Adding Encryption Context

Maybe it needs the user pool ID as context?

# Still doesn't work
decrypt_response = kms_client.decrypt(
    CiphertextBlob=encrypted_blob,
    EncryptionContext={"userpool-id": event.get('userPoolId', '')}
)

Result: Same error

Attempt 3: AWS-Prefixed Context

Perhaps it needs the AWS service prefix?

# Nope, still fails
decrypt_response = kms_client.decrypt(
    CiphertextBlob=encrypted_blob,
    EncryptionContext={"aws:cognito:userpool-id": event.get('userPoolId', '')}
)

Result: Still the same error

At this point, you start questioning everything. Your KMS permissions look right, the key exists, but nothing works.

The Real Issue: It's Not Actually KMS Ciphertext

Here's the key insight that changes everything: Cognito doesn't use direct KMS encryption. Instead, it uses something called the AWS Encryption SDK, which creates a completely different type of encrypted data.

What's the Difference?

Think of it this way:

Regular KMS encryption = A locked box
AWS Encryption SDK = A locked box inside a shipping container with labels, tracking info, and handling instructions

When you try to use KMS to "unlock" the shipping container, KMS says "I don't know what this is - this isn't a box I locked!"

The Technical Details

When Cognito encrypts an OTP code, here's what actually happens:

Cognito uses the AWS Encryption SDK (not direct KMS)
The SDK creates an "envelope" containing:
- Algorithm information
- An encrypted data key (this part uses KMS)
- The actual encrypted OTP code
- Integrity checks and metadata
This entire envelope gets passed to your Lambda function

The encrypted_blob you receive isn't simple KMS ciphertext - it's this complex envelope structure. That's why KMS can't decrypt it directly.

Why the Error Message is Confusing

The InvalidCiphertextException error just says:

An error occurred (InvalidCiphertextException) when calling the Decrypt operation:

That's it. No details, no explanation. This generic message makes you think the ciphertext is corrupted or you have permission issues, but the real problem is that KMS is saying "I can't even parse this data structure."

It's like trying to open a ZIP file with a text editor - the format is just wrong, but the error message doesn't tell you that.

Solution: AWS Encryption SDK Implementation

Once you understand the problem, the solution becomes clear. You need to use the same AWS Encryption SDK that Cognito uses to create the encrypted data.

What You Need

AWS Encryption SDK v4 - The encryption library
Material Providers Library (MPL) - Required for v4
Proper KMS permissions - Your Lambda still needs to access the KMS key
The right Python packages - Install them as a Lambda layer

The Working Code

Here's the code that actually works:

import os
import boto3
import aws_encryption_sdk
from aws_encryption_sdk import CommitmentPolicy
from aws_cryptographic_material_providers.mpl import AwsCryptographicMaterialProviders
from aws_cryptographic_material_providers.mpl.config import MaterialProvidersConfig
from aws_cryptographic_material_providers.mpl.models import CreateAwsKmsKeyringInput

def decrypt_cognito_code(encrypted_blob):
    # Set up the encryption client
    client = aws_encryption_sdk.EncryptionSDKClient(
        commitment_policy=CommitmentPolicy.REQUIRE_ENCRYPT_ALLOW_DECRYPT
    )

    # Create the material providers
    mat_prov = AwsCryptographicMaterialProviders(
        config=MaterialProvidersConfig()
    )

    # Set up the KMS keyring
    kms_key_id = os.environ.get('KMS_KEY_ID')
    keyring_input = CreateAwsKmsKeyringInput(
        kms_key_id=kms_key_id,
        kms_client=boto3.client('kms')
    )

    kms_keyring = mat_prov.create_aws_kms_keyring(input=keyring_input)

    # Now this actually works!
    plaintext_bytes, decryption_header = client.decrypt(
        source=encrypted_blob,
        keyring=kms_keyring
    )

    # Convert to string and return
    return plaintext_bytes.decode('utf-8')

Setting It Up

1. Install the packages:
Create a Lambda layer with these dependencies:

pip install "aws-encryption-sdk[MPL]">=4.0.0

2. Set your environment variable:

KMS_KEY_ID=your-kms-key-id-here

3. Update your IAM role:
Your Lambda execution role needs kms:Decrypt permission for your KMS key:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kms:Decrypt",
            "Resource": "arn:aws:kms:region:account:key/your-key-id"
        }
    ]
}

Why This Matters Beyond Cognito

This isn't just a Cognito quirk. Other AWS services also use the AWS Encryption SDK instead of direct KMS encryption:

Systems Manager Parameter Store (SecureString parameters)
Some S3 client-side encryption scenarios
Custom applications that need to encrypt large amounts of data

The pattern is becoming more common because the AWS Encryption SDK offers benefits that direct KMS doesn't:

No size limits (KMS is limited to 4KB)
Better performance for large data
Additional security features like key commitment

The Key Takeaway

When you see an AWS service that says it "uses KMS encryption," don't assume it's using direct KMS calls. It might be using the AWS Encryption SDK under the hood, which changes everything about how you decrypt the data.

Troubleshooting Tips

If you're still getting errors:

Check your Lambda layer - Make sure you have the right packages installed
Verify your KMS key ID - It should be the same one configured in your Cognito user pool
Test your IAM permissions - Try a simple KMS decrypt operation to verify access
Check the commitment policy - Use REQUIRE_ENCRYPT_ALLOW_DECRYPT for compatibility

Common mistakes:

Using the wrong version of the AWS Encryption SDK (you need v4)
Forgetting to install the Material Providers Library
Using the wrong KMS key (it must match what Cognito is configured to use)

Error message reference:

InvalidCiphertextException with no details = You're trying to decrypt AWS Encryption SDK data with direct KMS
IncorrectKeyException = You're using the wrong KMS key for real KMS ciphertext
AccessDeniedException = Actual permissions problem

Final Thoughts

This problem can stump a lot of developers because the error message is so generic and unhelpful. When you see InvalidCiphertextException with no details, it's usually a sign that you're trying to decrypt AWS Encryption SDK data with direct KMS calls.

Once you understand that Cognito uses the AWS Encryption SDK instead of direct KMS, everything clicks into place. The extra complexity is worth it - you get better security, no size limits, and future-proof encryption. Plus, now you know how to handle this pattern when you encounter it in other AWS services.

References

Understanding Amazon CloudFront's New Flat-Rate Pricing

Scott Burgholzer — Thu, 27 Nov 2025 00:45:35 +0000

On November 18th, AWS introduced new flat‑rate pricing plans for Amazon CloudFront designed to make content delivery and security costs more predictable for teams of all sizes. These plans sit alongside the existing pay‑as‑you‑go model and bundle multiple services into a single monthly price per distribution.

Why Flat-Rate Pricing Matters

Traditionally, CloudFront has used pay‑as‑you‑go pricing, which is great for starting at $0, scaling with actual usage, and only paying for what you consume. The tradeoff is that estimating costs can be difficult, especially when you also depend on AWS WAF, DDoS protection, Route 53, CloudWatch Logs, and S3 for a single application. You end up stitching together multiple pricing pages and trying to map them to your traffic patterns just to get a reasonable forecast of your monthly bill.

The new flat‑rate plans aim to simplify this. Instead of tracking every request, rule, and log line, you choose a plan tier per distribution and pay a fixed monthly fee that includes a defined bundle of features and usage allowances.

What's Included in the New Plans

Each flat‑rate plan bundles the following into one monthly price per CloudFront distribution:

Amazon CloudFront CDN
AWS WAF and DDoS protection
Bot management and analytics
Amazon Route 53 DNS
Amazon CloudWatch Logs ingestion
Serverless edge compute (via CloudFront Functions)
Monthly Amazon S3 storage credits

Because everything is packaged together, you get a predictable bill for that distribution with no overage charges. There are no surprises at the end of the month.

A few structural details to keep in mind:

Plans are per distribution. If you put two distributions on the Business plan at $200 each, you will pay $400 per month for those two.
You can upgrade at any time, but downgrades/cancellations or a move back to pay‑as‑you‑go only take effect at the start of the next billing cycle.
There is a limit of three free plans per AWS account and a limit of 100 total plans per account, which matters if you run many environments or tenants.

Handling Attacks and Traffic Spikes

One of the more reassuring aspects of these plans is how they treat malicious traffic. DDoS attacks do not count against your usage allowance, and traffic blocked by AWS WAF or bot protections also does not consume your plan quota. That means an attack should not cause you to exceed your allowance.

You will receive email notifications when you hit 50%, 80%, and 100% of your usage allowance for the month. If you exceed the allowance, AWS will not tack on overage fees, but you may experience reduced performance. That performance reduction is at AWS's discretion and depends on overall network conditions and other factors, so consistently bumping into your limits is a good signal that it is time to move to a higher‑tier plan.

Edge Compute: CloudFront Functions vs. Lambda@Edge

All of the flat‑rate plans include CloudFront Functions, which are ideal for lightweight JavaScript logic at the edge such as header manipulation, redirects, and simple request or response normalization. For many use cases, this will cover the bulk of what you might previously have used Lambda@Edge for.

However, Lambda@Edge is not supported on the flat‑rate plans. If your distribution relies on Lambda@Edge, such as more complex request processing, integrations, or heavy transformations, you must keep that distribution on the pay‑as‑you‑go pricing model. This is an important constraint to verify before flipping an existing production distribution over to a flat‑rate plan.

Pay-As-You-Go Is Still Available

All of this arrives in addition to, not instead of, the existing pay‑as‑you‑go model. The traditional CloudFront pricing structure remains available with its free tier, which is still a solid option for:

Very low‑traffic or experimental workloads
Architectures that depend on Lambda@Edge or other features not supported in flat‑rate plans
Teams that prefer pure usage‑based billing and are comfortable modeling and monitoring their costs

In practice, many organizations will likely end up with a mix: flat‑rate plans for high‑volume, business‑critical sites where predictability matters, and pay‑as‑you‑go for specialized or experimental distributions.

What is AWS Device Farm and Using it to Test a Simple Android App

Scott Burgholzer — Sat, 23 Nov 2024 04:37:15 +0000

Introduction

As I have been busy with my classes for my doctorate and work, I haven’t had much time to write articles like I had hoped. I was hoping to have a project done that I could discuss how I created a Web App using multiple AWS services for others to learn how they all can interact. As that project is still ongoing, I decided to focus on something else.

I choose AWS Device Farm because eventually I would need to test that web app on mobile devices and on desktop browsers. Device Farm allows us to use physical devices to test our apps. Not only can we test web apps, we can also test Android and iOS apps on physical devices. There is a lot to learn for Device Farm, so I started very basic with using it, and hence this article was born. I hope, while this article just uses a simple app and the defaults, it can expose more people to this AWS service and see if it’ll work in their use cases.

View this article as a “I’ve never heard or seen AWS Device Farm, so I want to start from the basics and work my way up” as that is exactly what I’m doing with you!

What is AWS Device Farm?

AWS Device farm is a cloud-based testing platform that allows developers to test their mobile and web apps on physical devices instead of simulators. There are two main ways of using Device Farm: automated testing of apps and remote access of devices [1].

Automated app testing

AWS Device Farm allows developers the ability to upload their own tests or to use built-in and script-free compatibility. AWS Device Farm performs testing in parallel, tests on multiple devices begin in minutes. As these tests are completed, a report is generated that contains high-level results, low-level logs, pixel-to-pixel screenshots and performance data. Device Farm supports the testing of native and hybrid Android and iOS Apps. Some frameworks that work with Device Farm are PhoneGap, Titanium, Xamarin, Unity, and others. You also get remote access of Android and iOS apps for interactive testing.

Remote access interaction

This allows you to swipe, gesture and interact with a device through your web browser in real time. An example of where this is beneficial is a customer service rep can guide customers through the use or setup of their device. You have the ability to install apps on a device running in a remote access session and reproduce bugs or other issues. While the session is live, Device Farm will collect details about the actions that are completed on the device. Logs with those details and a video capture of the session are available at the end of the session [1].

Note about regions: Per the documentation, Device Farm is only available in the us-west-2 (Oregon) region [1].

Walk-through

I’m going to skip over the setting up section. If you wish to go through it in detail (signing up for an account, create or use an IAM user) you may read that here. Below is an example IAM policy [1] to allow full access to Device Farm. For this walk-through, this is fine, but if you are going to implement this service, you are going to want to restrict it as much as you can to the least privilege access.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "devicefarm:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Device Farm Console

Step 1: Create a project

Go to Device Farm in the console
Expand the left hand navigation if necessary
Click on Projects underneath Mobile Device Testing
Click on Create mobile project
Enter a project name and assign any tags you wish the project to have
Click Create

Step 2: Create and start a run

You will need an app in order to start the run. I have created a very simple Android App on my GitHub. Go ahead and download that if you do not have your own Android or Apple App (note in order to get an .ipa file for iOS you will need to have an Apple Developer Account.
Stay on the Automated tests page and click on Create Run
Device Farm uses the app name as the default run name, feel free to choose your own name! Click Next
You’ll be taken to a configure page to setup your test framework. You have the ability to choose one of the testing frameworks provided or built-in test suites. Since we are not worrying about our own tests for this example, we can use the Built-In: Fuzz for our case (can keep all the defaults). When you start writing your own test packages, you can choose the corresponding testing framework and upload the file with your tests. This is not applicable in our example. Click Next
For our example, we will use the Top Devices Device pool. You do have the ability to create your own device pools, but for our simple app, Top Devices is just fine. Click Next
We are going to keep the next page as is. You can read more about the options in the Developer Guide, which you can find at the end of this article tagged [1]. Click Next
We will leave the execution timeout to the default value provided and will then click Confirm and start run. This can take a while, so wait patiently.

Run Results

Step 3: View the run’s results

— When the run is done, you will see the results of the runs. In my case 3 tests were ran and all passed.

— The unique problems would list any problems that were discovered. A video can be downloaded to view the recording of the test that had the problem. There are additional data provided that I won’t get into detail in this article.

— Screenshots will display any screenshots Device Farm took during the run.

Where to go from here?

The next thing I personally would suggest, and plan on doing, is to learn more about test packages. There are frameworks that work with both iOS and Android, and some that only work for only Android or only iOS. This is where you are able to specify your tests the way you want them instead of using just the built-in tests AWS provides. This gives you complete control over how you want the tests done and what you want tested. This is where the real power of AWS Device Farm comes into play.

The other thing to look into is the Desktop Browser Testing for any web apps. Personally for me, my project is going to start as just a web app, so Desktop Browser Testing is of personal importance. As always, you need to figure out your use cases and determine what your needs are.

There is so much more to learn about AWS Device Farm, and I will continue posting about it as I go through it and learn it myself. For now, happy developing and testing!

[1] AWS Device Farm Developer Guide

Launching a simple EC2 Instance with Apache Web Server — Part 1

Scott Burgholzer — Mon, 11 Mar 2024 02:06:15 +0000

Little backstory: I’ve had this in my drafts for a while, wanting to do it all as one article. I’ve decided that is not the best way of presenting this information. As a newer content creator, I’ve been playing with how I want my pages to look like, and have decided to go with how it looks right now.

Introduction

In this article, I will be doing what some call “click-ops” in AWS console as that is the easiest way to get started. If you want more challenges, I will also be posting on how to use AWS CLI (an article post on this will come at a later date, when completed, I will be updating this with a link to it!) to do the same thing. I will also show how to use a Cloud Development Kit (CDK). I will be using Python in my CDK examples. I will also use AWS CloudFormation. Finally, for an even more advanced challenge, I will be creating this process again using Terraform. If you are new, the console way will be the easiest, but I also want to introduce you to the CLI, CDK, CloudFormation and Terraform as these are advanced tools used!

Scope of this tutorial

In this multi-part tutorial, we are wanting to set up an EC2 instance, configure security group(s), create a key-pair SSH Key to remote into the instance, and install Apache Web Server.

Method 1: Using AWS Console (without using userdata)

There are actually two ways we can use the AWS console to create the EC2 instance, and install Apache and run it and create the test HTML file. This method will require us to use SSH to remote into our EC2 instance to preform some steps.

Step 1: When you are logged into the AWS Console, search for EC2 in the search bar.

Step 2: Click on Launch Instance

Step 3: Configure your instance and install Apache and test web page

Give the instance a name, I called mine “Web Server Test” * Keep the application and OS Images as is
Keep the instance type as t2.micro
Click on new key par and give it a name, then click on Create key pair. It’ll download a file, move it to a safe location so you don’t loose it!

Under Network settings we will be keeping the selection Create Security Group. Allow SSH traffic from should be checked, and you’ll also want to check the Allow HTTP traffic from the internet. Next to SSH there is a dropdown menu that you can say any IP can connect to the instance, a custom IP or range, or your IP only. For this example, I’m keeping it as anywhere

We will leave everything else as default
Click on Launch Instance

Click on the instance ID shown in the green success box

Click the checkbox next to the instance, then go to actions and click on Connect

You’ll see there are four options of connecting. The only two that will work with our instance, due to the way we set it up, is EC2 Instance Connect and SSH client. If on Mac or Linux you can use command line to SSH into the instance’s command line, on Windows it is easiest to use a tool such as Putty.

We will be using EC2 Instance connect in our case, so click on that tab, then click on connect
After waiting a few moments, you’ll see a command line terminal for our instance.

We now want to install Apache and start the service. Run the following commands in the command prompt.

sudo yum update -y # We are asking the packacge manager to install any updates
sudo yum install httpd -y # We are asking the package manager to install Apache
sudo systemctl enable httpd # We are telling the server to automatically start Apache upon start up
sudo systemctl start httpd # we are telling the server to start the Apache Server
sudo systemctl status httpd # we are telling the server to tell us the status of the Apache Server

We are going to create a simple HTML file, run the below command to open a new file using a command line text editor

sudo nano /var/www/html/index.html

Put the following code into the text editor

<!DOCTYPE html>
<html>
  <head>
    <title>Test Web Page</title>
  </head>
  <body>
    <h1>Welcome to my website!</h1>
    <p>I love AWS</p>
  </body>
</html>

When you are done with the file, press ctl and x, it’ll prompt you if you want to save, type Y, press enter, then press enter again.
Go back to the EC2 Instances page, click on your instance, then click on open address under the Public IPv4 address, change the HTTPS to HTTP

If you see your test page, you were successful!

Step 4: Clean Up

Close all windows except for the Instances window.
Select it, then click on Instance state then click on Terminate instance so you don’t use up your free trial credits or get charged!

Congrats! You have just launched a simple EC2 Web server!

Launching a simple EC2 Instance with Apache Web Server — Part 1 (AWS Console)

Launching a simple EC2 Instance with Apache Web Server — Part 2 (AWS Console with Userdata) (Coming Soon)

Launching a simple EC2 Instance with Apache Web Server — Part 3 (AWS CLI with user data) (Coming Soon)

Launching a simple EC2 Instance with Apache Web Server — Part 4 (Terraform with user data) (Coming Soon)