DEV Community

AWS Community Builders profile image Scott Burgholzer
Scott Burgholzer for AWS Community Builders

Posted on • Originally published at blog.scottburgholzer.tech

The InvalidCiphertextException Mystery: Decrypting Cognito's Encrypted OTP Codes

#aws #encryption #lambda #kms

The Problem

You're building a custom email sender for Cognito because the default emails are pretty basic. In this case you want branded HTML emails for a One Time Password code that actually explain what the code is for and when it expires. Sounds straightforward, right?

Then you hit the encryption wall. Cognito encrypts those OTP codes for security (which is great), but when you try to decrypt them in your Lambda function using standard KMS operations, you get this error:

InvalidCiphertextException: An error occurred (InvalidCiphertextException) when calling the Decrypt operation:
Enter fullscreen mode Exit fullscreen mode

That's it. No helpful details, no explanation of what went wrong. The error message makes it seem like there's something fundamentally wrong with the ciphertext, but even after triple-checking your KMS permissions, it still doesn't work. Here's why, and more importantly, how to fix it.

TL;DR - The Solution

Cognito doesn't use regular KMS encryption. It uses the AWS Encryption SDK, which creates a completely different data structure. You need to use the same AWS Encryption SDK (v4 with Material Providers Library) to decrypt the codes. Jump to the working code if you just want the fix.

What Everyone Tries First, Including Myself (And Why It Fails)

When you think AWS encryption, you think KMS. So naturally, you try the standard approaches:

Attempt 1: Basic KMS Decryption 

# This seems logical but doesn't work
kms_client = boto3.client('kms')
decrypt_response = kms_client.decrypt(CiphertextBlob=encrypted_blob)
Enter fullscreen mode Exit fullscreen mode

Result: InvalidCiphertextException

Attempt 2: Adding Encryption Context

Maybe it needs the user pool ID as context?

# Still doesn't work
decrypt_response = kms_client.decrypt(
    CiphertextBlob=encrypted_blob,
    EncryptionContext={"userpool-id": event.get('userPoolId', '')}
)
Enter fullscreen mode Exit fullscreen mode

Result: Same error

Attempt 3: AWS-Prefixed Context

Perhaps it needs the AWS service prefix?

# Nope, still fails
decrypt_response = kms_client.decrypt(
    CiphertextBlob=encrypted_blob,
    EncryptionContext={"aws:cognito:userpool-id": event.get('userPoolId', '')}
)
Enter fullscreen mode Exit fullscreen mode

Result: Still the same error

At this point, you start questioning everything. Your KMS permissions look right, the key exists, but nothing works.

The Real Issue: It's Not Actually KMS Ciphertext

Here's the key insight that changes everything: Cognito doesn't use direct KMS encryption. Instead, it uses something called the AWS Encryption SDK, which creates a completely different type of encrypted data.

What's the Difference?

Think of it this way:

  • Regular KMS encryption = A locked box
  • AWS Encryption SDK = A locked box inside a shipping container with labels, tracking info, and handling instructions

When you try to use KMS to "unlock" the shipping container, KMS says "I don't know what this is - this isn't a box I locked!"

The Technical Details

When Cognito encrypts an OTP code, here's what actually happens:

  1. Cognito uses the AWS Encryption SDK (not direct KMS)
  2. The SDK creates an "envelope" containing:
    • Algorithm information
    • An encrypted data key (this part uses KMS)
    • The actual encrypted OTP code
    • Integrity checks and metadata
  3. This entire envelope gets passed to your Lambda function

The encrypted_blob you receive isn't simple KMS ciphertext - it's this complex envelope structure. That's why KMS can't decrypt it directly.

Why the Error Message is Confusing

The InvalidCiphertextException error just says:

An error occurred (InvalidCiphertextException) when calling the Decrypt operation:
Enter fullscreen mode Exit fullscreen mode

That's it. No details, no explanation. This generic message makes you think the ciphertext is corrupted or you have permission issues, but the real problem is that KMS is saying "I can't even parse this data structure."

It's like trying to open a ZIP file with a text editor - the format is just wrong, but the error message doesn't tell you that.

Solution: AWS Encryption SDK Implementation

Once you understand the problem, the solution becomes clear. You need to use the same AWS Encryption SDK that Cognito uses to create the encrypted data.

What You Need

  1. AWS Encryption SDK v4 - The encryption library
  2. Material Providers Library (MPL) - Required for v4
  3. Proper KMS permissions - Your Lambda still needs to access the KMS key
  4. The right Python packages - Install them as a Lambda layer

The Working Code

Here's the code that actually works:

import os
import boto3
import aws_encryption_sdk
from aws_encryption_sdk import CommitmentPolicy
from aws_cryptographic_material_providers.mpl import AwsCryptographicMaterialProviders
from aws_cryptographic_material_providers.mpl.config import MaterialProvidersConfig
from aws_cryptographic_material_providers.mpl.models import CreateAwsKmsKeyringInput

def decrypt_cognito_code(encrypted_blob):
    # Set up the encryption client
    client = aws_encryption_sdk.EncryptionSDKClient(
        commitment_policy=CommitmentPolicy.REQUIRE_ENCRYPT_ALLOW_DECRYPT
    )

    # Create the material providers
    mat_prov = AwsCryptographicMaterialProviders(
        config=MaterialProvidersConfig()
    )

    # Set up the KMS keyring
    kms_key_id = os.environ.get('KMS_KEY_ID')
    keyring_input = CreateAwsKmsKeyringInput(
        kms_key_id=kms_key_id,
        kms_client=boto3.client('kms')
    )

    kms_keyring = mat_prov.create_aws_kms_keyring(input=keyring_input)

    # Now this actually works!
    plaintext_bytes, decryption_header = client.decrypt(
        source=encrypted_blob,
        keyring=kms_keyring
    )

    # Convert to string and return
    return plaintext_bytes.decode('utf-8')
Enter fullscreen mode Exit fullscreen mode

Setting It Up

1. Install the packages:
Create a Lambda layer with these dependencies:

pip install "aws-encryption-sdk[MPL]">=4.0.0
Enter fullscreen mode Exit fullscreen mode

2. Set your environment variable:

KMS_KEY_ID=your-kms-key-id-here
Enter fullscreen mode Exit fullscreen mode

3. Update your IAM role:
Your Lambda execution role needs kms:Decrypt permission for your KMS key:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kms:Decrypt",
            "Resource": "arn:aws:kms:region:account:key/your-key-id"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Why This Matters Beyond Cognito

This isn't just a Cognito quirk. Other AWS services also use the AWS Encryption SDK instead of direct KMS encryption:

  • Systems Manager Parameter Store (SecureString parameters)
  • Some S3 client-side encryption scenarios
  • Custom applications that need to encrypt large amounts of data

The pattern is becoming more common because the AWS Encryption SDK offers benefits that direct KMS doesn't:

  • No size limits (KMS is limited to 4KB)
  • Better performance for large data
  • Additional security features like key commitment

The Key Takeaway

When you see an AWS service that says it "uses KMS encryption," don't assume it's using direct KMS calls. It might be using the AWS Encryption SDK under the hood, which changes everything about how you decrypt the data.

Troubleshooting Tips

If you're still getting errors:

  1. Check your Lambda layer - Make sure you have the right packages installed
  2. Verify your KMS key ID - It should be the same one configured in your Cognito user pool
  3. Test your IAM permissions - Try a simple KMS decrypt operation to verify access
  4. Check the commitment policy - Use REQUIRE_ENCRYPT_ALLOW_DECRYPT for compatibility

Common mistakes:

  • Using the wrong version of the AWS Encryption SDK (you need v4)
  • Forgetting to install the Material Providers Library
  • Using the wrong KMS key (it must match what Cognito is configured to use)

Error message reference:

  • InvalidCiphertextException with no details = You're trying to decrypt AWS Encryption SDK data with direct KMS
  • IncorrectKeyException = You're using the wrong KMS key for real KMS ciphertext
  • AccessDeniedException = Actual permissions problem

Final Thoughts

This problem can stump a lot of developers because the error message is so generic and unhelpful. When you see InvalidCiphertextException with no details, it's usually a sign that you're trying to decrypt AWS Encryption SDK data with direct KMS calls.

Once you understand that Cognito uses the AWS Encryption SDK instead of direct KMS, everything clicks into place. The extra complexity is worth it - you get better security, no size limits, and future-proof encryption. Plus, now you know how to handle this pattern when you encounter it in other AWS services.

References

Top comments (0)