DEV Community

Cover image for Proactively Securing AWS: Automated Alarms for Pending Deletion KMS Keys and Secrets Using Terraform
Alexey Vidanov for AWS Community Builders

Posted on • Edited on • Originally published at tecracer.com

3 1

Proactively Securing AWS: Automated Alarms for Pending Deletion KMS Keys and Secrets Using Terraform

In cloud computing, security is a critical concern. While AWS provides backup solutions for many resources, custom configurations often require additional protection. Two key services, AWS Key Management Service (KMS) and AWS Secrets Manager, don't offer direct backup options. However, they implement a deletion grace period— by default 30 days and this is the maximum — allowing for potential restoration.

In cloud computing, security is a critical concern. While AWS provides backup solutions for many resources, custom configurations often require additional protection. Two key services, AWS Key Management Service (KMS) and AWS Secrets Manager, don't offer direct backup options. However, they implement a deletion grace period— by default 30 days and this is the maximum — allowing for potential restoration.

A crucial question arises: How can you be notified of a pending deletion? This is where alarms play a vital role. Implementing alarms for AWS KMS keys and AWS Secrets Manager secrets that are pending deletion can significantly enhance your security posture.

This blog post demonstrates how to create these essential alarms using Terraform, an infrastructure-as-code tool. By leveraging Terraform, you can automate the process of setting up notifications for potential security risks, ensuring that your sensitive information and encryption keys remain protected.

We'll walk through the process of configuring alarms that alert you when KMS keys or Secrets Manager secrets are scheduled for deletion, giving you time to review and potentially prevent unintended data loss or security breaches.

Prerequisites

Before we begin, ensure you have:

  • An AWS account with appropriate permissions
  • Terraform installed and configured
  • Basic knowledge of AWS services, particularly CloudTrail, KMS, Secrets Manager, and CloudWatch
  • Familiarity with Terraform syntax and AWS provider

Step-by-Step Guide

1. Setting Up Terraform Variables

First, define the necessary variables in your terraform.tfvars or within your Terraform configuration. These variables include the notification email, the name of the CloudWatch log group attached to CloudTrail logs, and the AWS region.

variable "notification_email"
{
    description = "The email address to notify in case of an alarm"
  type        = string
}

variable "cloud_trail_cloudwatch_log_group" {
  description = "The name of the CloudWatch log group attached to CloudTrail logs"
  type        = string
}

variable "region" {
  description = "The AWS region"
  default     = "eu-central-1"
  type        = string
}
Enter fullscreen mode Exit fullscreen mode

2. Configuring the AWS Provider and Resources

Ensure your AWS provider is configured with the specified region. We also need the information for the AWS CloudWatch log group to create the resources.

provider "aws" {
  region = var.region
}

data "aws_cloudwatch_log_group" "this" {
  name = var.cloud_trail_cloudwatch_log_group
}
Enter fullscreen mode Exit fullscreen mode

Create an IAM role and policy that allows CloudTrail to interact with CloudWatch logs. Additionally, set up an SNS topic for sending notifications and subscribe it to the provided email.

resource "aws_iam_role" "this" {
  name = "CloudTrailCloudWatchLogsRole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "cloudtrail.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cloudtrail_cloudwatch_logs" {
  role       = aws_iam_role.this.name
  policy_arn = aws_iam_policy.this.arn
}

# SNS Topic for Notifications

resource "aws_sns_topic" "pending_deletion_topic" {
  name = "kms-key-pending-deletion-topic"
}

# Subscription to send notifications to an email

resource "aws_sns_topic_subscription" "email_subscription" {
  topic_arn = aws_sns_topic.pending_deletion_topic.arn
  protocol  = "email"
  endpoint  = var.notification_email
}
Enter fullscreen mode Exit fullscreen mode

3. Detecting Deletion Attempts with CloudWatch Log Metric Filters

Use CloudWatch log metric filters to detect deletion attempts or unauthorized access. For AWS Secrets Manager, filter events where a pending deletion secret is accessed. For AWS KMS, filter events where a pending deletion key is used.

# Metric filter to detect the attempt to access secrets marked for deletion

resource "aws_cloudwatch_log_metric_filter" "secrets_deletion_attempts_filter" {
  name           = "SecretsDeletionAttempts"
  pattern        = "{ $.eventName = \"GetSecretValue\" && $.errorMessage = \"*secret because it was marked for deletion*\" }"
  log_group_name = data.aws_cloudwatch_log_group.this.name
  metric_transformation {
    name      = "SecretsDeletionAttempts"
    namespace = "SecretsManagerMetrics"
    value     = "1"
  }
}
Enter fullscreen mode Exit fullscreen mode

If you want to receive notifications not only for attempts to access secret values but also immediately after an attempt to delete secrets, you can change the pattern to $.eventName = \"DeleteSecret\" instead of $.eventName = \"GetSecretValue\"

# Metric filter to detect the attempt to access MS keys marked for deletion

resource "aws_cloudwatch_log_metric_filter" "kms_key_pending_deletion_filter" {
  name           = "KMSKeyPendingDeletion"
  pattern        = "{ $.eventSource = kms* && $.errorMessage = \"* is pending deletion.\"}"
  log_group_name = data.aws_cloudwatch_log_group.this.name
  metric_transformation {
    name      = "KMSKeyPendingDeletionEvents"
    namespace = "CloudTrailMetrics"
    value     = "1"
  }
}
Enter fullscreen mode Exit fullscreen mode

If you want to receive notifications not only for attempts to access keys but also immediately after an attempt to delete keys, you can change the pattern to $.eventSource = kms* instead of $.eventSource = ScheduleKeyDeletion*

4. Creating CloudWatch Alarms

Finally, create CloudWatch alarms based on the metric filters. These alarms will trigger when an attempt is made to access a secret or use a pending deletion KMS key.

# CloudWatch Alarm for KMS Key Pending Deletion

resource "aws_cloudwatch_metric_alarm" "kms_key_pending_deletion_alarm" {
  alarm_name          = "KMSKeyPendingDeletionAlarm"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = aws_cloudwatch_log_metric_filter.kms_key_pending_deletion_filter.metric_transformation[0].name
  namespace           = aws_cloudwatch_log_metric_filter.kms_key_pending_deletion_filter.metric_transformation[0].namespace
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alarm when KMS key pending deletion is used."
  alarm_actions       = [aws_sns_topic.pending_deletion_topic.arn]
  treat_missing_data  = "notBreaching"
}

# CloudWatch Alarm for Secrets Manager Secrets Deletion Attempts

resource "aws_cloudwatch_metric_alarm" "secrets_deletion_attempts_alarm" {
  alarm_name          = "SecretsDeletionAttemptsAlarm"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = aws_cloudwatch_log_metric_filter.secrets_deletion_attempts_filter.metric_transformation[0].name
  namespace           = aws_cloudwatch_log_metric_filter.secrets_deletion_attempts_filter.metric_transformation[0].namespace
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alarm when an attempt to delete an AWS Secrets Manager secret is made."
  alarm_actions       = [aws_sns_topic.pending_deletion_topic.arn]
  treat_missing_data  = "notBreaching"
}
Enter fullscreen mode Exit fullscreen mode

Operation

  1. After running Terraform, a confirmation message will be sent to the email address registered in Terraform. You must confirm the subscription in this message.

    1. In case of an alarm, you will receive a warning message to the registered email address.
- **[KMS Keys](https://eu-central-1.console.aws.amazon.com/kms/home?region=eu-central-1#/kms/keys): Check the KMS page for Customer Managed Keys with the status "Pending deletion."**
  ![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h7emdddanpjo5c885dbu.jpg)

- **[Secrets Manager](https://eu-central-1.console.aws.amazon.com/secretsmanager/listsecrets?region=eu-central-1): Adjust the settings to view deleted secrets as shown in the provided image.**
  ![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7eiu7dpwz853t1juz9jk.jpg)
Enter fullscreen mode Exit fullscreen mode

Important Considerations

Choosing the Right Alarms Filter

In a production environment, you might want to be instantly notified of any deletion attempts. For staging or development, it may make more sense to only receive notifications when someone tries to access secrets or KMS keys marked for deletion. Consider the application lifecycle, especially if secrets or encryption keys are rarely used.

Deletion Grace Period

When setting up alarms for pending deletion of KMS keys and AWS Secrets, it's crucial to consider the deletion grace period. This period should be long enough to allow responsible parties to react to any unauthorized deletion attempts.

For KMS keys, AWS enforces a minimum waiting period of 7 days, with a default and maximum of 30 days. For Secrets Manager secrets, the minimum waiting period is also 7 days, with a default and maximum of 30 days.

When configuring your deletion grace periods, consider the following:

  • Team Responsiveness: How quickly can your team typically respond to alerts?
  • Criticality of the Resource: More critical resources might warrant longer grace periods.

To set an appropriate grace period:

# Creating an AWS kms key with 14 day for the deletion grace period

resource "aws_kms_key" "example" {
  description             = "Example KMS key"
  deletion_window_in_days = 14  # Adjust this value as needed (7-30 days)
}

# Creating a secret with 14 day for the deletion grace period

resource "aws_secretsmanager_secret" "example" {
  name                    = "example-secret"
  recovery_window_in_days = 14  # Adjust this value as needed (7-30 days)
}
Enter fullscreen mode Exit fullscreen mode

By setting an appropriate grace period, you give your team ample time to respond to alerts and prevent accidental or malicious deletions of critical resources.

Proper Access Rights

Ensure that access rights are configured properly to prevent unauthorized access and deletion of secrets and KMS keys. This includes implementing the principle of least privilege and regularly reviewing access policies to limit exposure.

Use SCP on the Organization Level

To further limit the risk of deletion, use Service Control Policies (SCPs) at the organization level. SCPs allow you to set permission guardrails that apply to all accounts within an AWS Organization. By using SCPs, you can prevent the deletion of critical resources such as secrets and KMS keys across all accounts, ensuring a higher level of security and compliance.

Cross-Account Protection

For enhanced security, consider placing production-critical KMS keys and secrets in a separate account. This isolation adds an additional layer of protection against unauthorized access and deletion. For more details on setting up cross-account and cross-region backups, refer to another tecRacer blog post on Automating Cross-Account/Cross-Region Backups with AWS Backup in AWS Organizations.

Troubleshooting

  1. If you're not receiving alarm notifications:
    • Check if you've confirmed the SNS topic subscription.
    • Verify that the CloudWatch log group exists and contains the expected log events.
  2. If alarms are triggering too frequently:
    • Review your metric filter patterns and adjust if necessary.
  3. If deletion of KMS keys or secrets are not being captured:
    • Ensure that CloudTrail is properly configured to log these events.
    • Verify that the IAM role has the necessary permissions to write to CloudWatch Logs.

Conclusion

By implementing these alarms, you significantly enhance your AWS environment's security posture. You'll be promptly informed about potentially malicious activities or misconfigurations related to KMS keys and Secrets Manager secrets. Remember, proactive security measures are crucial in cloud computing.

References

— Alexey

Image of Timescale

Timescale – the developer's data platform for modern apps, built on PostgreSQL

Timescale Cloud is PostgreSQL optimized for speed, scale, and performance. Over 3 million IoT, AI, crypto, and dev tool apps are powered by Timescale. Try it free today! No credit card required.

Try free

Top comments (0)

Best Practices for Running  Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK cover image

Best Practices for Running Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK

This post discusses the process of migrating a growing WordPress eShop business to AWS using AWS CDK for an easily scalable, high availability architecture. The detailed structure encompasses several pillars: Compute, Storage, Database, Cache, CDN, DNS, Security, and Backup.

Read full post

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay