DEV Community: Yusuf Ishola

Optimising Cloud Costs and Security: With Scientific Research Platform Using AWS Manager Store

Yusuf Ishola — Sat, 26 Apr 2025 06:40:11 +0000

In today’s cloud-driven world, managing sensitive credentials securely while keeping costs in check presents a significant challenge for organisations. Our recent publishers project demonstrates how thoughtful implementation of AWS services can achieve both goals. This article explores how we leveraged AWS Secrets Manager and AWS Systems Manager Parameter Store to securely manage database credentials and API keys while optimising cloud expenditure for our green technology research platform.

The Publishers Project: Architecture and Requirements

Our publishers project is a comprehensive platform that retrieves scientific articles about green technology from multiple publisher’s researcher using their respective APIs. The workflow follows these steps:

Lambda functions query publisher APIs using specific keywords.
Retrieved articles are processed and uploaded to Amazon S3 buckets.
Article Metadata is inserted into a PostgreSQL database for indexing and research.
AWS Step Functions orchestrate the entire workflow to avoid Lambda timeouts

This data pipeline requires secure management of two types of sensitive information:

Database credentials for PostgreSQL
Multiple publisher-specific API keys

Understanding AWS Secrets Management Options in Depth

AWS offers two primary services for managing sensitive information, each with distinct capabilities and cost profiles. Understanding their similarities and differences is crucial for optimal implementation.

AWS Secrets Manager: AWS Secrets Manager helps you manage, retrieve, and rotate database credentials, application credentials, OAuth tokens, API keys, and other secrets throughout their lifecycles. Many AWS services store and use secrets in Secrets Manager.

Secrets Manager helps you improve your security posture, because you no longer need hard-coded credentials in application source code. Storing the credentials in Secrets Manager helps avoid possible compromise by anyone who can inspect your application or the components. You replace hard-coded credentials with a runtime call to the Secrets Manager service to retrieve credentials dynamically when you need them.

AWS Systems Manager: AWS Systems Manager helps you centrally view, manage, and operate nodes at scale in AWS, on-premises, and multi-cloud environments. With the launch of a unified console experience, Systems Manager consolidates various tools to help you complete common node tasks across AWS accounts and Regions.

Parameter Store: Parameter Store, a tool in AWS Systems Manager, provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, API keys, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values. You can store values as plain text or encrypted data.

Similarities Between Secrets Manager and Parameter Store

Encryption Capabilities
Both services leverage AWS KMS to encrypt values, providing an additional security layer beyond IAM permissions. This encryption is essential for compliance requirements and protecting sensitive information.

Key/Value Store Structure
Both implement a key/value store model, allowing values to be organised with prefix schemas like application/environment/parametername. In our research platform, we organise our parameters logically:

/publisher/green-technology/api-key
/publisher/climate-chnage/api-key
/publisher/global-warming/api-key
/publisher/Sustainable-development/api-key
/publisher/clean-energy/api-key

CloudFormation Integration
Both services integrate with CloudFormation, enabling infrastructure-as-code without embedding secrets in templates. This allows our CI/CD pipeline to reference secure values rather than hardcoding them.

Versioning Support
Both services maintain version history of secrets, allowing you to track changes and restore previous values if needed. This provides an audit trail of credential modifications.

AWS Secrets Manager

AWS Secrets Manager is a specialised service designed specifically for high-security storage of sensitive credentials with advanced features:

Automatic secret rotation: Automatically updates credentials on a schedule.
Cross-account access: Shares secrets across multiple AWS accounts.
Fine-grained permissions: Controls access through IAM policies.
Integration with AWS services: Seamlessly works with RDS, Redshift, and other services.
Higher cost: $0.40 per secret per month + $0.05 per 10,000 API calls.
Larger storage capacity: Can store secrets up to 10KB in size.
Multiple region replication: Enables disaster recovery and multi-region applications.
Service limits: Supports up to 500,000 secrets per region per account.

AWS Systems Manager Parameter Store

Parameter Store provides hierarchical configuration data management with tiered pricing:

Free standard tier: Stores up to 10,000 parameters at no cost.
Hierarchical organisation: Uses path-like structures for better organisation.
Optional encryption: Can use AWS KMS for sensitive parameters.
No automatic rotation: Requires manual management of parameter updates.
Lower cost: Free for standard parameters, $0.05 per 10,000 API calls for advanced tier.
Storage limitations: 4KB for standard parameters, 8KB for advanced parameters.
No built-in cross-region replication: Requires custom implementation.
Service limits: 10,000 standard parameters per region per account.

Our Cost-Optimised Implementation Strategy

After analysing our requirements, we implemented a hybrid approach using both services strategically:

Database Credentials in AWS Secrets Manager
We store PostgreSQL database credentials in AWS Secrets Manager because:

Database credential rotation: Secrets Manager can automatically rotate database credentials, reducing security risks.
Single point of access: Our application only retrieves database credentials during initialisation.
IAM integration: Secrets Manager integrates directly with our RDS instance.
Critical security: Database access requires the highest level of protection.

The implementation in our Lambda functions uses the following approach:

def get_db_credentials():
    secret_name = "publishers_db_credentials"
    region_name = "eu-west-2"
    session = boto3.session.Session()
    client = session.client(service_name='secretsmanager', region_name=region_name)
    try:
        get_secret_value_response = client.get_secret_value(SecretId=secret_name)
        secret = json.loads(get_secret_value_response['SecretString'])
        return {
            "host": secret["postgresql"]["host"],
            "port": secret["postgresql"]["port"],
            "dbname": secret["postgresql"]["dbname"],
            "user": secret["postgresql"]["user"],
            "password": secret["postgresql"]["password"]
        }
    except ClientError as e:
        print(f"Error retrieving secret: {e}")

        raise

API Keys in AWS Systems Manager Parameter Store

We store the publisher API keys in Parameter Store because:

Higher retrieval frequency: Our Lambda functions retrieve API keys for every article request.
Multiple distinct keys: We manage five different publisher API keys.
Hierarchical organisation: We organise keys by publisher in a structured hierarchy.
Cost efficiency: Parameter Store’s free tier significantly reduces operational costs.
Adequate security: Parameter Store provides sufficient encryption for API keys.

Our implementation uses a clear hierarchical structure:

def get_springer_api_key():
    ssm = boto3.client('ssm')
    try:
        response = ssm.get_parameter(
            Name='publisher/green-technology/api-key',
            WithDecryption=True
        )
        return response['Parameter']['Value']
    except ClientError as e:
        raise Exception(f"Failed to retrieve Climate Change API key: {str(e)}")

Cost Analysis: The Financial Impact of Our Strategy
Let’s analyse the cost implications of our approach:

Scenario 1: Everything in Secrets Manager

If we stored all credentials (1 database secret + 5 API keys) in AWS Secrets Manager:

6 secrets × $0.40 per month = $2.40 per month.
Assuming 1 million API calls per month: (1,000,000 ÷ 10,000) × $0.05 = $5.00 per month.
Total: $7.40 per month.

Scenario 2: Our Hybrid Approach

Database credentials in Secrets Manager: 1 secret × $0.40 = $0.40 per month.
API keys in Parameter Store: Free (within standard tier).
API calls to Secrets Manager (less frequent): (10,000 ÷ 10,000) × $0.05 = $0.05 per month.
API calls to Parameter Store (more frequent): Free (standard tier).
Total: $0.45 per month.

Annual savings: $83.40

This cost optimisation becomes even more significant as the number of API keys increases or when API call volume grows.

Choosing Between Secrets Manager and Parameter Store: Decision Criteria

When deciding which service to use for your specific use case, consider these criteria:

Choose Secrets Manager if:

You need automatic rotation of credentials.
Cross-account access is required.
You’re using AWS services with built-in Secrets Manager integration.
You need multi-region replication.
Compliance requirements mandate advanced security features.
You need to store secrets larger than 4KB.

Choose Parameter Store if:

Cost optimisation is a priority.
You’re storing a mix of configuration data and credentials.
You don’t require automatic rotation.
You’re working within a single AWS account.
Your secrets are smaller than 4KB (standard tier).
You have a large number of parameters to store.

Alternative Solutions

While AWS provides robust solutions for secrets management, there are alternatives worth considering:

Hashicorp Vault: An open-source solution offering advanced secret management with pluggable authentication.
Azure Key Vault: Microsoft’s equivalent service for managing keys, secrets, and certificates.
Azure App Configuration: Similar to Parameter Store but for Azure environments.

For our scientific research platform, AWS services provided the best integration with our existing infrastructure.

AWS Best Practices for Credential Management
Our implementation follows AWS best practices for credential management:

1. Separation of Sensitive Information from Application Code
By storing credentials in dedicated services, we’ve completely separated sensitive information from our application code. This improves security and simplifies credential rotation without code changes.

2. Appropriate Service Selection
We’ve followed AWS’s guidance by selecting:

Secrets Manager for critical credentials requiring rotation (database credentials).
Parameter Store for configuration data and less critical secrets (API keys).

3. Hierarchical Organisation
We’ve implemented a structured parameter hierarchy in Parameter Store:

/publisher/green-technology/api-key
/publisher/climate-change/api-key
/publisher/global-warming/api-key
/publisher/Sustainable-development/api-key
/publisher/clean-energy/api-key

This organisation improves manageability and allows for more granular IAM permissions.

4. Least Privilege Access
Our Lambda functions have IAM roles with permissions only to:

Access specific secrets or parameters.
Access only the necessary actions (GetSecretValue, GetParameter).
Operate only in the required region.

5. Encryption for All Sensitive Data
We ensure that all sensitive data is encrypted:

Secrets Manager automatically encrypts all secrets with AWS KMS.
Parameter Store API keys use the SecureString type with KMS encryption.

Conclusion: Balancing Security and Cost Efficiency

Our green technology research platform demonstrates that with thoughtful service selection, organisations can achieve both robust security and cost efficiency. By leveraging AWS Secrets Manager for critical database credentials and AWS Systems Manager Parameter Store for API keys, we’ve implemented a solution that:

Follows AWS security best practices.
Reduces monthly operational costs by over 90%.
Maintains appropriate security levels for different types of credentials.
Scales efficiently as our API usage grows.

This hybrid approach represents an optimal balance between security and cost-effectiveness, proving that with careful planning, organisations don’t need to compromise on either factor. For any cloud application managing multiple types of credentials, we recommend conducting a similar analysis to determine the most appropriate credential management strategy based on security requirements and cost constraints.

As cloud architectures become increasingly complex, strategic service selection becomes more critical for maintaining both security and cost efficiency. Our scientific research platform serves as a practical example of how this balance can be achieved, demonstrating that well-architected applications can be both secure and cost-effective.

Thank you for following along. I hope you find this valuable. Build On!

If you found this helpful, don’t forget to give this article a like and follow me for more tips and insights! Your support means a lot.

How to Fix Psycopg2 ‘ModuleNotFoundError’:Using AWS Lambda-Layer

Yusuf Ishola — Sat, 22 Mar 2025 14:49:27 +0000

Introduction

The psycopg2 module is a PostgreSQL adapter for Python, which is commonly used in applications requiring database interactions. However, when deploying such applications to AWS Lambda, the module must be packaged as a Lambda layer due to Lambda’s isolated environment. This involves creating a virtual environment, installing psycopg2-binary, and zipping the site-packages directory to create the layer. Once uploaded to AWS, this layer can be attached to Lambda functions, ensuring that psycopg2 is available during execution.

What is Python Code Error: No Module Named ‘psycopg2’
When you try to run a Python script in AWS Lambda, it might not find certain libraries like psycopg2, even if they’re installed on your computer. This happens because AWS Lambda runs in a different environment than your local machine. To fix this, you need to package these libraries into something called a “Lambda layer” and attach it to your Lambda function. Think of a Lambda layer like a special package that includes all the extra tools your script needs to work properly.

Resolving the psycopg2 Module Error in AWS Lambda

Even if it's installed in your local IDE. This error occurs because AWS Lambda requires a specific environment setup for dependencies, which differs from your local development environment.

Understanding the Error

The psycopg2 module is a PostgreSQL database adapter for Python. When you install it locally using pip, it works fine in your IDE. However, AWS Lambda has its own environment and dependencies must be packaged differently.

Why psycopg2 Isn’t Found in Lambda

Local Installation vs. Lambda Environment: When you install psycopg2 locally, it’s installed in your system’s Python environment. AWS Lambda, however, runs in a separate environment (Amazon Linux) and doesn’t access your local Python environment.
Dependency Packaging: AWS Lambda requires that all dependencies be included in the deployment package. If psycopg2 isn’t properly packaged, it won’t be available in the Lambda environment.

Step 1: Guide to Creating a psycopg2 Lambda Layer
To resolve this issue, we are going to create a Lambda layer for psycopg2.

Create a Directory for the Layer:

mkdir psycopg2_layer
cd psycopg2_layer

Create a Virtual Environment:
Ensure using a compatible Python version (e.g., Python 3.9).

python3 -m venv env
source env/bin/activate

Install psycopg2-binary in the Virtual Environment:

Upgrade pip inside the virtual environment.

pip install --upgrade pip

Install psycopg2-binary:

pip install psycopg2-binary

Package psycopg2 as a Lambda Layer: Create a directory for the Lambda layer.

mkdir python
cp -r env/lib/python3.*/site-packages/* python/

Zip the directory:

zip -r psycopg2_layer.zip python/

Mac Terminal & VScode views:

Step 2: Create a Lambda Layer in AWS

Go to the AWS Lambda console.
Navigate to the “Layers” section.
Click “Create layer”: Name- psycopg2-layer
Description: PostgreSQL-adapter
Choose “Upload a .zip file” and select the psycopg2_layer.zip file.
Set the runtime to match your Lambda function’s runtime Python 3.9.

Step 3: Attach the Layer to Your Lambda Function

Open your Lambda function in the AWS console.
Go to the “Layers” section.
Click “Add a layer” and select the psycopg2 layer you created.

Conclusion

By following these steps, you can successfully resolve the psycopg2 “ModuleNotFoundError” in AWS Lambda by creating and using a Lambda layer. This approach ensures that your application has access to all necessary dependencies in the Lambda environment.

Share Your Thoughts

Have you encountered similar issues with AWS Lambda? How did you resolve them? Share your experiences in the comments below!

Amazon EKS Now Defaults to Envelope Encryption for Kubernetes API Data

Yusuf Ishola — Tue, 11 Mar 2025 07:22:04 +0000

What is Amazon EKS

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed Kubernetes service that eliminates the need to operate and maintain the availability and scalability of Kubernetes clusters in Amazon Web Services (AWS) and in your own data centre's.

Kubernetes: is an open source system that automates the management, scaling, and deployment of containerised applications.

Amazon Elastic Kubernetes Service (EKS) is rolling out a major security enhancement for Kubernetes users: default envelope encryption **for all Kubernetes API data in clusters running **version 1.28 or higher. This update integrates AWS Key Management Service (KMS) with Kubernetes KMS provider v2, delivering a fully managed, defense-in-depth security layer for your Kubernetes applications — straight out of the box.

What’s Changing?

Previously, envelope encryption in EKS was an optional feature, available through KMS provider v1. With this update, it becomes the default configuration for all Kubernetes API objects. AWS will now provide a default KMS encryption key, but you still have the flexibility to create or import your own keys into AWS KMS for your cluster’s control plane.
If you’ve already been using a customer-managed key (CMK) to encrypt Kubernetes Secrets, the same key will seamlessly extend to encrypt additional API data types in your cluster — no additional configuration needed.

Why This Matters for DevOps Engineers

This update simplifies security management for Kubernetes workloads:

Reduced Configuration Overhead: No need to manually enable encryption; it’s built-in by default.
Seamless CMK Integration: If you’re already using a CMK, no extra steps are required to extend encryption across API data.
Stronger Security & Compliance: Automatically enhances data protection without increasing operational complexity.
More Time for Innovation: Focus on delivering applications, not configuring encryption settings.

For teams prioritising security and compliance, this change reinforces data protection while maintaining flexibility. Whether you’re a seasoned cloud architect or a DevOps engineer managing multiple clusters, this update means one less security concern to worry about.

What do you think, how will this impact your workflows? Let’s discuss below!

Troubleshooting Kubernetes Persistent Volume Binding Issues.

Yusuf Ishola — Sun, 09 Feb 2025 18:41:15 +0000

As Kubernetes becomes the backbone of modern infrastructure, engineers are increasingly facing the challenges of managing PersistentVolumes (PV) and PersistentVolumeClaims (PVC) effectively. In this article, we’ll walk through a real-world scenario of troubleshooting a PVC stuck in a “Pending” state, sharing the insights and best practices that every DevOps engineer should know.

Understanding the Problem: PVC Stuck in “Pending” State

The Pending state of a PVC is often a bottleneck for engineers deploying stateful applications in Kubernetes. In our scenario, a PVC (devops-1-claim0) was failing to bind to its associated PV, with the following error:

Warning FailedBinding persistentvolume-controller volume “kt-migration-devops-1-pv” already bound to a different claim.

At first glance, this error might seem vague, but it points to a classic issue with Kubernetes’ PV lifecycle: the PV was in the Released state, preventing it from being re-bound.

The Root Cause: Understanding the “Released” State in Kubernetes

Kubernetes PVs follow a well-defined lifecycle:

Available: Ready for a PVC to claim it.

Bound: Attached to a PVC.

Released: The PVC is deleted, but the data on the PV remains, and the volume isn’t yet ready for another claim.

Failed: Something went wrong in the lifecycle.

In this case, the ReclaimPolicy for the PV was set to Retain, meaning that even after the original PVC was deleted, the data remained intact, and the PV was marked as Released. However, Kubernetes doesn’t automatically clean or make the volume available for reuse, leading to the binding failure.

Troubleshooting Steps: Clearing the “ClaimRef” Field

To resolve this issue, we needed to make the PV available again by clearing its ClaimRef, which was still pointing to the deleted PVC.

Check the PV’s Status: First, confirm the PV’s state and check if it’s in the Released state:

kubectl get pv kt-migration-devops-1-pv

Manually Clear the ClaimRef: The quickest way to reset the PV is by patching it to clear the old reference to the previous PVC:

kubectl patch pv kt-migration-devops-1-pv -p ‘{“spec”:{“claimRef”: null}}’

This effectively releases the PV for new claims, allowing the PVC to bind again.

Check the PVC Binding: Once the ClaimRef is cleared, check the status of the PVC:

kubectl get pvc -n kt-migration

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE devops-1-claim0 Bound kt-migration-devops-1-pv 300Gi RWO gp3 <unset> 2d4h devops-2-claim0 Bound kt-migration-devops-2-pv 300Gi RWO gp3 <unset> 2d4h devops-3-claim0 Bound kt-migration-devops-3-pv 300Gi RWO gp3 <unset> 2d4h

In our case, the PVC was successfully bound to the PV after clearing the ClaimRef.

Lessons Learned and Best Practices

This experience teaches several valuable lessons for managing stateful applications in Kubernetes:

Understand PV Lifecycles: As DevOps engineers, it’s crucial to understand how PVs move through their lifecycle. The Released state often trips engineers up when they expect Kubernetes to handle the cleanup automatically, especially with the Retain policy.
Monitor PVC and PV States: Keep an eye on both your PVC and PV states using kubectl describe to catch issues early. When a PVC gets stuck in Pending, don’t just focus on the PVC — check the associated PV to diagnose the problem.
Use Dynamic Provisioning When Possible: Static provisioning (pre-creating PVs) can lead to issues like this, where manual intervention is required to reset or clean up volumes. Whenever possible, rely on dynamic provisioning, where Kubernetes automatically creates and deletes volumes as needed.
Choose the Right Reclaim Policy: Be mindful of the ReclaimPolicy. In scenarios where data persistence is critical, Retain is the right choice. However, if volumes don’t need to retain data after the PVC is deleted, consider using Delete to avoid manual cleanup.

Persistent storage management in Kubernetes can be tricky, but understanding how PVs and PVCs interact will save you a lot of time and headaches. By sharing this real-world experience, I hope to help other engineers navigate similar issues with greater confidence and insight.

Kubernetes is powerful, but it requires a strong grasp of its underlying systems to ensure smooth deployments — especially in complex, stateful workloads like databases.