DEV Community: Kevin Kiruri

Building Serverless Microservices on AWS with ECS Fargate, ECR, and Terraform

Kevin Kiruri — Fri, 02 Jan 2026 17:36:09 +0000

In the evolving world of cloud-native architectures, serverless doesn’t always mean Lambda. With Amazon ECS Fargate, you can run containers without managing servers, combining the scalability of containers with the simplicity of serverless operations.

In this article, we’ll build a serverless microservices architecture using AWS ECS Fargate, ECR, and Terraform.
Our setup includes two Django-based microservices:

Reader Service – handles read operations from a shared database.

Writer Service – handles write operations to the same database.

Both services connect to a shared Amazon RDS PostgreSQL instance. The entire infrastructure is defined and provisioned through Infrastructure as Code (IaC) using Terraform.

Key Components

Amazon ECS Fargate – Runs containerized Django microservices without provisioning EC2 instances.

Amazon ECR (Elastic Container Registry) – Hosts Docker images for both services.

Amazon RDS (PostgreSQL) – Central database for the microservices.

Application Load Balancer (ALB) – Distributes traffic between services.

Terraform – Automates infrastructure provisioning.

1. Containerizing the Microservices

Each Django service (Reader and Writer) is containerized using Docker. Here is the docker compose file that builds both services and a local db container for local testing (test containers before pushing to cloud).

  db:
    image: postgres:15
    container_name: postgres-db
    environment:
      - POSTGRES_DB=${DB_NAME:-library_db}
      - POSTGRES_USER=${DB_USER:-postgres}
      - POSTGRES_PASSWORD=${DB_PASSWORD:-password}
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-postgres}"]
      interval: 10s
      timeout: 5s
      retries: 5

  reader-service:
    container_name: reader-svc
    build: ./app/reader-service/reader
    ports:
      - "8000:8000"
    environment:
      - DB_USER=${DB_USER:-postgres}
      - DB_PASSWORD=${DB_PASSWORD:-password}
      - DB_NAME=${DB_NAME:-library_db}
      - DB_HOST=db
      - DB_PORT=5432
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health/"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M
    restart: unless-stopped

  writer-service:
    container_name: writer-svc
    build: ./app/writer-service/writer
    ports:
      - "8001:8000"
    environment:
      - DB_USER=${DB_USER:-postgres}
      - DB_PASSWORD=${DB_PASSWORD:-password}
      - DB_NAME=${DB_NAME:-library_db}
      - DB_HOST=db
      - DB_PORT=5432
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health/"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M
    restart: unless-stopped

volumes:
  postgres_data:

2. After testing locally, build and push each image to ECR.

aws ecr create-repository --repository-name reader-service
aws ecr create-repository --repository-name writer-service

docker build -t reader-service .
docker tag reader-service:latest <account-id>.dkr.ecr.<region>.amazonaws.com/reader-service:latest
docker push <account-id>.dkr.ecr.<region>.amazonaws.com/reader-service:latest

3. Defining Infrastructure with Terraform

Your Terraform project will include these key modules:

VPC and Networking

RDS Instance

ECR Repositories

ECS Cluster and Task Definitions

Application Load Balancer

resource "aws_ecs_cluster" "main" {
  name = "serverless-cluster"
}

resource "aws_ecs_task_definition" "reader_task" {
  family                   = "reader-service"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 256
  memory                   = 512
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_execution_role.arn

  container_definitions = jsonencode([
    {
      name      = "reader"
      image     = "${aws_ecr_repository.reader.repository_url}:latest"
      essential = true
      portMappings = [
        {
          containerPort = 8000
          hostPort      = 8000
        }
      ]
      environment = [
        { name = "DB_HOST", value = aws_db_instance.app_db.address },
        { name = "DB_NAME", value = "app_db" },
        { name = "DB_USER", value = "admin" },
        { name = "DB_PASS", value = var.db_password }
      ]
    }
  ])
}

resource "aws_ecs_service" "reader_service" {
  name            = "reader-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.reader_task.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = aws_subnet.public[*].id
    assign_public_ip = true
    security_groups  = [aws_security_group.ecs_service.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.reader.arn
    container_name   = "reader"
    container_port   = 8000
  }

  depends_on = [aws_lb_listener.frontend]
}

Repeat a similar configuration for the Writer Service.

4. RDS Database and Networking

Both services share the same RDS instance, accessed via private networking for security.

# create database subnet group
resource "aws_db_subnet_group" "database_subnet_group" {
  name        = "${var.project_name}-${var.environment}-database-subnets"
  subnet_ids  = [aws_subnet.private_data_subnet_az1.id, aws_subnet.private_data_subnet_az2.id]
  description = "subnets for database instance"

  tags = {
    Name = "${var.project_name}-${var.environment}-database-subnets"
  }
}

# create the rds instance
resource "aws_db_instance" "database_instance" {
  engine                 = "postgres"
  engine_version         = "14"
  multi_az               = var.multi_az_deployment
  identifier             = var.database_instance_identifier
  username               = var.db_user
  password               = var.db_password
  db_name                = var.db_name
  instance_class         = var.database_instance_class
  allocated_storage      = 200
  db_subnet_group_name   = aws_db_subnet_group.database_subnet_group.name
  vpc_security_group_ids = [aws_security_group.database_security_group.id]
  availability_zone      = data.aws_availability_zones.available_zones.names[0]
  skip_final_snapshot    = true
  publicly_accessible    = var.publicly_accessible
}

Find the terraform template at: ECS Django Microservices

then run:

Terraform init

Terraform plan

Terraform apply

5. Why ECS Fargate is Serverless

When most developers hear serverless, they immediately think of AWS Lambda. But serverless is more than just event-driven functions, it’s a paradigm. It’s about abstracting infrastructure management, automating scaling, and paying only for what you use.
That’s exactly what Amazon ECS Fargate delivers for containerized workloads.

Serverless Principles and How Fargate Fits In

Let’s break it down:

No Server Management

With ECS Fargate, you don’t provision, scale, or patch EC2 instances.
You define your task requirements (vCPU, memory, networking), and AWS automatically runs your containers in a managed compute environment. This removes the operational overhead of managing ECS clusters on EC2, configuring capacity providers, or dealing with auto scaling groups.

Your focus shifts entirely to application logic, not infrastructure maintenance.

Automatic Scaling and Orchestration

Fargate integrates natively with ECS Service Auto Scaling and Application Auto Scaling.
When your traffic spikes, ECS launches more Fargate tasks; when traffic drops, it scales down automatically, no manual intervention needed.

This elasticity ensures cost-efficiency and performance stability without managing the underlying scaling policies or EC2 lifecycle.

Pay-per-Use Model

You only pay for the exact CPU and memory resources your containers use, billed per second while tasks are running.
There’s no idle server cost, no over-provisioned clusters, and no unused capacity. This aligns perfectly with serverless economics — true usage-based billing.

Seamless AWS Integration

Fargate runs as part of the broader AWS ecosystem. It integrates smoothly with:

CloudWatch Logs for centralized logging,

CloudWatch Alarms for health and performance metrics,

IAM Roles for Tasks for fine-grained permissions,

AWS X-Ray for distributed tracing, and

ECR for secure container image storage and retrieval.

This ecosystem synergy allows teams to build secure, observable, and automated serverless container platforms end-to-end.

Security and Isolation

Each Fargate task runs in its own isolated compute environment, with dedicated kernel and network interfaces.
Unlike EC2-based ECS, where containers share the host kernel, Fargate tasks achieve stronger isolation, closer to the Lambda-level security boundary, making it ideal for multi-tenant or microservice architectures.

Infrastructure as Code Ready

By pairing Fargate with Terraform, you extend serverless principles to your entire infrastructure.
Your ECS services, networking, IAM roles, and monitoring configurations all live in code that is reproducible, version-controlled, and automated. This enables serverless operations not just in runtime, but also in provisioning and deployment.

6. Observability and Monitoring

Add logging and metrics using:

AWS CloudWatch Logs – For container logs and AWS CloudWatch Alarms – For task health and CPU/memory usage using Terraform

# ECS CloudWatch Monitoring
resource "aws_cloudwatch_log_group" "ecs_logs" {
  name              = "/ecs/${var.project_name}"
  retention_in_days = 7
}

# ECS Service Alarms
resource "aws_cloudwatch_metric_alarm" "ecs_cpu_high" {
  alarm_name          = "${var.project_name}-ecs-cpu-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "CPUUtilization"
  namespace           = "AWS/ECS"
  period              = "300"
  statistic           = "Average"
  threshold           = "80"
  alarm_description   = "ECS CPU utilization is too high"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    ServiceName = aws_ecs_service.ecs_service1.name
    ClusterName = aws_ecs_cluster.ecs_cluster.name
  }
}

resource "aws_cloudwatch_metric_alarm" "ecs_memory_high" {
  alarm_name          = "${var.project_name}-ecs-memory-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "MemoryUtilization"
  namespace           = "AWS/ECS"
  period              = "300"
  statistic           = "Average"
  threshold           = "80"
  alarm_description   = "ECS Memory utilization is too high"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    ServiceName = aws_ecs_service.ecs_service1.name
    ClusterName = aws_ecs_cluster.ecs_cluster.name
  }
}

# ALB Target Health
resource "aws_cloudwatch_metric_alarm" "alb_unhealthy_targets" {
  alarm_name          = "${var.project_name}-alb-unhealthy-targets"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "UnHealthyHostCount"
  namespace           = "AWS/ApplicationELB"
  period              = "300"
  statistic           = "Average"
  threshold           = "0"
  alarm_description   = "ALB has unhealthy targets"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    LoadBalancer = aws_lb.application_load_balancer.arn_suffix
  }
}

# SNS Topic for Alerts
resource "aws_sns_topic" "alerts" {
  name = "${var.project_name}-alerts"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

Conclusion

This setup demonstrates that serverless microservices don’t have to rely on AWS Lambda.
With ECS Fargate, ECR, and Terraform, you can build production-grade, scalable, and cost-efficient systems while maintaining full control over your architecture.

It’s the perfect middle ground between full container control and the simplicity of serverless.

Happy coding!

Follow me for more demos and networking. Kevin Kiruri LinkedIn
Find the source code here: ECS Django Microservices

Secure Serverless with HashiCorp Vault and Lambda: Dynamic Database Credentials

Kevin Kiruri — Fri, 02 Jan 2026 17:35:37 +0000

In the era of cloud-native applications, managing secrets and database credentials remains one of the most critical security challenges. Traditional approaches of hardcoding credentials or storing them in environment variables create significant security risks. This article explores a revolutionary approach: dynamic database credentials using HashiCorp Vault in serverless architectures.

The Problem with Static Credentials

Most serverless applications today rely on static database credentials that suffer from several critical issues:

Long-lived secrets that increase exposure risk
Shared credentials across multiple services
Manual rotation processes prone to human error
Limited audit trails for credential usage
Credential sprawl across configuration files and environment variables

Enter Dynamic Credentials

Dynamic credentials represent a paradigm shift in secrets management. Instead of storing permanent passwords, credentials are generated on-demand with automatic expiration and cleanup. This approach provides:

Zero hardcoded credentials in application code
Short-lived credentials (1-hour lifespan by default)
Automatic rotation and cleanup
Unique credentials per request
Complete audit trail of all credential operations
IAM-based authentication eliminating API key management

Architecture Overview

The solution consists of four main components working together:

┌─────────────┐    ┌──────────────┐    ┌─────────────┐
│   Lambda    │───▶│    Vault     │───▶│ RDS MySQL   │
│  Function   │    │   (EC2)      │    │  Database   │
└─────────────┘    └──────────────┘    └─────────────┘
      │                     │                  │
   IAM Auth          Dynamic Creds      Temp User

Component Breakdown

HashiCorp Vault Server: Deployed on EC2, manages the database secrets engine and handles credential generation.

AWS Lambda Function: Authenticates with Vault using IAM roles and retrieves dynamic credentials for database access.

RDS MySQL Database: Target database where temporary users are created and automatically cleaned up.

AWS Systems Manager: Securely stores the Vault root token for infrastructure management.

1. Vault Database Secrets Engine Configuration

The heart of the solution is Vault's database secrets engine, configured to manage MySQL credentials:

resource "vault_database_secret_backend_connection" "mysql" {
  backend       = vault_mount.database.path
  name          = "mysql"
  allowed_roles = ["lambda-role"]

  mysql {
    connection_url = "{{username}}:{{password}}@tcp(${aws_db_instance.main.address}:3306)/"
    username       = var.database_master_username
    password       = random_password.rds_master_password.result
    max_open_connections = 10
    max_idle_connections = 5
  }
}

2. Dynamic Role Definition

The database role defines how temporary users are created and what permissions they receive:

resource "vault_database_secret_backend_role" "lambda" {
  backend             = vault_mount.database.path
  name                = "lambda-role"
  db_name             = vault_database_secret_backend_connection.mysql.name
  creation_statements = [
    "CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';",
    "GRANT SELECT, INSERT, UPDATE ON ${var.database_name}.* TO '{{name}}'@'%';"
  ]
  revocation_statements = [
    "DROP USER '{{name}}'@'%';"
  ]
  default_ttl = 3600  # 1 hour
  max_ttl     = 86400 # 24 hours
}

3. IAM Authentication Setup

Lambda functions authenticate with Vault using AWS IAM, eliminating the need for API keys:

resource "vault_aws_auth_backend_role" "lambda" {
  backend                   = vault_auth_backend.aws.path
  role                      = "lambda-role"
  auth_type                 = "iam"
  bound_iam_principal_arns  = [aws_iam_role.lambda_exec.arn]
  token_policies            = [vault_policy.lambda.name]
  token_ttl                 = 3600
  token_max_ttl             = 86400
}

4. Lambda Implementation

The Lambda function demonstrates the complete workflow:

def get_vault_token():
    """Authenticate with Vault using AWS IAM method"""
    session = boto3.Session()
    credentials = session.get_credentials()
    frozen_creds = credentials.get_frozen_credentials()

    # Create signed STS request for Vault authentication
    request = AWSRequest(
        method="POST",
        url="https://sts.amazonaws.com/",
        headers={"Content-Type": "application/x-www-form-urlencoded"},
        data="Action=GetCallerIdentity&Version=2011-06-15".encode('utf-8')
    )

    sigv4 = SigV4Auth(frozen_creds, "sts", "us-east-1")
    sigv4.add_auth(request)

    # Authenticate with Vault
    iam_request = {
        "role": "lambda-role",
        "iam_http_request_method": request.method,
        "iam_request_url": base64.b64encode(request.url.encode('utf-8')).decode('utf-8'),
        "iam_request_body": base64.b64encode(request.body).decode('utf-8'),
        "iam_request_headers": base64.b64encode(
            json.dumps(dict(request.headers)).encode('utf-8')
        ).decode('utf-8')
    }

    auth_response = requests.post(
        f"{vault_addr}/v1/auth/aws/login",
        json=iam_request,
        timeout=10
    )

    return auth_response.json()['auth']['client_token']

def get_database_credentials(vault_token):
    """Retrieve dynamic database credentials from Vault"""
    headers = {'X-Vault-Token': vault_token}

    creds_response = requests.get(
        f"{vault_addr}/v1/database/creds/lambda-role",
        headers=headers,
        timeout=10
    )

    return creds_response.json()['data']

Security Benefits

1. Credential Lifecycle Management

Every database credential has a defined lifecycle:

Generation: Created on-demand when requested
Usage: Valid for exactly 1 hour by default
Expiration: Automatically expires without manual intervention
Cleanup: Database user is automatically deleted upon expiration

2. Principle of Least Privilege

Each credential is granted only the minimum permissions required:

Scoped permissions: Only SELECT, INSERT, UPDATE on specific database
Time-bounded access: Maximum 24-hour lifetime
Role-based access: Tied to specific Lambda execution role

3. Comprehensive Audit Trail

Vault provides complete visibility into credential operations:

Authentication events: Who requested access and when
Credential generation: Which credentials were created
Usage patterns: How credentials are being utilized
Expiration tracking: When credentials expire and are cleaned up

Operational Advantages

1. Zero-Touch Credential Management

Once deployed, the system requires no manual intervention:

Automatic rotation: New credentials for every request
Self-healing: Failed credentials don't affect subsequent requests
Scalable: Handles thousands of concurrent credential requests

2. Developer Experience

Developers work with a simple, consistent API:

# Get credentials
vault_token = get_vault_token()
db_creds = get_database_credentials(vault_token)

# Use credentials
connection = pymysql.connect(
    host=rds_endpoint,
    user=db_creds['username'],
    password=db_creds['password'],
    database=database_name
)

3. Infrastructure as Code

The entire solution is defined in Terraform, enabling:

Reproducible deployments across environments
Version-controlled infrastructure changes
Automated testing and validation
Disaster recovery capabilities

Find the terraform template at: Hashicorp Vault Dynamic Credentials

then run:

Terraform init

Terraform plan

Terraform apply

Production Deployment Considerations

1. High Availability

For production environments, implement Vault clustering:

resource "aws_instance" "vault" {
  count = 3  # Multi-node cluster
  # ... configuration
}

resource "aws_lb" "vault" {
  # Load balancer for Vault cluster
}

2. Network Security

Deploy Vault in private subnets with proper network controls:

resource "aws_security_group" "vault" {
  ingress {
    from_port       = 8200
    to_port         = 8200
    protocol        = "tcp"
    security_groups = [aws_security_group.lambda.id]  # Only Lambda access
  }
}

Monitoring and Alerting

Key Metrics to Monitor

Credential Generation Rate: Track requests per second
Authentication Failures: Monitor failed Vault logins
Database Connection Errors: Alert on connection failures
Credential Expiration: Track credential lifecycle

CloudWatch Integration

import boto3

cloudwatch = boto3.client('cloudwatch')

def publish_metric(metric_name, value, unit='Count'):
    cloudwatch.put_metric_data(
        Namespace='VaultDynamicCredentials',
        MetricData=[
            {
                'MetricName': metric_name,
                'Value': value,
                'Unit': unit,
                'Timestamp': datetime.utcnow()
            }
        ]
    )

Cost Analysis

Infrastructure Costs

Vault EC2 Instance: $8-15/month (t3.micro)
RDS Database: $15-25/month (db.t3.micro)
Lambda Execution: Pay-per-use, typically <$1/month
Total: $25-40/month for development environment

Cost vs. Security Trade-off

While dynamic credentials add infrastructure costs, they provide:

Reduced security incidents: Potential savings of thousands in breach costs
Compliance benefits: Easier audit and regulatory compliance
Operational efficiency: Reduced manual credential management overhead

Comparison with Alternatives

Approach	Security	Complexity	Cost	Scalability
Static Credentials	Low	Low	Low	High
AWS Secrets Manager	Medium	Medium	Medium	High
Dynamic Credentials	High	High	Medium	High
IAM Database Auth	Medium	Low	Low	Medium

Conclusion

Dynamic database credentials with HashiCorp Vault represent a significant advancement in serverless security. By eliminating static credentials and implementing just-in-time access, organizations can dramatically reduce their security risk while maintaining operational efficiency.

The implementation demonstrated here provides a production-ready foundation that can be extended and customized for specific organizational needs. As serverless architectures continue to evolve, dynamic credential management will become increasingly critical for maintaining security at scale.

Run the command below to confirm your set up works.

curl -s -H "X-Vault-Token: <vault-token>" \
  "http://<vault-server-ip>:8200/v1/database/creds/lambda-role" | jq .

Key Takeaways

Dynamic credentials eliminate the risks associated with static database passwords
IAM-based authentication removes the need for API key management
Automatic credential lifecycle management reduces operational overhead
Complete audit trails provide visibility into all credential operations
Infrastructure as Code enables reproducible, version-controlled deployments

The future of secrets management is dynamic, ephemeral, and automated. Organizations that adopt these practices today will be better positioned to handle the security challenges of tomorrow's cloud-native landscape.

Follow me for more demos and networking. Kevin Kiruri LinkedIn
Find the source code here: Hashicorp Vault Dynamic Credentials

Building a Serverless CRUD API with AWS SAM, Lambda, API Gateway, and DynamoDB

Kevin Kiruri — Wed, 23 Jul 2025 20:05:58 +0000

Serverless development has become a go-to strategy for modern application architectures, and with good reason: it allows developers to focus more on building features and less on managing infrastructure. In this article, I walk you through building a fully serverless CRUD API using AWS SAM, Lambda, API Gateway, and DynamoDB all neatly wrapped in Python.

Whether you're new to AWS SAM or just looking for a practical CRUD example to learn from, this guide has you covered.

Project Overview

The goal of this project was simple: create a clean, scalable, and serverless API capable of Create, Read, Update, and Delete (CRUD) operations on a DynamoDB table. The architecture leverages the following:

API Gateway as the HTTP interface
AWS Lambda as the compute layer (Python-powered)
DynamoDB as the persistent data store
AWS SAM (Serverless Application Model) for easy infrastructure deployment

One Lambda function handles all CRUD operations, reducing overhead and keeping the solution neat and maintainable.

1. Project Initialisation

Scaffold a new Python-based SAM project using:

sam init --runtime python3.12 --dependency-manager pip --app-template hello-world --name .

This generates the standard SAM directory structure, giving you a great starting point. Ensure that you initialise the SAM project in a new, empty subdirectory as sam init does not allow initialising a project in a non-empty directory.

2. One Lambda to Rule Them All

Rather than creating separate functions for each CRUD operation, I designed one Lambda function (core/app.py) to handle all actions. It inspects the HTTP path to determine which operation (/create, /read, /update, /delete) to execute.
Why?

Less boilerplate
Centralized logic
Easier maintenance

Here is a code snippet of how I structured the 4 api endpoints in one lambda function

if http_method == 'POST':
        try:
            data = event.get('body', {})
            while isinstance(data, str):
                data = json.loads(data)
            logger.info(f"Request Data (body): {data}")
            match path:
                case '/create':
                    return create(data)
                case '/read':
                    return read(data)
                case '/update':
                    return update(data)
                case '/delete':
                    return delete(data)
                case _:
                    return make_response(404, {'message': 'Path Not Found'})

3. API Gateway Configuration

Inside template.yaml,map the four CRUD routes to the single Lambda function:

Events:
        CreateApi:
          Type: Api
          Properties:
            Path: /create
            Method: post
        ReadApi:
          Type: Api
          Properties:
            Path: /read
            Method: post
        UpdateApi:
          Type: Api
          Properties:
            Path: /update
            Method: post
        DeleteApi:
          Type: Api
          Properties:
            Path: /delete
            Method: post

All these endpoints funnel into the same function, which processes the request and routes it accordingly.

find the rest of the sam template code at: ""

4. DynamoDB Integration

Provision a DynamoDB table named crud, using a simple primary key (id). Through boto3, AWS’s Python SDK, the Lambda function will perform the necessary CRUD operations on the table. As shown in the example below, the function performs a write to the DynamoDB table called 'crud' and returns a response accordingly. This also applies to read, update and delete functions.

def create(data):
    """
    Create a new item in the DynamoDB table.
    """
    try:
        dynamodb = boto3.resource('dynamodb')
        table = dynamodb.Table('crud')
        response = table.put_item(Item=data)
        return make_response(200, {'message': 'Item created successfully'})
    except Exception as e:
        logger.error(f"Error creating item: {e}")
        return make_response(500, {'message': 'Internal Server Error', 'error': str(e)})

5.Application level observability with aws-lambda-powertools & Cloudwatch

Structure logging and tracing using aws-lambda-powertools, this makes debugging easier and ensures operational visibility in CloudWatch. This was a small addition but made a big difference. Simply import and use the tools in your lambda code as shown below.

from aws_lambda_powertools import Logger, Tracer
logger = Logger()
tracer = Tracer()

How they complement cloud monitoring

Tool	Monitoring Type	Purpose
Logger	Application Logging (CloudWatch Logs)	Understand what happened and why.
Tracer	Distributed Tracing (AWS X-Ray)	Understand how requests flow and where bottlenecks occur.

logger in cloudwatch log groups:

6. Deployment with SAM

This is done in 3 steps. validate, build and deploy.

1. sam validate

This command checks your template.yaml (SAM template) for syntax errors and verifies that it’s a valid AWS SAM template.
Think of it like linting for your infrastructure it won’t deploy anything, just confirms your YAML and structure are correct.

2. sam build

This command packages your application code and dependencies, preparing them for deployment.
It:

Creates a .aws-sam directory with the build artifacts.
Resolves dependencies (like your Python packages from requirements.txt).
Zips the code for Lambda functions as needed.

You run this before deploying to ensure everything is packaged correctly.

3. sam deploy --guided

This command deploys your serverless application to AWS.

It uploads the artifacts created by sam build to S3.
It creates or updates AWS resources defined in your template.yaml (API Gateway, Lambda, DynamoDB, etc.).
--guided prompts you interactively for settings like stack name, region, and capabilities.

During your first deploy be keen on the prompts as the answers determine how your project runs.

Then a CloudFormation stack is created and you get prompted to allow the deployment of the stack. AWS SAM uses CloudFormation under the hood to manage resources.

After only a few minutes, your CRUD backend is up and running in AWS.

7. Testing the API

To validate functionality, I created test events mimicking API Gateway POST requests. These can be tested via the Lambda console or SAM CLI.

Here are payloads you can use to test the endpoints through lambda console:

Create (POST /create)

{
  "httpMethod": "POST",
  "path": "/create",
  "body": {
    "id": "123",
    "attribute": "name",
    "value": "John Doe"
  }
}

And the successful lambda response:

Read (POST /read)

Now let's retrieve what we created in the create test, using a read event test payload

{
  "httpMethod": "POST",
  "path": "/read",
  "body": {
    "id": "123"
  }
}

lambda response:

Read (POST /update)

Now lets update John Doe to Jane Doe using an update event payload.

{
"httpMethod": "POST",
"path": "/update",
"body": {
"id": "123",
"attribute": "name",
"value": "Jane Doe"
}
}

lambda response:

The name was updated successfully in DynamoDB.

Delete (POST /delete)

Finally, let's test the delete endpoint using a delete event payload.

{
  "httpMethod": "POST",
  "path": "/delete",
  "body": {
    "id": "123"
  }
}

Our record with ID 123 was successfully deleted from DynamoDB.

Lessons Learned & Pro Tips

SAM is a Productivity Booster
It simplifies the complexity of packaging and deploying serverless applications.

Single Lambda for Multiple Endpoints = Simplicity
Keeping all logic in one handler makes routing clear and concise.

SAM Deployment Prompts Are Repetitive but Important
Sometimes SAM asks the same question twice — answer carefully, especially around authentication and capabilities.

Mind Your Dependencies
Ensure libraries like aws-lambda-powertools are in requirements.txt to avoid runtime issues.
I came across such errors in the lambda response.

To fix this, just add the dependency in the requirements.txt, then sam build and sam deploy, the process should be shorter and faster.

Happy coding!

Follow me for more demos and networking. Kevin Kiruri LinkedIn
Find the source code here: AWS SAM

Serverless CI/CD: Automating Deployments with AWS SAM, CDK and GitHub actions

Kevin Kiruri — Mon, 10 Feb 2025 17:47:19 +0000

Serverless CI/CD is a modern approach to software development that leverages serverless computing to automate the building, testing, and deployment of applications. By using AWS services like AWS SAM (Serverless Application Model) or AWS CDK (Cloud Development Kit) alongside GitHub Actions, you can create a fully automated CI/CD pipeline that requires minimal infrastructure management.

AWS SAM, CDK and github actions for deployment automation

Below is a guide to creating a simple AWS Lambda function, defining it using AWS SAM or AWS CDK, and setting up a GitHub Actions CI/CD pipeline to automate deployment.

Create your working repo

The structure may look like:

my-serverless-app/
├── .github/
│   └── workflows/
│       └── deploy.yml           # GitHub Actions CI/CD pipeline
├── lambda_function.py           # Lambda function code
├── template.yml                 # AWS SAM template (if using SAM)
├── cdk_app.py                   # AWS CDK app (if using CDK)
├── requirements.txt             # Python dependencies (if any)
├── README.md                    # Documentation
└── (other files as needed)

Create a Simple AWS Lambda Function

Let's start by creating a basic Lambda function in Python:

# lambda_function.py
import json

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Define in AWS SAM(template.yml) or AWS CDK(cdk_app.py)

Define the Lambda Function in AWS SAM (template.yml)

# template.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Simple Lambda Function

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: lambda_function.lambda_handler
      Runtime: python3.9
      CodeUri: .
      Events:
        HelloWorldApi:
          Type: Api
          Properties:
            Path: /hello
            Method: get

This template defines: A Lambda function (HelloWorldFunction) with the Python 3.9 runtime and an API Gateway trigger that exposes the Lambda function at the /hello endpoint.

Define the Lambda Function in AWS CDK (cdk_app.py)

Alternatively, you can use AWS CDK to define your Lambda function in Python:

# cdk_app.py
from aws_cdk import (
    core,
    aws_lambda as _lambda,
    aws_apigateway as apigateway,
)

class CdkAppStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # Define the Lambda function
        hello_lambda = _lambda.Function(
            self, 'HelloWorldFunction',
            runtime=_lambda.Runtime.PYTHON_3_9,
            handler='lambda_function.lambda_handler',
            code=_lambda.Code.from_asset('.')
        )

        # Expose the Lambda function via API Gateway
        apigateway.LambdaRestApi(
            self, 'HelloWorldApi',
            handler=hello_lambda
        )

# Initialize the CDK app
app = core.App()
CdkAppStack(app, "CdkAppStack")
app.synth()

Setting up a Github Actions CI/CD Pipeline

GitHub Actions automate the deployment of your Lambda function whenever you push changes to your repository. Here's how to set it up:

Step 1: Create a GitHub Actions Workflow File

Create a .github/workflows/deploy.yml file in your repository:

name: Deploy Lambda Function

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install aws-sam-cli

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v3
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Build and Deploy with SAM
        run: |
          sam build
          sam deploy --no-confirm-changeset --no-fail-on-empty-changeset

Step 2: Add AWS Credentials to GitHub Secrets

Go to your GitHub repository.
Navigate to Settings > Secrets > Actions.
Add the following secrets:

AWS_ACCESS_KEY_ID: Your AWS access key.
AWS_SECRET_ACCESS_KEY: Your AWS secret key.

Commit and push to trigger CI/CD pipeline

Push your code to the main branch to trigger the GitHub Actions workflow:

git add .
git commit -m "Initial commit with Lambda function and CI/CD pipeline"
git push origin main

Verify the Deployment

Once the pipeline runs successfully:

Go to the AWS Management Console, navigate to the API Gateway service.

Find the deployed API and test the /hello endpoint.
Click on the API to test. Naviagate to stages and copy the invoke url, it should look like:

https://8tx1anr2sa.execute-api.us-east-1.amazonaws.com

Open the link in a new tab and add your route, /hello.
You should see the response: "Hello from Lambda!".

Best Practices

Start with SAM for simpler deployments
Gradually adopt CDK as complexity grows
Implement GitHub Actions early for consistent delivery
Maintain separate configurations for different environments

In this guide, we walked through creating a simple Lambda function, defining it using AWS SAM or CDK, and setting up a GitHub Actions pipeline to automate deployments. We also covered troubleshooting steps to ensure your API Gateway and Lambda resources are deployed correctly.

With this approach, you can focus on writing code while AWS and GitHub Actions handle the heavy lifting of infrastructure management and deployment. Whether you're building a small project or a large-scale application, serverless CI/CD empowers you to deliver software faster and with greater reliability.

Happy coding!
Follow me for more demos and networking. Kevin Kiruri LinkedIn

Lambda

Kevin Kiruri — Sat, 08 Feb 2025 13:50:51 +0000

AWS Aurora DSQL for Django Developers: A Step-by-Step Guide

Kevin Kiruri — Fri, 07 Feb 2025 06:15:55 +0000

Amazon Aurora is a cloud-native relational database service that provides high performance and scalability. With the introduction of Aurora DSQL (Distributed SQL), developers can now leverage distributed database capabilities to enhance reliability and performance. This guide will walk you through setting up Aurora DSQL for a Django project in four steps:

Prerequisites

Before proceeding, ensure you have the following:

An AWS account with necessary IAM permissions.
A Django project.
AWS CLI installed and configured.
boto3 installed in your Django environment (pip install boto3).
Ensure you have the right dependancies in the requirements.txt

Django==4.0
django-admin-cli==0.1.1
djangorestframework==3.15.1
psycopg2==2.9.9
sqlparse==0.5.0
python-dotenv==0.19.0

aurora_dsql_django
boto3>=1.35.74
botocore>=1.35.74

Step 1.Provision an Aurora DSQL Cluster

Navigate to the Aurora DSQL Console.
Create Cluster
Security Group for network access. Inbound - Best practice: Restrict inbound access to your VPC CIDR (if the application runs inside the same VPC) or your server’s specific IP and avoid using 0.0.0.0/0 unless for testing purposes Outbound - Is set to Allow all traffic from 0.0.0.0/0 by default

Step 2: Connect Django to Aurora DSQL

Change Database settings in your django app's settings.py

DATABASES = {
    'default': {
        'HOST': 'uiabtxahshv6at5pcfidcxfnbq.dsql.us-east-1.on.aws',
        'USER': 'postgres',
        'NAME': 'postgres',
        'ENGINE': 'aurora_dsql_django',
        'OPTIONS': {
            'sslmode': 'require',
            'region': 'us-east-1',
            'region': 'us-east-2',
            'expires_in': 60
        }

    }
}

Step 3: Migrate the Database

Run the following commands to apply migrations:

python manage.py makemigrations

python manage.py migrate

Remember to clear any previous migrations, using:

find . -path "*/migrations/*.py" -not -name "__init__.py" -delete

find . -path "*/migrations/*.py" -not -name "__init__.py" -delete

Step 4: Monitor and Scale

Use Amazon CloudWatch and Performance Insights to monitor query performance and optimize configurations.

Aurora DSQL enhances Django applications by providing high availability, scalability, and better performance. By following this guide, you can integrate Aurora DSQL into your Django project efficiently. Explore additional optimizations and best practices to get the most out of your Aurora database setup.

Follow me for more demos and networking. Kevin Kiruri LinkedIn

Amazon Aurora DSQL: The New Era of Distributed SQL

Kevin Kiruri — Mon, 30 Dec 2024 18:35:35 +0000

Amazon Aurora DSQL (dee-sequel) is a groundbreaking serverless, distributed SQL database that delivers active-active high availability. Announced at AWS re:Invent 2024, it is now available in preview across US East (N. Virginia), US East (Ohio), and US West (Oregon). With its distributed architecture, Aurora DSQL enables simultaneous read and write operations across multiple regions while maintaining strong consistency. Transactions are processed locally, with cross-region concurrency checks performed only at commit, ensuring performance and reliability.

What sets Aurora DSQL apart?

Effortless Scalability: Seamlessly scale reads, writes, and storage independently, providing virtually unlimited horizontal scaling for workloads of any size.
Availability: Ensures 99.99% uptime in single-region deployments and an impressive 99.999% for multi-region clusters.
Performance: Up to 4x faster read and write operations compared to popular distributed SQL databases, making it a top choice for performance-critical applications.
Compatibility: PostgreSQL-compatible, allowing developers to use well-known relational database concepts, accelerating adoption.
Management: Completely serverless, Aurora DSQL eliminates the complexities of patching, upgrades, and maintenance, freeing developers to focus on innovation.
Event-Driven and Microservice Ready: Optimized for serverless and microservices architectures.

Core Architecture Components

Distributed Design:

Amazon Aurora DSQL is built on a robust distributed architecture, at its core, the architecture is composed of four key components:

Relay/Connectivity: Manages client connections, routing requests to appropriate resources while ensuring seamless communication across the system.
Compute/Databases: Handles query execution and processing, leveraging PostgreSQL compatibility for familiar operations.
Transaction Log and Concurrency Control: Provides atomicity and isolation for transactions while ensuring consistency across nodes and regions.
User Storage: Ensures data durability and redundancy by replicating user data across multiple Availability Zones.

These components are orchestrated through a control plane that maintains coordination and redundancy across three Availability Zones. This design enables self-healing capabilities, automatic scaling, and high availability, ensuring that infrastructure failures do not impact database operations.

Multi-Region Clusters:

Aurora DSQL takes availability and resilience to the next level with multi-region linked clusters, enabling applications to achieve high performance and reliability on a global scale.

Linked Clusters: Multi-region clusters operate in active-active mode, allowing read and write operations in multiple regions simultaneously. Data is synchronously replicated across regions, ensuring strong consistency and eliminating replication lag.
Resilience: Each linked cluster provides independent endpoints for concurrent operations. In the event of a failure in one region, the other region continues operating seamlessly, ensuring uninterrupted service.
Cross-Region Consistency: Transactions are processed locally in their originating region, with cross-region concurrency checks performed during the commit phase. This approach minimizes latency while maintaining strong data consistency.

Exploring Aurora DSQL Multi-Region linked clusters

Curious about Amazon’s latest innovation in distributed SQL databases, I decided to put Multi-Region clusters (one of Aurora DSQL's Core Architecture Components) to the test by creating clusters in different regions to demonstrate cross-region replication and consistent reads from both endpoints.

Creating a cluster in Aurora DSQL:

Navigate to https://console.aws.amazon.com/dsql and simply create a cluster, add linked regions and choose a region for your linked cluster region.

Connecting to the cluster using an authentication token and running SQL commands in aurora DSQL:

Choose the cluster you want to connect to, copy the end point, then use the command below to use psql to start a connection to your cluster. You should see a prompt to provide a password. generate an authentication token and use it as the password.

PGSSLMODE=require \ psql --dbname postgres \ --username admin \ --host uiabtxahshv6at5pcfidcxfnbq.dsql.us-east-1.on.aws \ --password

Writing in one region and reading from the second region:

You can perform write operations in one region and immediately read the data from another region.

Some of the queries to try:

INSERT INTO example (id, name) VALUES (1, 'Aurora DSQL Test');

SELECT * FROM example WHERE id = 1;

Why this stands out:

Both regions(us-east-1 and us-east-2)operate as active endpoints, enabling concurrent read and write operations across regions.
Aurora DSQL guarantees that all reads and writes return the latest committed writes no matter the region.
If us-east-1 experiences downtime, all operations can seamlessly continue in us-east-2 ensuring high availability.
Applications in different regions access the same database, reducing complexity.
Ease of management.

This setup is ideal for globally distributed applications such as e-commerce platforms, financial systems, or multi-region SaaS services, where uptime, consistency, and performance are critical.

Follow me for more demos and networking. Kevin Kiruri LinkedIn

Monitoring AWS Lambda Functions with AWS X-Ray and CloudWatch: Advanced Technique

Kevin Kiruri — Mon, 16 Dec 2024 23:33:33 +0000

It is essential to understand the state of your Lambda-based application to ensure its reliability and health. Monitoring provides information that helps you detect and resolve performance problems, outages and errors in your workloads. Lambda-based applications often integrate with multiple services, which makes it just as important that you monitor each service endpoint. In this article, we will explore advanced techniques that allow you to leverage AWS X-Ray and CloudWatch to monitor and observe your applications.

AWS X-Ray Advanced features with Lambda:

Since most serverless applications consist of multiple service integrations, troubleshooting performance issues or errors usually involves tracking a request from the source caller through all involved services. In this case, AWS X-Ray is a faster and more convenient tool to trace distributed capabilities as it allows you to visualise and analyse the flow of requests across your application.

To use X-Ray in Lambda you can activate it in the Lambda console for a specific function or enable it in the AWS SAM template:

Resources:
  GetLocations:
    Type: AWS::Serverless::Function
    Properties:
      Tracing: Active

Make sure you also provide permission using the AWSXRayDaemonWriteAccess managed policy. You must also activate X-Ray for each service in your workload. Once enabled X-Ray starts collecting tracing data for events. You can instrument your code using AWS X-Ray SDK to annotate traces and add custom metadata. In python:

from aws_xray_sdk.core import xray_recorder

@xray_recorder.capture('process_event')
def lambda_handler(event, context):
    # Your function logic here
    pass

You can isolate which part of the system is causing the most latency using the X-Ray service map, which visually represents the communication between your lambda function and other services on the X-Ray console.

Finally, using Subsegments, leverage granular tracing within a single Lambda invocation. In python:

with xray_recorder.in_segment('custom_subsegment'):
    # Specific operations to trace
    pass

AWS CloudWatch

AWS CloudWatch provides a platform for centralised monitoring and logging. All Lambda functions are automatically integrated with CloudWatch and log various standard metrics that get published to CloudWatch metrics. Key CloudWatch metrics for lambda include invocations, Errors, Duration, Concurrent executions and memory usage. You can create CloudWatch alarms based on these metrics to alert on threshold. Leverage the creation of custom Dashboards to display key metrics such as errors, execution duration, and invocations in one view. Analyze Logs using CloudWatch Logs Insights to query your application’s logs for debugging and optimization.

Advanced Monitoring with X-Ray and CloudWatch

Correlate Traces and Logs: Use trace IDs to link X-Ray traces with CloudWatch logs for detailed debugging. This enables you to trace a request’s journey and investigate logs for specific trace segments, offering precise context during troubleshooting.
Real-time Anomaly Detection: Enable anomaly detection in CloudWatch to establish dynamic thresholds based on historical patterns. This helps automatically identify unusual behaviours in your metrics, such as unexpected spikes in errors or latency, reducing the manual effort in monitoring setups.
Automating Alerting and Remediation: Set up CloudWatch alarms based on Lambda performance or error trends and automate remediation steps using SNS or step functions to handle issues as soon as they arise
Optimise Performance: Regularly analyze X-Ray traces and CloudWatch metrics to identify inefficiencies, such as redundant function calls or underutilized resources. These insights allow you to fine-tune function configurations, optimize code, and reduce operational costs.

Monitoring Lambda-based applications with AWS X-Ray and CloudWatch enables you to maintain high availability, optimize performance, and quickly resolve issues. By focusing on critical metrics like errors, execution time, and throttling, and leveraging the advanced capabilities of these tools, you can build robust and resilient serverless applications.

Happy Monitoring!
Let's connect on LinkedIn