DEV Community: Ogbeide Godstime Osemenkhian

The OpenSearch Outage You Can't Fix: Why 2-Node Clusters Always Fail.

Ogbeide Godstime Osemenkhian — Wed, 24 Dec 2025 12:37:07 +0000

The Silent Cluster Killer: What Happens When Your Search Engine Just Stops

Imagine this: it's 3 AM, your alerts start firing, and your application's search functionality is completely down. Your Amazon OpenSearch dashboard shows a hauntingly empty metrics screen. You try to restart nodes, but nothing responds. Your cluster isn't just unhealthy, it's brain-dead. This is quorum loss, and it's every OpenSearch administrator's nightmare scenario.

What Exactly Is Quorum Loss (And Why Should You Care)?

Quorum loss occurs when your OpenSearch cluster can't maintain enough master-eligible nodes to make decisions. Think of it like a committee that needs a majority vote to function, but too many members have left the room. The cluster becomes completely paralyzed:

Search and indexing operations halt immediately
CloudWatch metrics disappear as if your cluster never existed
All administrative API calls fail
The console shows "Processing" indefinitely

But here's what makes this particularly dangerous: Once quorum loss occurs, you might be lucky to update the cluster without it getting stuck, but in most cases, it gets stuck, and you cannot fix it yourself. Standard restarts won't work. Only AWS Support can perform the specialized backend intervention needed to revive your cluster, and this process typically takes 24-72 hours of complete downtime.

The Root Cause: Why Your Two-Node Cluster Is a Time Bomb

The most common path to quorum loss begins with a seemingly reasonable decision: running a two-node cluster to save costs. Here's the fatal math:

Quorum requires a majority of master-eligible nodes. With 2 nodes, you need N/2 + 1 = 2 nodes present. If just one node fails, the remaining node cannot reach quorum (1 out of 2 isn't a majority). Your cluster is now in a deadlock, unable to elect a leader, unable to make decisions, and completely stuck.

This isn't just theoretical. AWS explicitly warns against this configuration because it violates a fundamental distributed systems principle: always use an odd number of master nodes.

Your Recovery Playbook: What to Do When Disaster Strikes

Step 1: Recognize the Symptoms Immediately

CloudWatch metrics suddenly stop (no gradual decline, just complete silence)
Cluster health API returns no response or times out
Dashboard shows "Processing" with no change for hours
Application search/logging features completely fail

Step 2: Contact AWS Support (Your Only Option)

Open a HIGH severity support case immediately
Clearly state: "OpenSearch cluster has lost quorum and requires backend node restart."
Provide: Domain name, AWS region, and approximate failure time
Do not attempt console restarts they won't work and may complicate recovery

Step 3: Prepare for the Recovery Process

AWS Support will:

Use internal tools to identify stuck nodes
Safely terminate problematic nodes at the infrastructure level
Restart the cluster with proper initialization
Verify health restoration and data integrity

Critical reality check: During this entire process, your cluster will be completely unavailable. This is why prevention isn't just better, it's essential.

The Prevention Blueprint: Architecting for Resilience

1. Master Node Configuration: The Non-Negotiable Rule

NEVER USE:      ALWAYS USE:
- 1 master      - 3 masters (minimum for production)
- 2 masters     - 5 masters (for larger clusters)
- 4 masters     - Any ODD number (3, 5, 7, etc.)

Why odd numbers matter: With 3 master nodes, the cluster can lose 1 node and still maintain quorum (2 out of 3 is a majority). With 5 masters, it can withstand 2 failures. This is the foundation of high availability.

2. Dedicated Master Nodes: Separation of Concerns

Dedicated masters handle only cluster management tasks, not your data or queries. This separation prevents resource contention during peak loads and ensures stable elections.

Production minimum: 3 dedicated master nodes using instances like m6g.medium.search or c6g.medium.search.

3. Multi-AZ Deployment: Surviving Availability Zone Failures

Deploy your master nodes across three different Availability Zones. This ensures that even if an entire AZ goes down, your cluster maintains quorum and continues operating.

Production-Grade Configuration Examples

Option A: Cost-Optimized Production Setup (Recommended Baseline)

# Terraform configuration for resilient OpenSearch
resource "aws_opensearch_domain" "production" {
  cluster_config {
    instance_type          = "m6g.medium.search"  # Graviton for price-performance
    instance_count         = 3                    # 3 data nodes
    dedicated_master_enabled = true
    master_instance_type   = "m6g.medium.search"  # Same as data nodes
    master_instance_count  = 3                    # 3 dedicated masters
    zone_awareness_enabled = true
    availability_zone_count = 3                   # Spread across 3 AZs
  }
}

Option B: Development/Test Environment (Understanding the Trade-offs)

# For NON-PRODUCTION workloads only
resource "aws_opensearch_domain" "development" {
  cluster_config {
    instance_type        = "t3.small.search"     # Burstable instance
    instance_count       = 3                    
    zone_awareness_enabled = false               # Single AZ
  }

}

Critical clarification on T3 instances:

While T3 instances (t3.small.search, t3.medium.search) offer lower costs, they come with significant limitations:

Cannot be used with Multi-AZ with Standby (the highest availability tier)
Not recommended for production workloads by AWS
Best suited for development, testing, or very low-traffic applications

Cost vs. Risk: The Business Reality

Let's be brutally honest about the financial implications:

The "Savings" Trap:

2-node cluster: ~$100/month
Risk: Complete outage requiring AWS Support
Downtime: 24-72 hours
Business impact: Lost revenue, engineering panic, customer trust erosion
True cost: $100 + (72 hours of outage impact)

The Resilient Investment:

3 master + 3 data nodes: ~$300/month
Risk: Automatic failover, continuous availability
Downtime: Minutes during AZ failure (if properly configured)
Business impact: Minimal, transparent to users
True cost: $300 + (peace of mind)

The math becomes obvious when you consider that just one hour of complete search unavailability for a customer-facing application can cost thousands in lost revenue and damage to brand reputation.

Your Actionable Checklist

Immediate Actions

Audit your current OpenSearch clusters; identify any with 1 or 2 master nodes
Review your CloudWatch alarms; ensure you're monitoring ClusterStatus.red and MasterReachableFromNode.
Document your recovery contacts, know exactly how to open a high severity AWS Support case

Medium-Term Planning

Test your snapshot restoration process; regularly validate backups
Implement Infrastructure as Code using Terraform or CloudFormation for all changes
Schedule maintenance windows for any configuration changes

Long-Term Strategy

Migrate to 3+ dedicated master nodes during your next maintenance window
Enable Multi-AZ deployment for production workloads
Consider Reserved Instances for predictable costs (30-50% savings)
Evaluate OpenSearch Serverless for variable workloads

Quorum loss isn't a hypothetical concern; it's a predictable failure mode of improper OpenSearch architecture. The recovery process is painful, lengthy, and entirely dependent on AWS Support.

The solution is simple but non-negotiable: Always deploy with 3 or more nodes across multiple Availability Zones. The additional few hundred dollars per month isn't an expense; it's insurance against catastrophic failure.

Your search infrastructure is the backbone of modern applications. Don't let a preventable configuration error become your next production incident. Architect for resilience from day one.

Additional Resources

Have you experienced quorum loss in your OpenSearch clusters? Share your recovery stories in the comments below. Let's help the community learn from our collective experiences.

Xray and Adot

Ogbeide Godstime Osemenkhian — Fri, 22 Aug 2025 09:02:02 +0000

Building Event-Driven Microservices: My LocalStack Journey

Ogbeide Godstime Osemenkhian — Fri, 04 Jul 2025 15:53:58 +0000

In this blog, you will discover how LocalStack makes developing Lambda-based microservices effortless. Test event-driven architectures locally with S3, SQS, and more.

The Microservices Development Challenge

Last month, I was deep into building an event-driven microservices architecture using AWS Lambda. Picture this: file uploads triggering processing workflows, messages flowing between services, notifications being sent - the whole nine yards of modern serverless architecture.

The problem? Testing this locally was nearly impossible. I was constantly deploying to AWS just to see if my Lambda functions would trigger correctly, if my SQS messages were being processed, or if my S3 events were firing as expected. The feedback loop was painfully slow, and debugging felt like shooting in the dark.

That's when I discovered LocalStack, and it completely revolutionized how I develop event-driven systems.

What is LocalStack? (And Why Microservices Developers Love It)

Imagine having your own personal AWS cloud running right on your laptop. That's essentially what LocalStack is - a clever piece of software that mimics AWS services locally, perfect for testing complex event-driven architectures.

I was skeptical at first. "There's no way Lambda functions will actually trigger from S3 events locally," I thought. However, after using it for several months, I can confidently say it has transformed how I build microservices.

Here's what changed for my development workflow:

Testing became instant instead of waiting for deployments
I could develop complex event flows offline
Debugging Lambda functions has become as easy as debugging any local code
No more "deploy and pray" development cycles

Getting Started: My First LocalStack Setup

Setting up LocalStack was surprisingly straightforward. I remember being intimidated by the documentation at first, but it's just a Docker container that pretends to be AWS.

Here's the Docker Compose file I use (and yes, it is this simple):

version: '3.8'

services:
  localstack:
    container_name: my-local-aws
    image: localstack/localstack:4
    environment:
      SERVICES: s3,sqs,sns,ses  # Start with just what you need
      PERSISTENCE: 1            # Keep data between restarts
      DEBUG: 1                  # Helpful for learning

    ports:
      - "4566:4566"  # The magic port where AWS lives locally

    volumes:
      - localstack-data:/var/lib/localstack
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  localstack-data:

The first time I ran docker-compose up, I felt like a kid on Christmas morning. Within 30 seconds, I had S3, SQS, and SES running on my laptop. No credit card required.

The Services That Power My Event-Driven Architecture

Lambda: The Heart of Microservices

Lambda functions are the core of my event-driven architecture, and LocalStack makes testing them incredibly smooth. I can trigger functions from S3 events, SQS messages, or API Gateway calls - all locally.

The game-changer? I can set breakpoints in my Lambda code and debug it just like any other local application. No more adding print statements and redeploying to see what's happening.

# Your Lambda code works exactly the same
def lambda_handler(event, context):
    # Process S3 event, SQS message, etc.
    return {'statusCode': 200, 'body': 'Success'}

S3: Event-Driven File Processing

S3 events are crucial for microservices that process files. With LocalStack, I can upload a file and immediately see my Lambda function trigger, process the file, and send messages to other services.

The best part? I can test different file types, sizes, and edge cases without worrying about cleanup or storage costs.

SQS: Reliable Inter-Service Communication

SQS is the backbone of my microservices communication. LocalStack lets me test message flows, dead letter queues, and retry logic locally. I can even simulate message failures to test my error handling.

The magic moment was when I realized I could test my entire microservices choreography locally - watching messages flow from service to service in real-time.

Making LocalStack Feel Like Home

The Setup Script That Saved My Sanity

After a few weeks of manually creating the same microservices infrastructure every time I restarted LocalStack, I got smart and wrote a setup script. Now, every time LocalStack starts, it automatically creates my entire event-driven architecture.

Here's my initialization script for a typical microservices setup:

#!/bin/bash
echo "Setting up microservices infrastructure..."

# Create S3 buckets for different services
awslocal s3 mb s3://user-uploads
awslocal s3 mb s3://processed-files
awslocal s3 mb s3://service-logs

# Set up message queues for inter-service communication
awslocal sqs create-queue --queue-name user-events
awslocal sqs create-queue --queue-name file-processing
awslocal sqs create-queue --queue-name notification-queue

# Create SNS topics for pub/sub messaging
awslocal sns create-topic --name user-updates
awslocal sns create-topic --name system-alerts

echo "Microservices infrastructure ready! 🚀"

I put this in ./initialization/aws/init-aws.sh and LocalStack runs it automatically when it starts. It's like having a personal assistant set up your workspace every morning.

Pro Tips I Learned the Hard Way

Start Small: Don't enable every AWS service on day one. I made this mistake and LocalStack took forever to start. Begin with just the services you actually use.

Use Persistence: Add PERSISTENCE: 1 to your Docker Compose file. Trust me, you don't want to lose your test data every time you restart.

Install awslocal: This little CLI tool is a game-changer. Instead of typing aws --endpoint-url=http://localhost:4566, you just type awslocal. Your fingers will thank you.

How LocalStack Revolutionized My Microservices Testing

From "Deploy and Debug" to "Test First"

Before LocalStack, my microservices testing strategy was basically "write unit tests for individual functions and hope the integration works in production." Testing event-driven flows was nearly impossible locally.

Now I can test my entire microservices architecture locally:

Upload a file to S3 (triggers file-processing service)
Lambda processes file and sends message to SQS
Another Lambda picks up the SQS message
Processes data and stores results in DynamoDB
Publishes completion event to SNS
Notification service sends email via SES

All of this happens in seconds, and I can run it as many times as needed to perfect the flow.

The "Gotcha" Moments That LocalStack Catches

Let me tell you about some tricky bugs LocalStack has helped me catch:

The Event Structure Mixup: I once wrote a Lambda function that expected S3 events but was receiving SQS messages. The event structure was completely different, but I only discovered this when testing the full flow locally.

The Permission Puzzle: My Lambda function was failing silently because it didn't have permission to write to a specific S3 bucket. LocalStack's logs showed me the exact permission error immediately.

The Message Format Fiasco: I was sending JSON messages between services, but one service was expecting XML. LocalStack let me trace the entire message flow and spot the mismatch instantly.

Making LocalStack Lightning Fast

The Need for Speed

When I first started using LocalStack, it felt a bit sluggish. Turns out, I was being greedy and enabling every single AWS service. Here's what I learned about making it fast:

Only Enable What You Use: If you're just testing S3 and SQS, don't enable Lambda, DynamoDB, and 15 other services. Your startup time will go from 2 minutes to 20 seconds.

Persistence is Your Friend: Enable persistence so your data survives container restarts. Nothing's more frustrating than losing your test setup because Docker decided to restart.

Give It Enough Memory: LocalStack is doing a lot of heavy lifting. I give it at least 2GB of RAM, and it runs much smoother.

The Gotchas I Wish Someone Had Told Me

Port 4566 is Popular

Apparently, everyone uses port 4566 for something. If LocalStack won't start, check if something else is using that port. I usually just change it to 4567 in my Docker Compose file and move on with my life.

Your Code Needs One Tiny Change

The beautiful thing about LocalStack is that your existing AWS code works with almost no changes. You just need to point it to localhost:4566 instead of the real AWS endpoints:

# Production
s3_client = boto3.client('s3')

# LocalStack (just add endpoint_url)
s3_client = boto3.client('s3', endpoint_url='http://localhost:4566')

Lambda Functions Can Be Tricky

If you're testing Lambda functions, they need Docker to run inside LocalStack. Make sure you mount the Docker socket in your compose file, or your Lambdas will fail silently (and you'll spend hours debugging like I did).

When Things Go Wrong (And They Will)

Debugging Like a Detective

LocalStack is pretty reliable, but when something goes wrong, here's how I debug:

Check the Health: curl http://localhost:4566/_localstack/health tells you which services are running. If S3 shows as "disabled" when you expect it to be "available," you know where to start looking.

Read the Logs: docker logs my-local-aws -f is your best friend. LocalStack is pretty chatty about what it's doing, especially with DEBUG mode enabled.

List Your Resources: Sometimes I forget what I've created. awslocal s3 ls or awslocal sqs list-queues helps me remember what's actually there.

The Development Speed Boost You'll Love

Instant Feedback Loops: Testing microservices interactions used to take minutes (deploy, test, debug, repeat). Now it takes seconds. I can iterate on my event-driven architecture as fast as I can think.

Complete Offline Development: I can develop my entire microservices stack on a plane, in a coffee shop, or anywhere without internet. The whole AWS ecosystem runs on my laptop.

Fearless Experimentation: Want to try a new event pattern? Test a different message format? Experiment with Lambda triggers? Go ahead! There's no deployment overhead or cleanup required.

Real Integration Testing: I can test the actual integration between services, not just mock it. This catches so many issues that unit tests miss.

Making the Transition Smooth

The Environment Switch Pattern

Here's the pattern I use to seamlessly switch between LocalStack and real AWS:

import os

def get_aws_client(service):
    if os.getenv('ENVIRONMENT') == 'local':
        return boto3.client(
            service,
            endpoint_url='http://localhost:4566',
            aws_access_key_id='test',
            aws_secret_access_key='test'
        )
    else:
        return boto3.client(service)  # Uses real AWS credentials

Now I can develop locally and deploy to production with the same code. Just change an environment variable.

Keeping Your Data Safe

One thing I learned the hard way: back up your LocalStack data if you're working on something important. I once lost a week's worth of test data when I accidentally deleted my Docker volume.

Now I occasionally run:

docker cp my-local-aws:/var/lib/localstack ./backup

Just in case.

When LocalStack Misbehaves

The "Turn It Off and On Again" Solution

90% of LocalStack issues can be solved with:

docker-compose down -v
docker-compose up -d

The -v flag removes volumes, giving you a completely fresh start. Sometimes LocalStack just needs a clean slate.

The "It Was Working Yesterday" Problem

If LocalStack suddenly stops working, check if Docker is running out of space:

docker system df

If you're low on space, clean up with:

docker system prune

The Lambda Mystery

Lambda functions in LocalStack can be finicky. If they're not working, make sure you've mounted the Docker socket in your compose file. Lambda needs Docker to run, and without that mount, it fails silently.

Why Every Microservices Developer Should Try LocalStack

LocalStack isn't just a tool - it's a development philosophy. It's about building and testing your entire system locally before ever touching the cloud.

Your Next Steps

Start with Lambda + SQS: Copy my Docker Compose file and create a simple event-driven flow
Add S3 Events: Test file upload triggers and processing workflows
Build Your Architecture: Gradually add more services as your microservices grow
Share Your Experience: I'd love to hear how LocalStack changes your microservices development

The Bottom Line

If you're building event-driven microservices with AWS, LocalStack will transform your development experience. It's the difference between hoping your services work together and knowing they do.

Trust me, once you experience the joy of testing complex event flows locally, you'll never go back to deploy-and-pray development.

Have you tried LocalStack? What's been your experience? Drop a comment below - I'd love to hear your stories!

Helpful Links

LocalStack Documentation - The official docs
awslocal CLI - Makes your life easier
LocalStack Community - Great place to get help

DLQ Redrive for Amazon SQS

Ogbeide Godstime Osemenkhian — Sat, 18 Jan 2025 23:31:22 +0000

SQS

Amazon Simple Queue Service (SQS) is a fully managed messaging service that helps decouple application components and manage message queues efficiently.
While SQS ensures reliable message delivery, there are cases where messages fail to be processed successfully. DLQ (Dead Letter Queue) Redrive is critical in handling such cases effectively.

What is DLQ Redrive?

A Dead Letter Queue (DLQ) is a secondary queue used to store messages that couldn’t be processed successfully by a consumer.

How Dead-Letter Queue Redrive Works

Dead-letter queue (DLQ) redrive is a powerful feature in Amazon SQS that helps you manage unconsumed messages in a dead-letter queue. Instead of leaving messages stuck in the DLQ, you can use the redrive functionality to move these messages back to their source queue for another attempt at processing or redirect them to a different queue for specialized handling.

By default, the DLQ redrive process moves messages from the DLQ to the original source queue. However, Amazon SQS also provides flexibility by allowing you to specify a different queue as the redrive destination. The key requirement is that the destination queue must match the type of the DLQ. For instance, if the DLQ is a FIFO queue, the destination queue must also be a FIFO queue to ensure message ordering and deduplication requirements are met.

Another essential configuration option is the redrive velocity, which controls the rate at which messages are moved from the DLQ to the destination queue. This lets you balance throughput with system stability, ensuring the destination queue isn’t overwhelmed with a sudden influx of messages.

Message Order During Redrive

When redriving messages, Amazon SQS processes them in the order they were received in the DLQ, starting with the oldest message first. However, it’s important to note how this interacts with new messages in the destination queue. The destination queue processes all incoming messages—whether redriven from the DLQ or newly published by a producer—in the order they arrive.

For example, imagine a FIFO queue receiving messages from a producer while also ingesting redriven messages from a DLQ. These two streams of messages will interweave based on their arrival timestamps, ensuring the destination queue processes messages in a consistent but mixed order.

Why Use DLQ Redrive?

Easy Debugging: Messages that fail repeatedly are moved to the DLQ, providing a safe space for analysis.
System Resilience: By isolating problematic messages, DLQ Redrive helps maintain the stability of the main queue.
Improved Visibility: Developers gain insights into recurring issues or patterns in message failures.

Setting up a DLQ in Amazon SQS

Setting up a DLQ involves creating a primary queue and associating a secondary queue (the DLQ) with it.

Create a Primary Queue

Open the Amazon SQS Console.
Click Create Queue.
Configure the queue settings (e.g., name, retention period, visibility timeout).
Note the ARN (Amazon Resource Name) of the queue for later use.

Create a Dead Letter Queue

Create a second queue, which will serve as the DLQ.
Note the ARN of the DLQ as you’ll associate it with the primary queue.

Associate the DLQ with the Primary Queue

Navigate to the Primary Queue in the SQS Console.
Under Redrive allow policy, specify:
The ARN of the DLQ.
The MaxReceiveCount, determines how many processing attempts a message can have before being sent to the DLQ.

Processing Messages in the DLQ

Once messages are in the DLQ, you’ll need to handle them manually or programmatically to address the underlying issues. AWS provides several ways to process messages in a DLQ:

Manually Inspect Messages:

Use the AWS Management Console to view the messages in the DLQ.
Analyze the content for potential errors or reasons for failure.

Programmatically Retrieve Messages:

Use the AWS SDK to fetch messages from the DLQ for automated inspection and reprocessing.

Automating DLQ Redrive Setup

Manual configuration can be time-consuming and error-prone, especially when dealing with multiple queues. Automation ensures consistency across environments. Below is an example setup using Terraform.

resource "aws_sqs_queue" "main-queue" {
  name = "redriveBlogQueue"
}

resource "aws_sqs_queue" "dlq" {
  name = "RedrivBlog-dlq"
  redrive_allow_policy = jsonencode({
    redrivePermission = "byQueue",
    sourceQueueArns   = [aws_sqs_queue.main-queue.arn]
  })
}

resource "aws_sqs_queue_redrive_policy" "redrive" {
  queue_url = aws_sqs_queue.main-queue.id
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 4
  })
}

Conclusion
The DLQ Redrive process is essential for building resilient, fault-tolerant systems with Amazon SQS. By isolating problematic messages and automating their handling, you can ensure the stability of your application while gaining insights into recurring issues.

Implementing DLQs and automating their redrive process is a best practice for any distributed system using SQS. Start small with manual setups, and scale up with automation using tools like the AWS SDK, CloudWatch, and Terraform.

How are you leveraging DLQ redrives in your projects? Share your thoughts and challenges!

Next.js Deployment on AWS Lambda, ECS, Amplify, and Vercel: What I Learned

Ogbeide Godstime Osemenkhian — Mon, 13 Jan 2025 16:16:06 +0000

Overview

Next.js, a React-based framework for building server-side rendered (SSR) applications, has gained immense popularity due to its performance and flexibility. However, selecting the right platform to deploy your Next.js application can significantly impact cost, scalability, and development experience.

There are numerous platform options for deploying a Next.js application, each with unique strengths and tradeoffs. To determine the most suitable platform for different scenarios, I investigated deploying a sample Next.js app on AWS Lambda, AWS ECS, AWS Amplify, and Vercel.

In this blog, I’ll share my findings, deployment experiences, tradeoffs for each platform, and what I learned.

Acceptance Criteria

The following criteria guided the investigation:

Cost: Affordability for various levels of usage.
Complexity: The effort and expertise required for setup and maintenance.
Ease of Deployment: How straightforward it is to deploy Next.js.
Performance: Response times, handling traffic spikes, and latency.
Resilience: Fault tolerance and high availability.
Scalability: The platform’s ability to handle growing traffic seamlessly.
Securely Accessing Private Resources: Capability to integrate with internal APIs or databases.

AWS Lambda: A Serverless Approach
AWS Lambda offers a serverless computing service where you only pay for what you use—great for cost-conscious projects. There is no server management, no upfront costs, just code execution. However, deploying a Next.js application to Lambda isn’t straightforward due to Lambda’s 250MB size limit and cold start delays.

What I Learned
Next.js generates lots of files during execution, often exceeding the size limit. To address this, I used the AWS Lambda Web Adapter, which allowed me to proxy requests between the Lambda runtime and the web application without heavy code refactoring.

Deployment Process
Here’s the approach I took:

Install Tools:

SAM CLI
AWS CLI
Docker Dockerize the App: Include the AWS Web Adapter in your Dockerfile (this is done by copying from awsguru public ecr).

`COPY --from=public.ecr.aws/awsguru/aws-lambda-adapter:0.8.4 /lambda-adapter /opt/extensions/lambda-adapter`

**Modify next.config.js: **Add standalone mode to the next.config.js file for compatibility

const nextConfig = {
    output: 'standalone',
}

module.exports = nextConfig

SAM: Finally sam was used to build, package and deploy the app on Lambda using sam build && sam deploy

For a more robust setup, CI/CD pipelines (e.g., GitHub Actions) can automate the deployment process.

Key Takeaways

Cold Starts: While cold starts were present, the AWS Lambda Web Adapter helped mitigate their impact.
File Size Limitations: Next.js’s generated files can exceed Lambda’s 250MB limit, requiring Dockerization or optimizations.

Trade-offs

Pros: Cost-effective, serverless scaling, and pay-as-you-go pricing.
Cons: Cold starts and limitations on file sizes.

AWS ECS: Containerized Application Management

Amazon Elastic Container Service (ECS) is a fully managed container orchestration service. It allows you to deploy, manage, and scale containerized applications, either using Amazon EC2 or AWS Fargate, a serverless compute engine for containers.

What I Learned
Deployment to ECS requires containerizing the app and defining infrastructure components like tasks, services, and scaling configurations. While more complex than Lambda, ECS offers powerful tools for handling high traffic.

Deployment Process

Containerize the App: Build a Docker image for the app and push it to Amazon ECR.

Define Infrastructure: Create a task definition, service, and autoscaling configurations.

Deploy: I used Terraform for the deployment, AWS CLI or Cloudformation can also be used to create and manage resources.

Key Takeaways

Resilience and Scaling: ECS excels at handling traffic with proper autoscaling.
Infrastructure Management: More effort is needed compared to serverless solutions like Lambda.

Trade-offs

Pros: High performance, flexible scaling, and resilience.
Cons: Higher complexity and cost compared to Lambda.

AWS Amplify

AWS Amplify is a fully managed service that simplifies front-end and mobile app development. It offers features like CI/CD pipelines, hosting, and backend services. Amplify is optimized for static and server-side rendered apps, making it an excellent choice for Next.js.

What I Learned

Amplify abstracts away the complexity of hosting Next.js applications. With just a few clicks, I was able to connect my repository, configure build settings, and deploy.

Deployment Process

Connect Repository: Link the GitHub repository to Amplify.
Select Branch: Choose the branch to deploy.
Build and Deploy: Amplify detects the framework and runs the necessary build commands (yarn install, etc.).
Manage Resources: Use the Amplify UI for custom domains, redirects, and more.

Key Takeaways

Ease of Use: Amplify’s automated process was incredibly simple, making it perfect for developers without extensive AWS experience.
Scaling: Amplify handled traffic spikes well, demonstrating good scalability.

Trade-offs

Pros: Low complexity, high ease of use, and seamless CI/CD.
Cons: Limited support for securely accessing private resources.

Vercel: Next.js’s Native Home

Vercel is the team behind Next.js, so it’s no surprise that their platform is designed to handle it flawlessly. With native SSR support, automatic scaling, and seamless integration, Vercel offers the easiest deployment experience for Next.js developers.

What I Learned
Deploying with Vercel was incredibly simple. I connected my GitHub repository, clicked Deploy, and in less than 5 minutes, my app was live on a .vercel.app domain.

Deployment Process

Sign Up and Connect Repository: Create a Vercel account and link your GitHub repository.
Deploy the App: Click Deploy, and Vercel automatically builds and deploys the app.
Access the App: The app will be available on a .vercel.app domain, which can be customized.

Key Takeaways

Optimal Performance: Vercel’s integration with Next.js made it the fastest and easiest deployment platform.
Cold Starts: Some initial delays were observed, likely due to serverless function cold starts.

Trade-offs

Pros: Extremely easy to use, high performance, and rich features.
Cons: Higher cost for advanced features and limited private resource access on lower-tier plans.

Conclusion

Through this investigation, I learned that each platform has its unique strengths and trade-offs. Ultimately, the best platform depends on your application’s needs, team expertise, and budget. For my use case, ECS was the best fit as it allowed more control over the infrastructure and enabled secure access to backend services using VPC.

Serverless Amazon API Gateway

Ogbeide Godstime Osemenkhian — Sun, 12 Feb 2023 23:45:28 +0000

Introduction

In this post, I will take you through a high-level walkthrough of the Amazon API gateway. I will have a future post on demos and hands-on teaching on creating our own API gateway, connecting it to a server, and serverless backends, and securing our API gateways by implementation of best practices.

What is an API

An Application Programming Interface (API), acts as an entry point or a front door for applications to gain access to business logic, data, or functionalities from backend services. API design is usually seen as a client-server design, the app sending the request is seen as the client while the app sending the response is usually the server.
APIs are mechanisms that allow communication between software parts using set protocols and definitions. An example of such communication is a request to the weather bureau’s software system and the weather App on your phone to get the weather information of a place at a particular time, and you get a response to the request sent by seeing the weather information of the place requested on your app. The popular types of API integrations are websocket APIs and REST APIs.

What is REST?
REST stands for Representational state transfer and is meant to be an architectural reference for developing modern and user-friendly web services.
Instead of defining custom methods and protocols such as SOAP or WSDL, REST is based on HTTP as the transport protocol. HTTP is used to exchange textual representations of web resources across different systems, using predefined methods such as GET, POST, PUT, PATCH, DELETE, etc. JSON is its standard representation format.

Amazon API Gateway

Amazon API Gateway is a fully managed service that increases the flexibility for developers to create, publish, maintain, monitor, and secure APIs at any scale. API Gateway securely handles every task involved in accepting and processing up to hundreds of thousands of concurrent API calls, including access control, traffic management, CORS support, authorization, monitoring, throttling, and API version management.
Amazon API Gateway creates RESTful APIs that are HTTP based, enable stateless client-server communication, and use standard HTTP methods such as GET, POST, PUT, PATCH, and DELETE. It also creates Websocket APIs that Adhere to the WebSocket protocol, which enables stateful, full-duplex communication between client and server, and also routes incoming messages based on message content.
Amazon API Gateway enables you to design and build RESTful interfaces and helps connect them to your application backend. With API Gateway You can have the flexibility of designing your personal resource structure, adding dynamic routing parameters, and developing custom authorization logic. Each API resource can be configured independently, while each stage can have specific cache, throttling, and logging configurations.
API Gateway is a serverless service, which means you don't have to bother about provisioning the underlining resources. AWS provisions all the resources under the hood for you while you select all the implementation and configuration needed for your application. This helps to boost developers' productivity by ensuring that the focus is on building their code and not bothering about the underlining infrastructure.
Amazon API Gateway can integrate with various AWS services such as Lambda, AWS load balancer, AWS WAF, and Cloudfronts.
With Amazon API Gateway, you can define resources, map them to custom models, specify which methods are available (i.e. GET, POST, etc.), and eventually bind each method to a particular Lambda function. Alternatively, you can attach more than one method to one single Lambda function. This way, you will maintain fewer functions and partially avoid the cold-start Lambda issues.

Amazon API Gateway Architecture

Moving AWS RDS from Single AZ To Multi AZ

Ogbeide Godstime Osemenkhian — Fri, 30 Dec 2022 23:19:26 +0000

Overview

Amazon Relational Database Service (RDS) is a fully managed database service that makes it easy to set up, operate, and scale relational databases in the cloud. Each Amazon RDS instance runs on a DB instance backed by an Amazon Elastic Block Store (Amazon EBS) volume for storage.
RDS supports seven different database engine that includes; Amazon Aurora with MySQL compatibility, Amazon Aurora with PostgreSQL compatibility, MySQL, MariaDB, PostgreSQL, Oracle, and SQL Server.
For Production workload, it is very important to bake in high availability in database planning, to avoid single point of failure in the event of an availability zone failure and also to reduce latency for high write or read applications, by using read replicas in the high availability setup.
To ensure high availability, AWS RDS supports two different methods;

RDS Multi-AZ deployments for MySQL, MariaDB, PostgreSQL,
Oracle, and SQL Server engines.
For Amazon Aurora engines, data can be replicated six ways
in three different availability zones.
In this article our focus will be on RDS Multi-AZ deployment (Postgres Engine)

Types of RDS setup

Single-AZ Setup: In a Single-AZ setup, one RDS DB instance and one or more EBS storage volumes are deployed in one Availability Zone across data centre.
Multi-AZ setup: In a Multi-AZ configuration, RDS DB instances and EBS storage volumes are deployed across multiple Availability Zones. A multi-AZ deployment has a Master database in one AZ and a Standby (or Secondary) database in another AZ. Only the Master database serves traffic. If the Master fails, then the Secondary takes over.
Multi-AZ DB Cluster: A Multi-AZ DB cluster deployment is a high-availability deployment mode of Amazon RDS with two readable standby DB instances. A Multi-AZ DB cluster has a writer DB instance and two reader DB instances in three separate Availability Zones in the same AWS Region. Multi-AZ DB clusters provide high availability, increased capacity for reading workloads, and lower write latency when compared to Multi-AZ DB instance deployments.

Benefits

Single-AZ database deployment;

It is cost-efficient as you only pay for the database being used without bothering about the standby database
It can function optimally and serve applications if there is no fault. It allows automatic backup as specified in the schedule.
It is Ideal for the development environment as it helps save the cost of Development.

Multi-AZ database deployment;

Amazon RDS maintains a redundant and consistent standby copy of your data using synchronous storage replication.
Amazon RDS detects and automatically recovers from the most common failure scenarios for Multi-AZ deployments so that you can resume database operations as quickly as possible without administrative intervention.
Amazon RDS automatically performs a failover in the event of loss of availability in the primary Availability Zone, loss of network connectivity, compute unit failure, and storage failure.
Having separate Availability Zones greatly reduces the likelihood that copies of database are concurrently affected by most types of disturbances and provides a highly available database architecture.
There is Lower Latency with Multi-AZ deployment.

Multi-AZ Cluster database deployment;

With a Multi-AZ DB cluster, Amazon RDS replicates data from the writer DB instance to both of the reader DB instances using the DB engine's native replication capabilities.
Reader DB instances act as automatic failover targets and also serve read traffic to increase application read throughput. If an outage occurs on your writer DB instance, RDS manages failover to one of the reader DB instances.
Multi-AZ DB clusters have lower write latency when compared to Multi-AZ DB instance deployments.

Sample Multi-AZ setup

Risks
For projects running single AZ production RDS databases, and want to consider migrating to Multi-AZ Database deployments after reading this article, some things to keep in mind and account for before carrying out your migration are listed in the multi-AZ sessions below.
Single-AZ DB
It is not highly available. When there is a failure in the AZ the database is inaccessible for that period of time.

Multi-AZ DB

Cost of operation increases because the database has double the resources and infrastructure handling the Database. Depending on the number of availability zones, the cost for two availability zones is double the cost for a single AZ.
Though this does not occur all the time, there could be a latency issue during migration.
When Migrating RDS from a single-AZ to Multi-AZ, latency can sometimes occurs because when converting a DB instance from Single-AZ to Multi-AZ, Amazon RDS creates a snapshot of the database volumes and restores these to new volumes in a different Availability Zone. Although the newly restored volumes are available almost immediately, they don’t reach their specified performance until the underlying storage blocks are copied from the snapshot.
Therefore, during the conversion from Single-AZ to Multi-AZ, you can experience elevated latency and performance impacts. This impact is a function of volume type, workload, instance, and volume size, and can be significant and may impact large write-intensive DB instances during the peak hours of operations.
This latency hardly occurs because for multi-az deployment the primary database will still be operating optimally. Except there is a case of capacity overload on the primary database before the migration.

Multi-AZ Cluster
There may be an Initial Replica Lag in a multi-AZ cluster deployment. Replica lag is the difference in time between the latest transaction on the writer DB instance and the latest applied transaction on a reader DB instance.
You can create a Multi-AZ DB cluster only with MySQL version 8.0.28 and higher 8.0 versions, and PostgreSQL version 13.4.
You can create Multi-AZ DB clusters only in the following AWS Regions: US East (Ohio) US East (N. Virginia) US West (Oregon) Asia Pacific (Singapore) Asia Pacific (Sydney) Asia Pacific (Tokyo) Europe (Ireland) Europe (Frankfurt) Europe (Stockholm) as at the time of writing this.
Multi-AZ DB clusters only support Provisioned IOPS storage.
You can't change a single-AZ DB instance deployment or Multi-AZ DB instance deployment into a Multi-AZ DB cluster. As an alternative, you can restore a snapshot of a single-AZ DB instance deployment or a Multi-AZ DB instance deployment to a Multi-AZ DB cluster.
You can't restore a Multi-AZ DB cluster snapshot to a Multi-AZ DB instance deployment or single-AZ deployment.
Multi-AZ DB clusters don't support modifications at the DB instance level because all modifications are done at the DB cluster level.

Recommendation

Multi-Az deployment of a database is always the best practice to combat faults and failure and have a secure architecture in times of disaster. The production environment should always have multi-AZ in mind when deploying resources, especially databases. This would not only make it possible to still operate optimally when a region's data center goes down, but It will also enhance database accessibilities by applications as there are enough IOPS to serve the workload. Multi-AZ should be carried out, In the case where cost is a challenge, then resources can be right-sized to have two copies at a fair price.

test post

Ogbeide Godstime Osemenkhian — Fri, 30 Dec 2022 22:41:55 +0000