DEV Community: Shubham Singh

Layering Approach (Lambda)

Shubham Singh — Wed, 28 May 2025 18:18:07 +0000

🧬 Layering Approach in YAML for AWS Lambda: Why It Matters

Serverless is awesome—until you hit the 50MB zipped size limit of AWS Lambda or have to reuse packages across multiple functions. That’s where Lambda Layers and the layering approach in YAML come in.

In this post, you’ll learn:

What Lambda Layers are
Why and when to use them
How to define them using SAM YAML
Pros, cons, and AWS Lambda limits

🧠 What Are Lambda Layers?

A Lambda Layer is a distribution mechanism for libraries, custom runtimes, and other dependencies. Instead of packaging everything inside your function zip, you split shared logic into reusable layers.

Real-world Example:

Say you have 5 Lambda functions using Pandas and NumPy. Rather than bundling them in each function (increasing size and build time), you create a shared layer containing those libraries.

📦 AWS Lambda Size Limits

Type	Max Size
Lambda code (zipped)	50MB (direct upload)
Lambda code (unzipped)	250MB (with layers)
Layer size (unzipped)	250MB per layer
Max Layers per Lambda	5

So if your function needs more than 50MB of libraries (like ML models or large packages), layers are the solution.

📄 YAML Layering with AWS SAM (Serverless Application Model)

🧱 Step 1: Define a Lambda Layer in `template.yaml`

Resources:
  MySharedLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: shared-python-packages
      Description: Shared Python libraries for all Lambdas
      ContentUri: layers/shared/
      CompatibleRuntimes:
        - python3.9
      RetentionPolicy: Retain

🧰 Folder Structure

project-root/
├── template.yaml
├── layers/
│   └── shared/
│       └── python/
│           ├── pandas/
│           ├── numpy/
│           └── other_lib/
├── functions/
│   ├── functionA/
│   │   └── app.py
│   └── functionB/
│       └── app.py

AWS requires your dependencies inside python/ (or nodejs/, ruby/, etc.)
Use a tool like pip install -r requirements.txt -t layers/shared/python to populate it.

🧾 Step 2: Reference the Layer in Your Lambda Function

  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.lambda_handler
      Runtime: python3.9
      CodeUri: functions/functionA/
      Layers:
        - !Ref MySharedLayer
      Events:
        HelloAPI:
          Type: Api
          Properties:
            Path: /hello
            Method: get

Now the function will inherit the layer and can import libraries as if they were local.

🚀 Benefits of Layering

Advantage	Description
✅ Reusability	One layer, many functions
✅ Smaller Deployment	Function ZIPs stay under 50MB
✅ Faster CI/CD	No need to repackage large libraries
✅ Security & Patching	Update the layer without touching code

⚠️ Drawbacks & Caveats

Disadvantage	Description
❌ Layer Limits	Max 250MB per layer; max 5 layers per function
❌ Versioning	Each update creates a new version (immutable)
❌ Cold Start Time	Larger layers can increase cold start latency
❌ Debugging Layers	Can be tricky to troubleshoot if layer fails silently

🧪 Testing Layers Locally

You can test layers locally using the AWS SAM CLI:

sam local invoke MyFunction --event events/input.json

Or use Docker to simulate the AWS Lambda environment:

docker run -v "$PWD":/var/task lambci/lambda:python3.9 app.lambda_handler

Make sure your layer path is in the Python path:

import sys
sys.path.append('/opt')
import pandas

/opt is the default mount path for Lambda layers.

🧱 Use Cases

Shared SDKs for internal services
ML Models and large Python dependencies
Custom Runtimes (e.g., Alpine builds)
Monitoring Tools (e.g., Datadog, New Relic)

⚙️ Pro Tip: Automate Layer Upload with Makefile

LAYER_NAME=my-layer
PYTHON_VERSION=python3.9

build-layer:
    pip install -r requirements.txt -t layers/shared/python

deploy-layer:
    aws lambda publish-layer-version \
        --layer-name $(LAYER_NAME) \
        --compatible-runtimes $(PYTHON_VERSION) \
        --zip-file fileb://layer.zip

📚 Summary

Feature	With Layer	Without Layer
Package Size	Smaller	Grows quickly
CI/CD Time	Faster	Slower
Code Duplication	Minimal	High
Setup Complexity	Slightly higher	Simpler initially
Max Runtime Support	Python, Node, etc.	Same

✅ Conclusion

Layering your Lambda functions using YAML gives you clean separation of concerns, optimized performance, and scalable deployment pipelines. While layers come with a few trade-offs, their benefits in large-scale or complex serverless systems are hard to ignore.

“Don’t just write functions—architect them.”

Docker & AWS Integration

Shubham Singh — Wed, 28 May 2025 17:40:01 +0000

🚀 Docker and AWS Integration: A Developer’s Complete Guide

Docker has revolutionized the way we build, ship, and run applications. Combine that with AWS, and you get a powerful, scalable infrastructure for deploying modern applications.

This guide explores how to integrate Docker with AWS, focusing on ECR (Elastic Container Registry), ECS (Elastic Container Service), and EC2—with real-world use cases and commands.

🧱 Why Docker + AWS?

Docker makes applications portable.
AWS gives scalable infrastructure.

Together, they allow you to build microservices, deploy quickly, and manage infrastructure easily.

⚙️ Prerequisites

Docker installed locally
AWS CLI configured (aws configure)
Basic knowledge of EC2, ECS, IAM, and VPC

🐳 Step 1: Dockerize Your Application

Let’s take a basic Python Flask app:

app.py:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello from Docker on AWS!"

Dockerfile:

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "app.py"]

requirements.txt:

flask

Build the Docker image:

docker build -t flask-aws-app .

📦 Step 2: Push Docker Image to AWS ECR

✅ Create an ECR Repository

aws ecr create-repository --repository-name flask-aws-app

✅ Authenticate Docker with ECR

aws ecr get-login-password | docker login --username AWS --password-stdin <your-account-id>.dkr.ecr.<region>.amazonaws.com

✅ Tag & Push Image to ECR

docker tag flask-aws-app:latest <your-account-id>.dkr.ecr.<region>.amazonaws.com/flask-aws-app

docker push <your-account-id>.dkr.ecr.<region>.amazonaws.com/flask-aws-app

Now your image is stored in AWS.

🚀 Step 3: Deploy Docker Image with ECS (Fargate)

Amazon ECS is a container orchestration service. Use it to run your containerized app on AWS without managing servers.

🧭 Setup ECS with Fargate (Simplified via Console or CLI)

Create a cluster (Fargate type)
Create a Task Definition

Use ECR image URI
Assign memory/CPU
1. Create Service
Attach to VPC and subnets
Use a Load Balancer (optional)
1. Run the Service

✅ Now your container is running in ECS.

🖥️ Alternative: Run Docker on EC2

You can manually deploy Docker containers on EC2 instances.

Step-by-step:

# SSH into EC2
ssh ec2-user@<ec2-instance-ip>

# Install Docker
sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo usermod -a -G docker ec2-user

# Pull and run image from ECR
$(aws ecr get-login --no-include-email)
docker pull <account-id>.dkr.ecr.<region>.amazonaws.com/flask-aws-app
docker run -d -p 80:5000 flask-aws-app

You now have Docker running on an EC2 instance.

🔄 Bonus: Automate with CI/CD (GitHub Actions + ECR + ECS)

.github/workflows/deploy.yml:

name: Deploy to AWS ECS

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Docker
        uses: docker/setup-buildx-action@v1

      - name: Log in to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v1

      - name: Build, tag, and push image to ECR
        run: |
          docker build -t flask-aws-app .
          docker tag flask-aws-app:latest ${{ secrets.ECR_REPOSITORY }}
          docker push ${{ secrets.ECR_REPOSITORY }}

      - name: Deploy to ECS
        uses: aws-actions/amazon-ecs-deploy-task-definition@v1
        with:
          task-definition: ecs-task.json
          service: my-service
          cluster: my-cluster
          wait-for-service-stability: true

Use GitHub Secrets to store ECR and AWS credentials.

🧠 Real-World Use Cases

Microservices: Deploy isolated services on ECS or EC2
Serverless Containers: Use Fargate for true serverless infra
CI/CD Pipelines: Build images and push on every commit
Scaling: Use auto-scaling policies in ECS to grow traffic

🛡️ Tips and Best Practices

Use ECR Lifecycle Policies to clean up old images.
Define resource limits in ECS Task Definitions.
Enable CloudWatch Logs for your containers.
Use IAM roles for tasks to give least-privilege access.
Use Load Balancer for zero-downtime deployments.

📚 Summary

Feature	ECR	ECS	EC2 with Docker
Stores Images	✅	❌	❌
Orchestrates	❌	✅	❌
Full control	❌	Partial	✅
Managed Infra	✅	✅	❌

✅ Conclusion

Docker + AWS is the go-to combination for building and scaling modern applications. Whether you're deploying a microservice or a full stack system, you can containerize locally and deploy globally using AWS.

Try both ECS and EC2 methods and explore what works best for your architecture. Keep it containerized, scalable, and cloud-ready!

API Rate Limiting

Shubham Singh — Wed, 28 May 2025 17:24:43 +0000

Understanding API Rate Limiting and How to Implement It

APIs are the backbone of modern web services. Whether it's weather data, payment processing, or social media interactions, APIs help us exchange data quickly and securely.

But with great power comes great responsibility.

To ensure fair use and prevent abuse, API rate limiting is crucial. In this blog, we'll break down what API rate limiting is, why it's important, and how to implement common strategies like Token Bucket and Leaky Bucket in Python.

🧠 What is API Rate Limiting?

Rate limiting controls how many requests a client can make to an API in a given time frame.

Example:

"You can make 60 requests per minute."
"No more than 1000 requests per day."

Why use it?

Protects APIs from abuse and DDoS attacks
Helps maintain performance and uptime
Ensures fair usage among clients

🧩 Common Algorithms for Rate Limiting

1. Fixed Window Counter

In this approach, a counter tracks the number of requests during a fixed time window (e.g., 60 seconds).

✅ Easy to implement
❌ Can result in bursts at window edges

from time import time

class FixedWindowRateLimiter:
    def __init__(self, max_requests, window_size):
        self.max_requests = max_requests
        self.window_size = window_size
        self.window_start = int(time())
        self.request_count = 0

    def allow_request(self):
        current_time = int(time())
        if current_time - self.window_start >= self.window_size:
            self.window_start = current_time
            self.request_count = 0

        if self.request_count < self.max_requests:
            self.request_count += 1
            return True
        return False

2. Sliding Window Log

Instead of a fixed window, this keeps a timestamp log of requests and removes outdated ones.

✅ More accurate than Fixed Window
❌ Uses more memory (O(n) for n requests)

from collections import deque
from time import time

class SlidingWindowRateLimiter:
    def __init__(self, max_requests, window_size):
        self.max_requests = max_requests
        self.window_size = window_size
        self.request_log = deque()

    def allow_request(self):
        current_time = time()
        while self.request_log and self.request_log[0] <= current_time - self.window_size:
            self.request_log.popleft()

        if len(self.request_log) < self.max_requests:
            self.request_log.append(current_time)
            return True
        return False

3. Token Bucket Algorithm

Imagine a bucket that fills up with tokens over time. Each request consumes one token. If no tokens are left, the request is denied.

✅ Allows burst traffic
✅ Easy to tune
❌ Slightly more complex

from time import time

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_rate = refill_rate
        self.last_refill = time()

    def allow_request(self):
        now = time()
        elapsed = now - self.last_refill
        refill = elapsed * self.refill_rate
        self.tokens = min(self.capacity, self.tokens + refill)
        self.last_refill = now

        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

capacity: maximum tokens
refill_rate: tokens added per second

4. Leaky Bucket Algorithm

This approach is like a water bucket with a leak. Requests are queued, and only allowed out at a fixed rate.

✅ Smooth traffic flow
❌ Can delay burst traffic

from collections import deque
from time import time, sleep

class LeakyBucket:
    def __init__(self, leak_rate):
        self.queue = deque()
        self.last_checked = time()
        self.leak_rate = leak_rate

    def allow_request(self):
        now = time()
        elapsed = now - self.last_checked

        leaks = int(elapsed * self.leak_rate)
        for _ in range(leaks):
            if self.queue:
                self.queue.popleft()

        self.last_checked = now
        if len(self.queue) < self.leak_rate * 2:  # max burst size
            self.queue.append(now)
            return True
        return False

🛠️ Applying Rate Limiting to API Routes (Flask Example)

from flask import Flask, request, jsonify

app = Flask(__name__)
limiter = TokenBucket(capacity=10, refill_rate=1)  # 10 reqs, 1 per sec

@app.route('/api/data')
def get_data():
    if limiter.allow_request():
        return jsonify({"data": "Here is your API data"})
    else:
        return jsonify({"error": "Rate limit exceeded"}), 429

💡 Best Practices

Use Headers: Send X-RateLimit-Remaining, X-RateLimit-Reset headers for client transparency.
Different Limits for Users: Use API keys or IPs to set custom limits.
Persistent Storage: For distributed apps, store counters in Redis or a database.
Fail-Safe Defaults: Never fully block system-critical clients (use lower rate but allow some access).

✅ Summary

Algorithm	Accuracy	Memory	Burst Support	Use When…
Fixed Window	Low	Low	❌	Simple APIs, low traffic
Sliding Window	Medium	Medium	❌	Need better accuracy
Token Bucket	High	Low	✅	Need burst handling
Leaky Bucket	High	Medium	❌	Require smooth traffic output

📦 Try It Yourself

You can test these rate limiters in your own Flask or FastAPI projects. Want to scale up? Use Redis, API Gateway, or Nginx with Lua for advanced rate limiting in production environments.

“Build secure APIs, not open floodgates.”

🚀 Let Me Know

Was this helpful? Let me know your thoughts or which algorithm you want implemented with Redis or FastAPI next!

DEV Community: Shubham Singh

Layering Approach (Lambda)

🧬 Layering Approach in YAML for AWS Lambda: Why It Matters

🧠 What Are Lambda Layers?

Real-world Example:

📦 AWS Lambda Size Limits

📄 YAML Layering with AWS SAM (Serverless Application Model)

🧱 Step 1: Define a Lambda Layer in template.yaml

🧰 Folder Structure

🧾 Step 2: Reference the Layer in Your Lambda Function

🚀 Benefits of Layering

⚠️ Drawbacks & Caveats

🧪 Testing Layers Locally

🧱 Use Cases

⚙️ Pro Tip: Automate Layer Upload with Makefile

📚 Summary

✅ Conclusion

Docker & AWS Integration

🚀 Docker and AWS Integration: A Developer’s Complete Guide

🧱 Why Docker + AWS?

⚙️ Prerequisites

🐳 Step 1: Dockerize Your Application

📦 Step 2: Push Docker Image to AWS ECR

✅ Create an ECR Repository

✅ Authenticate Docker with ECR

✅ Tag & Push Image to ECR

🚀 Step 3: Deploy Docker Image with ECS (Fargate)

🧭 Setup ECS with Fargate (Simplified via Console or CLI)

🖥️ Alternative: Run Docker on EC2

Step-by-step:

🔄 Bonus: Automate with CI/CD (GitHub Actions + ECR + ECS)

🧠 Real-World Use Cases

🛡️ Tips and Best Practices

📚 Summary

✅ Conclusion

API Rate Limiting

Understanding API Rate Limiting and How to Implement It

🧠 What is API Rate Limiting?

🧩 Common Algorithms for Rate Limiting

1. Fixed Window Counter

2. Sliding Window Log

3. Token Bucket Algorithm

4. Leaky Bucket Algorithm

🛠️ Applying Rate Limiting to API Routes (Flask Example)

💡 Best Practices

✅ Summary

📦 Try It Yourself

🚀 Let Me Know

🧱 Step 1: Define a Lambda Layer in `template.yaml`