DEV Community: Samuel Vazquez

AWS Lambda Strategies - Auto-Calling Like a Pro 🚀🔥

Samuel Vazquez — Wed, 09 Apr 2025 17:08:33 +0000

Tired of manually handling batch processing in AWS Lambda? Wanna scale like a boss without worrying about timeouts? Let’s talk about auto-calling Lambda functions and how they can supercharge your workflows! 💪

The Problem

When dealing with large datasets in Lambda, processing everything in one go can lead to timeouts and memory issues. We need a better way—one that leverages asynchronous execution and auto-invocation to scale effortlessly.

Why This Matters

Imagine you have a list of resources in your database, and each needs AI-driven analysis, file complex generation or a high math effort. Running all of them in a single function execution? Not ideal. Instead, we’ll trigger the function recursively, allowing it to handle each resource independently, in parallel. No more memory bloat, no more slow executions. Just pure efficiency. 🫡

Lambda Sample:

#Lambda name autocalling-lambda-sample
import json
import boto3
import random

def get_waifu_list():
    return ["Asuna", "Rem", "Hinata", "Zero Two", "Nezuko", "Mikasa", "Kurisu"]

def custom_ai_analysis(name):
    personality_traits = {
        "Asuna": "Brave, Loyal, Strong-willed",
        "Rem": "Devoted, Kind, Fierce when needed",
        "Hinata": "Shy, Gentle, Loving",
        "Zero Two": "Mysterious, Playful, Protective",
        "Nezuko": "Cute, Silent, Deadly when angered",
        "Mikasa": "Stoic, Determined, Fierce",
        "Kurisu": "Smart, Witty, Tsundere"
    }
    return {"name": name, "personality": personality_traits.get(name, "Unknown")}

def lambda_handler(event, context):
    lambda_client = boto3.client('lambda')
    function_name = context.function_name
    waifu_list = get_waifu_list()

    if 'waifu_index' in event:
        waifu_name = waifu_list[event['waifu_index']]
        analysis_result = custom_ai_analysis(waifu_name)
        print(f"Processed: {json.dumps(analysis_result)}")
        return {'statusCode': 200, 'body': json.dumps(analysis_result)}
    else:
        for i in range(len(waifu_list)):
            lambda_client.invoke(
                FunctionName=function_name,
                InvocationType='Event',  # Asynchronous invocation
                Payload=json.dumps({'waifu_index': i})
            )
            print(f"Triggered async invocation for waifu {waifu_list[i]}")

        return {'statusCode': 200, 'body': json.dumps('Waifu analysis triggered!')}

Don forget:

Adding to your lambda the rights to auto call him with a custom policy like this:

Debug:

Enable your CloudWatch logs to see how your lambda was called n times and enjoy !

Why This Is Awesome 😎

✨ Scales Automatically – Each invocation only processes a single resource, preventing timeouts.
✨ Parallel Execution – Multiple Lambda instances run at the same time.
✨ Cost-Efficient – Pay only for what you use, avoiding unnecessary execution time.
✨ Works on Any Dataset – Whether you have 10 or 10 million resources, this approach just works.

And That's It!

💬 Got questions or improvements? Drop a comment below!
🔥 If this helped you, give it a ❤️ and share it with other Developers!

🔗 Connect With Me!

💼 LinkedIn: CodexMaker
📂 GitHub: CodexMaker
🎥 YouTube: CodexMaker

Have questions or improvements? Drop a comment below! 🚀🔥

AWS Lambda Layers with Docker Like a Pro 🥵

Samuel Vazquez — Thu, 03 Apr 2025 23:21:48 +0000

Tired of "No module found" errors in your Lambda functions? Architecture mismatches got you down? Let me show you how to build Lambda layers the right way! 💪

The Problem

Building Lambda layers locally often leads to compatibility issues, especially when your development machine doesn't match your Lambda's architecture (x86_64 vs ARM64).

Our use case and why this matters

For this example lets study a bit the Langfuse library for python if you are a Windows user and you try to generate this in your local independent if is a virtual environment or standard, once you generate the zip this will crash once you upload it as a layer and link it with your lambda due the pydantic_core.

This approach ensures that compiled modules (like pydantic_core) are built for the exact architecture your Lambda uses, eliminating those frustrating "no module found" errors.

Lets avoid the arch mistmatches forever 🫧💗✨

The Solution

I created this simple Docker Compose setup that builds Lambda layers with the exact same environment as AWS Lambda:

AWS Official images link : (https://gallery.ecr.aws/sam/)

version: '3.8'

services:
  lambda-builder:
    image: public.ecr.aws/sam/build-python3.12:latest-arm64 
    volumes:
      - .:/var/task
    command: >
      bash -c "
        echo 'Building Lambda dependencies for ARM64...' &&
        # Install dependencies directly in the root directory where Lambda will look for them
        pip install langfuse -t /var/task/package/ &&
        cd /var/task/package &&
        zip -r /var/task/deployment-package.zip . &&
        cd /var/task &&
        if [ -f 'lambda_function.py' ]; then
          zip -g /var/task/deployment-package.zip lambda_function.py; 
        else
          echo 'Warning: lambda_function.py not found in the current directory';
        fi &&
        if [ -f '.env' ]; then
          zip -g /var/task/deployment-package.zip .env;
        fi &&
        echo 'Deployment package created at: /var/task/deployment-package.zip' &&
        ls -la /var/task/deployment-package.zip
      "

How to Use It

1️⃣ Save the Docker Compose file in your project folder
Create a new directory with the above docker-compose.yml file

mkdir lambda-layers-generator

2️⃣ Run the build process

docker-compose up

3️⃣ Upload the generated deployment package
The output file 'deployment-package.zip' will be in your project directory, ready to upload to AWS Lambda!

What's Actually Happening?

Uses AWS's official Lambda container image (identical runtime environment)
Installs dependencies directly at the root level (where Lambda expects them)
Creates a properly structured ZIP package with all necessary files
Works for both ARM64 and x86_64 (just change the image tag)
No more "but it works on my machine" moments! 😅

And That's It!

💬 Got questions or improvements? Drop a comment below!
🔥 If this helped you, give it a ❤️ and share it with other Developers!

🔗 Connect With Me!

💼 LinkedIn: CodexMaker
📂 GitHub: CodexMaker
🎥 YouTube: CodexMaker

Have questions or improvements? Drop a comment below! 🚀🔥

AWS #Lambda #DevOps #Python #CloudComputing #Docker

Move Your Hugging Face LLM to S3 Like a Pro (Without Wasting Local Space!) 🚀

Samuel Vazquez — Sat, 08 Mar 2025 01:56:26 +0000

So you've got this huge LLM (we're talking 100+ GB of pure AI magic), and you need to get it into AWS S3 without clogging up your disk?

Let's do it the smart way—download, upload, and free up your space in one go! 💪

Best part? This guide works inside AWS SageMaker (Using a Jupyter Notebook) or your local machine with AWS CLI configured.

🔧 Prerequisites

Before we start, make sure you have:
👉 AWS CLI configured (if running locally)
👉 SageMaker Notebook (if using AWS)
👉 Hugging Face & Boto3 installed:

!pip install huggingface_hub boto3

Then, import the necessary libraries:

import os
import boto3
from huggingface_hub import hf_hub_download

🎯 The Mission

Download model files from Hugging Face.
Upload them to an AWS S3 bucket.
Free up local storage by removing the files after upload.
Clean up Hugging Face cache to reclaim disk space.

📥 Step 1: Set Up S3 and Model Paths

s3_client = boto3.client('s3')

# CHANGE this to your bucket
BUCKET_NAME = 'your-s3-bucket-name'  

# CHANGE this to your bucket
MODEL_PATH = "deepseek" 

# Make sure you have enough space!
SAVE_DIR = "/home/sagemaker-user/"  

# CHANGE from your selected model at https://huggingface.co/
repo_id = "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"

We've listed all the files to grab from Hugging Face:

List all your transformers and the required data the LLM needs, in this case each transformer goes around 9GB * 17 resources this model have is a total of 153GB not as good for a single file download

model_files = [
    "model-00001-of-000017.safetensors", 
    "model-00002-of-000017.safetensors", ...
    "tokenizer.json", 
    "tokenizer_config.json"
]

📄 Step 2: Upload to S3

Define a function to upload and other to delete files:

def upload_to_s3(local_file_path, s3_key):
    s3_client.upload_file(local_file_path, BUCKET_NAME, s3_key)
    print(f"Uploaded {local_file_path} to S3 bucket {BUCKET_NAME} with key {s3_key}")

def remove_local_file(file_path):
    try:
        os.remove(file_path)
        print(f"Removed local file {file_path}")
    except Exception as e:
        print(f"Error removing file {file_path}: {str(e)}")

Now, download, upload, and delete locally each file:

for file_name in model_files:
    local_file_path = hf_hub_download(repo_id=repo_id, filename=file_name, local_dir=SAVE_DIR)
    print(f"Downloaded {file_name} to {local_file_path}")

    s3_key = f"models/deepseek/{file_name}"
    upload_to_s3(local_file_path, s3_key)

    remove_local_file(local_file_path)

🗑 Extra Step: Clear Hugging Face Cache and debug your storage

import shutil

def clear_huggingface_cache(cache_dir):
    """Clear Hugging Face cache directory"""
    try:
        shutil.rmtree(cache_dir)
        print(f"Cleared Hugging Face cache at {cache_dir}")
    except Exception as e:
        print(f"Error clearing cache: {str(e)}")

clear_huggingface_cache("/home/sagemaker-user/.cache/huggingface")

def check_disk_space(directory):
    """Log disk space for a specific directory"""
    total, used, free = shutil.disk_usage(directory)
    print(f"Disk space for {directory} - Total: {total} GB, Used: {used} GB, Free: {free} GB")

check_disk_space("/home/sagemaker-user/.cache")

And That's It!

✅ Your Model is now safely stored in S3.
✅ Your local disk is free of clutter.
✅ You're ready to deploy or fine-tune without worrying about storage!🚀

💬 Got questions or improvements? Drop a comment below!
🔥 If this helped you, give it a ❤️ and share it with other LLM builders!

🔗 Connect With Me!

💼 LinkedIn: CodexMaker
📂 GitHub: CodexMaker
🎥 YouTube: CodexMaker

Have questions or improvements? Drop a comment below! 🚀🔥

"Remember AI is poetry, words become numbers, numbers will shape the
future"
Codexmaker