DEV Community: Cristián Andrés

[Boost]

Cristián Andrés — Thu, 23 Jan 2025 13:32:54 +0000

Copier vs Cookiecutter

Cloud Native Engineer ・ Jan 20

#beginners #programming #python #productivity

Installing Python Dependencies on AWS Lambda Using EFS

Cristián Andrés — Thu, 21 Nov 2024 14:37:40 +0000

When working with AWS Lambda, one of the common challenges developers face is managing large Python dependencies. Libraries like Pandas, Shapely, and GeoPandas, essential for tasks like geospatial analysis, often exceed Lambda's 250 MB unzipped layer limit. A practical solution? Store your dependencies on an EFS (Elastic File System) and mount it to your Lambda function.

In this post, we’ll walk through the process of setting this up, including the prerequisites, key benefits, and step-by-step implementation.

Prerequisites

This post is intended for users with advanced AWS experience. It assumes a solid understanding of AWS services such as Lambda, EFS, VPC, and security groups, as well as familiarity with managing infrastructure and deploying scalable solutions in the cloud.
Before we dive into the setup, ensure you have the following:

AWS Lambda Function: A deployed Lambda function you’ll configure with EFS.
EFS File System: An Elastic File System created in the same AWS region.
EFS access point: An EFS access point created in the same AWS region, with the root directory path to /data , Ensure you set the POSIX permissions and directory creation permissions appropriately as follows, 1101 and 1001, Secondary Group ID 1002 and Permissions 0755.
VPC and Networking: Ensure the Lambda function is in the same VPC as the EFS, with subnets and security groups properly configured.
IAM Permissions: Your Lambda function needs permissions to access the EFS. Attach the appropriate policy (e.g., elasticfilesystem:ClientMount, elasticfilesystem:ClientWrite).

Handler Code for Installing Packages

The handler installs Python dependencies directly on an Amazon EFS storage mounted to an AWS Lambda function. This approach bypasses the size limitations of Lambda layers, making it suitable for heavy dependencies like pandas, geopandas, and shapely, often required for geospatial data processing. It ensures the required libraries are available in the /mnt/data directory for Lambda to use during execution:

import os
import subprocess

PACKAGE_DIR = "/mnt/data/lib/{}/site-packages/"

def get_python_version_tag():
    """Generates a Python version tag like 'python3.11'."""
    return f"python{os.sys.version_info.major}.{os.sys.version_info.minor}"

def install_package(package):
    """Installs a Python package into the EFS-mounted directory."""
    target_dir = PACKAGE_DIR.format(get_python_version_tag())
    os.makedirs(target_dir, exist_ok=True)
    try:
        subprocess.run(
            [
                "pip",
                "install",
                package,
                "--target",
                target_dir,
                "--upgrade",
                "--no-cache-dir",
            ],
            check=True,
        )
        print(f"Package {package} installed successfully!")
    except subprocess.CalledProcessError as e:
        print(f"Failed to install package {package}: {e}")

def handler(event, context):
    """AWS Lambda Handler for installing packages."""
    try:
        # List of packages to install from the event input
        packages = event.get("packages", [])
        for package in packages:
            install_package(package)
        #optional for see packages installed
        #os.system(f"ls -la {PACKAGE_DIR.format(get_python_version_tag())}")
        return {"statusCode": 200, "body": "Packages installed successfully!"}
    except Exception as e:
        print(f"Error: {e}")
        return {"statusCode": 500, "body": f"An error occurred: {e}"}

Steps to Test

When invoking your Lambda function, pass the following JSON payload:

{
    "packages": ["requests", "pandas"]
}

Verify Package Installation

Navigate to your EFS mount point (e.g., /mnt/data/lib/) using an SSH session or AWS CLI.
Check the installed packages under the site-packages/ directory.
or simple use a for see a packages installed

os.system(f"ls -la {PACKAGE_DIR.format(get_python_version_tag())}")

Finally Use the Installed Dependencies in Lambda

Update your Lambda function’s handler to include the dependencies installed on EFS, the key here is mount the path of dependencies in efs to a PYTHONPATH of lambda handler:

Important Note

All Lambda functions that wish to use the installed dependencies must attach the EFS to the Lambda. Without this attachment, the Lambda will not be able to access the required dependencies stored on EFS.

import sys
sys.path.append("/mnt/data/lib/python3.11/site-packages/")  # Adjust Python version as needed
# Dependencies are now available!!!
import pandas as pd  

def lambda_handler(event, context):
    return {"message": "Dependencies loaded successfully!"}

Key Benefits

While installing Python dependencies directly in EFS is not a common practice, it offers certain advantages in scenarios where Lambda's default limitations, such as the 250 MB unzipped layer size, become restrictive. This approach is particularly beneficial for applications requiring geospatial computations with heavy libraries like Pandas, Shapely, and GeoPandas, which often exceed the layer size limit.

Benefits of Using EFS for Dependencies:

Bypass Lambda Layer Size Limits: Install and use libraries without worrying about packaging constraints.
Enable Large-Scale Geospatial Processing: Handle complex spatial computations in a serverless environment.
Streamline Dependency Management: Add or update libraries dynamically without redeploying your Lambda function.

This solution is ideal for advanced data processing tasks, such as geospatial analysis,also allows for the easy scaling of storage as required, while maintaining the flexibility of a serverless architecture.

🚀 Never Lose Your Clipboard History Again on Windows!

Cristián Andrés — Thu, 21 Nov 2024 13:47:22 +0000

Ever hit Windows + V only to realize your clipboard forgot the text you just copied? 🤯
By default, the Windows clipboard only stores a limited history, and after a busy day, finding that one specific copy-paste can feel impossible.

But don’t worry – we’ve got you covered! Introducing a tool that automatically saves everything you copy from your clipboard. Whether it's text, links, or more, this tool ensures you never lose track of your clipboard history again.

Ready to take control of your clipboard?

ditto-cp
Source Superuser

How can I use the python logging in Ray?

Cristián Andrés — Thu, 22 Feb 2024 16:24:57 +0000

Mar 21 '19 Comments: 1 Answers: 3

I use the logging module in the main function/process, it works well, but it seems can't work in Actor process/subprocess. How to make it work? In the sample below code, logging.info work in the main process but failed in the worker process. Thanks.

import logging
import ray

@ray.remote
class Worker(object):

…

Open Full Question

Migrando geometries con DMS

Cristián Andrés — Fri, 26 Jan 2024 01:14:53 +0000

Hola de nuevo, después de un buen tiempo desaparecido, ¡vuelvo con novedades en el mundo de los ETLs y pipelines automatizados!

Esta vez, me encontré con una agradable sorpresa: AWS DMS ha solucionado un bug en su última versión (3.5.1) que complicaba la migración de geometrías muy extensas entre bases PostgreSQL. Los que han lidiado con esto entenderán el alivio que representa para futuros procesos.

Por ahora, no está claro cómo afecta a otros destinos como S3 o Redshift, o formatos como Parquet, pero es probable que implique una transformación a WKT, lo que al menos facilita la lectura en Athena.

Eso es todo por hoy. Tres ETLs fallaron y tengo mucho que corregir... ¡pero ya habrá más tiempo para seguir compartiendo!

Answer: AWS Step Function - Dynamic parallelism MaxConcurrency Field

Cristián Andrés — Thu, 18 Jan 2024 23:22:53 +0000

Dec 18 '21

You can set maxConcurrency at state machine definition-time, but not at execution-time. As you experienced, Map's maxConcurrency expects a number, but the state machine language uses strings to pass variables dynamically.

(Note: Step Functions are concurrent by default. Docs: maxConcurrency's "default value is 0, which…

Open Full Answer

Full screen chrome WITHIN a window?

Cristián Andrés — Thu, 30 Nov 2023 13:15:44 +0000

Full screen chrome WITHIN a window?

open-as-popup

Smooth scroll in Visual Studio code

Cristián Andrés — Wed, 29 Nov 2023 00:35:17 +0000

Jun 20 '18

100

// Controls if the editor will scroll using an animation

  "editor.smoothScrolling": false

// Enable this workaround if scrolling is no longer smooth after restoring a minimized VS Code window. This is a workaround for an issue (https://github.com/Microsoft/vscode/issues/13612) where scrolling starts to lag on devices with precision trackpads like…

Open Full Answer

You miss Copilot ? Say hi GPT-3 pilot 🖖[deprecated]

Cristián Andrés — Fri, 06 Jan 2023 13:01:28 +0000

Hi, today I am sharing a cool tool that will allow you to quickly code with the programming language of your choice. This is the VS Code extension 'VSCode-ChatGPT'.

With this extension, programmers can write and run code quickly and easily. It offers helpful autocomplete moments, autocorrect to help catch language errors, and as a bonus (best in my opinion💎), a set of preconfigured templates for different specifics. Even if you've never programmed before, you may find ChatGPT's syntax wizard useful for learning the syntax of different languages.

I invite you to give it a try today to find out what it can do to improve your workflow. It's free and easy to use!

VSCode-ChatGPT

Answer: Impossible to get CRS with tomcat : org.opengis.referencing.NoSuchAuthorityCodeException

Cristián Andrés — Thu, 29 Dec 2022 08:50:35 +0000

Aug 8 '18

I found a workaround even if it doesn't solve the initial issue (related to dependencies of geotools and the deployement into Tomcat)

don't use gt-epsg-hsql which leads your app searching for the EPSG code to decode into the HSQL database.

Rather than using CRS.decode(targetCrs); with

CRS.parseWKT("PROJCS[\"WGS 84 / Plate Carree

…

Open Full Answer

Como ⚡ publico una udf para Athena ? Parte I

Cristián Andrés — Tue, 20 Dec 2022 23:54:44 +0000

Bueno esta fue la pregunta inicial que dará origen a la siguiente aventura, suponga que en su datalake implemento un complejo pipelile de datos y se dio cuenta que en medio del proceso su motor no cuenta con una funcionalidad sql 🙇, en mi caso fue st_buffer, por cierto que en Athena si esta implementada pero en grados decimales (iuhggg), por lo que necesitábamos la preciada función en metros.
Bueno luego de un corto googleo aparece el siguente Workshop de bastante utilidad y bien explicado:
Taller de Amazon Athena :: Hands on Labs

Para comenzar recomienda usar Cloud9, pero es yo utilice WSL Ubuntu 22 sin mayores problemas.
Bueno en simples pasos una vez que se configura el sam-aws-cli , credencial aws , maven y git , nos clonamos un repo el cual contiene un boilerplate con la estructura de un proyecto java 11 , que posee handler con algunos metodos a modo hello world, listo en este punto solo falta compilar el jar y ejecutar un sh para hacer un deploy de la aplicación , se necesita un permiso especial para poder hacerlo "serverlessrepo:CreateApplication" en el recurso "arn:aws:serverlessrepo:us-east-1::applications/*"

Listo se publicó la aplicación ahora qué ?, solo seguir el paso "Configuración del conector UDF" , en ese caso seguir el enlace y buscar aplicaciones privadas seleccionar "AthenaUserDefinedFunctions" y completar los datos , luego deployar , tómese una cerveza y Prepárese para probar su función UDF

USING EXTERNAL FUNCTION redact(col1 VARCHAR ) RETURNS VARCHAR LAMBDA 'customudf' SELECT redact('StringToBeCompressed');

Excelente nuestro hello world , próximamente en la parte II detallare el proceso para manipular datos geoespaciales y hacer el buffer con java, que esta algo fuera de mi zona de confort pero con GPT3 veremos si da alguna sorpresa.

Representing Directory & File Structure in Markdown Syntax

Cristián Andrés — Fri, 08 Jan 2021 13:47:58 +0000

Jul 18 '19

182

If you're using VS Code, this is an awesome extension for generating file trees.

Added directly to markdown...

📦quakehunter
 ┣ 📂client
 ┣ 📂node_modules
 ┣ 📂server
 ┃ ┗ 📜index.js
 ┣ 📜.gitignore
 ┣ 📜package-lock.json
 ┗ 📜package.json

Open Full Answer