DEV Community: juan jose orjuela

Modelos fundacionales y Amazon Bedrock: ajustando las perillas de la IA como si fuera un equipo de sonido

juan jose orjuela — Sun, 22 Mar 2026 03:19:52 +0000

Cuando empecé a jugar con modelos de IA en AWS, me di cuenta rápido de algo: usar un modelo fundacional “en bruto” es como comprar un televisor 4K y nunca tocar los ajustes de imagen. Funciona, sí, pero te estás perdiendo la mejor parte.

En este post quiero contarte, como desarrollador a desarrollador, qué son los modelos fundacionales, qué es Amazon Bedrock y, sobre todo, cómo usar los parámetros de inferencia (temperature, top‑p, top‑k, longitud, etc.) para que el modelo haga más exactamente lo que tú quieres. Nada de fórmulas mágicas, solo perillas que vale la pena entender.

¿Qué es un modelo fundacional?

Un modelo fundacional (Foundation Model, FM) es un modelo de IA muy grande, entrenado con cantidades ridículas de texto, imágenes y otros datos, para aprender patrones generales del mundo.

Me gusta verlo así:

Es como alguien que se ha leído “todo Internet” y ahora puede escribir, resumir, traducir, razonar y hasta generar imágenes, sin que tú tengas que entrenarlo desde cero.

En Amazon Bedrock tienes un “catálogo” de estos modelos de diferentes proveedores:
Claude (Anthropic), Titan (Amazon), Llama (Meta), Mistral, etc.

Algunos son mejores escribiendo texto largo.
Otros son más rápidos y ligeros para chat en tiempo real.
Otros se especializan en imágenes o en generar embeddings.

Tú eliges el modelo como quien elige la herramienta en una caja de herramientas: para clavar clavos uso un martillo, no un destornillador.

¿Qué es Amazon Bedrock?

Amazon Bedrock es el servicio de AWS que te da acceso gestionado a esos modelos fundacionales a través de una API.

AWS se encarga de:

Infraestructura (GPU, escalado, disponibilidad).
Seguridad, autenticación, cuotas.
Catálogo de modelos y nuevas versiones.

Tú solo te preocupas de:

Elegir el modelo.
Enviar prompts.
Ajustar parámetros.
Integrarlo en tu aplicación.

Piensa en Bedrock como un Netflix de modelos de IA: tú no gestionas los servidores de streaming; solo eliges qué contenido ver (qué modelo usar) y con qué calidad (parámetros).

Los “controles remotos” de la IA: parámetros de inferencia
Cuando llamas a un modelo en Bedrock, no solo le mandas el prompt; también puedes ajustar una serie de parámetros que afectan cómo responde.

La metáfora que uso siempre es la de un equipo de sonido:

El prompt es la canción.
El modelo es el amplificador.
Los parámetros son las perillas de graves, agudos y volumen.

La canción es la misma, pero si mueves las perillas, la experiencia cambia mucho.

Temperature: cuánta creatividad quieres

Qué hace: controla qué tan “arriesgado” es el modelo al elegir la siguiente palabra.
Valores bajos (0.1–0.3): respuestas muy predecibles, casi siempre iguales.
Valores medios (0.4–0.7): equilibrio entre creatividad y control.
Valores altos (0.8–1): respuestas más creativas, pero también más “locas”.

Analogía:

Es como la persona que siempre pide lo mismo en el restaurante (temperature baja) vs. la que cada vez prueba algo nuevo del menú (temperature alta).

Uso típico:

Descripciones de producto que sigan una guía de marca → temperature ~0.4–0.6.
Brainstorming de ideas locas → temperature ~0.8–0.9.
Generar código muy preciso → temperature ~0.1–0.3.

top‑p y top‑k: qué tan amplio es el menú
Estos dos controlan de qué conjunto de palabras posibles puede elegir el modelo.

top‑k: “elige solo entre las k palabras más probables”.

k pequeño → modelo muy enfocado.
k grande → más diversidad.

top‑p (o nucleus sampling): “elige entre las palabras que, juntas, suman p de probabilidad acumulada”.

p bajo (~0.5–0.8) → más conservador.
p alto (~0.9–1.0) → más diverso.

Analogía:

Si temperature es qué tan aventurero eres,
top‑k/top‑p es qué tan grande es la carta del restaurante que estás dispuesto a mirar.

En muchas apps con Bedrock, un buen punto de partida es:

temperature: 0.5
top‑p: 0.8
top‑k: 20–50

y luego ajustar según veas las respuestas.

Longitud de respuesta y stop sequences
Además de qué dice el modelo, puedes controlar cuánto dice y dónde se detiene.

maxTokens / response length: límite máximo de tokens/palabras que puede generar. “Entre 50 y 100 palabras para la descripción de producto” → ajustas este valor.
Stop sequences: cadenas de texto que indican “hasta aquí, gracias”. Por ejemplo, "\n\nUser:" para que pare antes de volver a mostrar el nombre del usuario.

Analogía:

Es como poner un temporizador al horno: no quieres que el pastel se siga horneando hasta quemarse.

Casos de uso:

Chatbots: evitar que el modelo hable demasiado y rompa la experiencia.
Integraciones con sistemas legados: cortar la salida justo donde la aplicación espera un delimitador.

Ejemplos de casos de uso en Bedrock

Generador de descripciones de producto Imagina que tienes un e‑commerce y quieres generar descripciones entre 50 y 100 palabras, creativas pero alineadas con la marca.

Configuración típica:

Modelo: Claude o Titan (texto).
temperature: 0.5 (creatividad controlada).
top‑p: 0.8.
maxTokens configurado para ~100 palabras.

Prompt del estilo:

{
  "prompt": "Eres el redactor de la marca X. Escribe una descripción de 50 a 100 palabras para este producto, manteniendo un tono cercano y profesional. Producto: {{nombre}}, características: {{caracteristicas}}"
}

Resultado: textos que no son copias entre sí, pero se sienten de la misma familia.

Asistente de soporte más “serio” Para un chatbot de soporte técnico quieres menos creatividad y más precisión.

temperature: 0.2–0.3.
top‑p: 0.5–0.7.
Stop sequence para cortar al final de la respuesta.

Analogía:

Aquí el modelo es como el amigo que te ayuda a hacer la declaración de impuestos: mejor que no improvise demasiado.

Brainstorming de ideas de campaña Si estás en fase de ideación de marketing, quieres justo lo contrario.

temperature: 0.8–0.9.
top‑p: 0.9–1.0.
maxTokens más alto para dejarlo explayarse.

Prompt ejemplo:

{
  "prompt": "Dame 10 ideas creativas y poco convencionales para una campaña en redes sociales sobre {{tema}}. Incluye una breve explicación para cada una."
}

Cómo empezar a jugar con estos parámetros en Bedrock

Mi recomendación práctica:

Abre el playground de Amazon Bedrock en la consola.
Elige un modelo de texto (por ejemplo Claude).
Escribe un mismo prompt y prueba combinaciones:
temperature 0.1, 0.5, 0.9.
top‑p 0.6 vs 0.9.
Cambia maxTokens y mira cómo se recortan las respuestas.
Observa cómo cambian el tono, la diversidad y la precisión.

Hazlo como si ajustaras el brillo, contraste y saturación de una foto: no necesitas entender todas las ecuaciones detrás, pero sí qué sensación te produce cada cambio.

Cierre: no es magia, son perillas

Lo bonito de trabajar con modelos fundacionales en Amazon Bedrock es que no tienes que ser investigador en IA para sacarle partido.

Los modelos ya vienen entrenados.
Bedrock te quita de encima el dolor de cabeza de la infraestructura.
Los parámetros de inferencia son tus perillas para adaptar la IA a tu caso de uso concreto. Si piensas en ellos como en el ecualizador de tu playlist favorita, el resto es práctica: prueba, escucha, ajusta, repite.

Si te interesa, en otro post puedo entrar en temas un poco más avanzados: cómo combinar estos parámetros con prompt engineering, RAG (recuperación aumentada) y guardrails para construir sistemas que no solo sean creativos, sino también seguros y alineados con tu negocio.

Introducción a Grafana

juan jose orjuela — Wed, 15 Oct 2025 16:32:27 +0000

Una guía completa para principiantes sobre la instalación e inicio con Grafana en macOS.

1️⃣ Introducción

¿Qué es Grafana?

Grafana es una plataforma de código abierto para monitoreo y observabilidad que te permite consultar, visualizar, alertar y comprender tus métricas sin importar dónde estén almacenadas. Proporciona una manera poderosa y elegante de crear, explorar y compartir dashboards con tu equipo y fomentar una cultura basada en datos.

¿Para qué se usa Grafana?

Grafana se utiliza comúnmente para:

Monitoreo de infraestructura: Rastrear CPU, memoria, uso de disco y métricas de red
Monitoreo del rendimiento de aplicaciones (APM): Supervisar la salud y el rendimiento de las aplicaciones
Análisis de negocio: Visualizar KPIs y métricas empresariales
Visualización de datos IoT: Mostrar datos de sensores y métricas de dispositivos
Análisis de logs: Agregar y analizar datos de registros de múltiples fuentes

Arquitectura y Componentes Principales

La arquitectura de Grafana consiste en varios componentes clave:

Fuentes de Datos (Data Sources): Conexiones a bases de datos de series temporales y otros almacenes de datos (Prometheus, InfluxDB, Elasticsearch, MySQL, PostgreSQL, etc.)
Dashboards: Colecciones de paneles que muestran visualizaciones de tus datos
Paneles (Panels): Componentes de visualización individuales (gráficos, tablas, mapas de calor, etc.)
Consultas (Queries): Solicitudes de datos enviadas a tus fuentes de datos configuradas
Alertas (Alerting): Sistema de notificaciones basado en reglas para alertarte cuando las métricas alcanzan ciertos umbrales
Usuarios y Equipos: Control de acceso basado en roles para gestionar permisos
Plugins: Arquitectura extensible que soporta fuentes de datos, paneles y aplicaciones personalizadas

2️⃣ Instalación en macOS

Esta guía utiliza Homebrew, el popular gestor de paquetes para macOS. Si no tienes Homebrew instalado, visita brew.sh para obtener instrucciones de instalación.

Requisitos Previos

Asegúrate de que Homebrew esté instalado en tu sistema:

brew --version

Deberías ver una salida similar a: Homebrew 4.x.x

Pasos de Instalación

Paso 1: Instalar Grafana

Abre tu terminal y ejecuta:

brew install grafana

Salida esperada:

==> Downloading grafana...
==> Pouring grafana--12.2.0.arm64_sequoia.bottle.1.tar.gz
🍺  /opt/homebrew/Cellar/grafana/12.2.0: 10,910 files, 625.8MB

💡 Consejo: La instalación incluye todas las dependencias necesarias y tomará algunos minutos dependiendo de tu conexión a internet.

Paso 2: Verificar la Instalación

Comprueba que Grafana se instaló correctamente:

brew list | grep grafana

Deberías ver grafana en la salida.

Paso 3: Iniciar Grafana

Tienes dos opciones para iniciar Grafana:

Opción A: Usando Servicios de Homebrew (recomendado para uso persistente)

brew services start grafana

Esto iniciará Grafana como un servicio en segundo plano que se lanza automáticamente al iniciar sesión.

Salida esperada:

==> Successfully started `grafana` (label: homebrew.mxcl.grafana)

Opción B: Ejecutar Grafana directamente (para uso temporal)

grafana server --config /opt/homebrew/etc/grafana/grafana.ini \
  --homepath /opt/homebrew/opt/grafana/share/grafana

💡 Nota: Si el puerto 3000 ya está en uso, puedes especificar un puerto diferente:

grafana server --config /opt/homebrew/etc/grafana/grafana.ini \
  --homepath /opt/homebrew/opt/grafana/share/grafana \
  cfg:default.server.http_port=3001

Paso 4: Verificar que Grafana Está Ejecutándose

Verifica el estado del servicio:

brew services list | grep grafana

O verifica si Grafana está respondiendo en el puerto predeterminado:

curl -I http://localhost:3000

Deberías ver una respuesta HTTP con estado 302 (redirección a la página de inicio de sesión).

Paso 5: Acceder a la Interfaz Web de Grafana

Abre tu navegador web y navega a:

http://localhost:3000

O si cambiaste el puerto:

http://localhost:3001

¡Deberías ver la página de inicio de sesión de Grafana!

Detener Grafana

Para detener el servicio de Grafana:

brew services stop grafana

Gestionar el Servicio de Grafana

Ver todos los servicios de Homebrew:

brew services list

Reiniciar Grafana:

brew services restart grafana

3️⃣ Primera Vista de Grafana

Inicio de Sesión Inicial

Cuando accedas por primera vez a Grafana, verás la pantalla de inicio de sesión.

Credenciales predeterminadas:

Usuario (Username): admin
Contraseña (Password): admin

Ingresa estas credenciales y haz clic en Log In.

Solicitud de Cambio de Contraseña

En el primer inicio de sesión, Grafana te solicitará inmediatamente que cambies la contraseña predeterminada por razones de seguridad.

Puedes:

Ingresar una nueva contraseña segura y hacer clic en Submit
Hacer clic en Skip para cambiarlo más tarde (no recomendado para uso en producción)

⚠️ Advertencia de Seguridad: Siempre cambia la contraseña predeterminada en entornos de producción para prevenir accesos no autorizados.

¡Bienvenido a Grafana!

Después de iniciar sesión, verás el Dashboard de Inicio (página de bienvenida).

Comprendiendo la Interfaz de Usuario

La interfaz de Grafana consiste en varias áreas clave:

Barra Lateral Izquierda (Navegación Principal)

La barra lateral izquierda plegable contiene el menú de navegación principal:

🏠 Home: Volver al dashboard de inicio
🔍 Explore: Exploración de datos ad-hoc sin crear un dashboard
⚠️ Alerting: Configurar y gestionar reglas de alerta y notificaciones
📊 Dashboards: Explorar, crear y gestionar dashboards
🔌 Connections:
- Data sources: Configurar conexiones a tus bases de datos y servicios
- Plugins: Explorar e instalar plugins adicionales
⚙️ Administration (solo admin):
- Users: Gestionar cuentas de usuario
- Teams: Organizar usuarios en equipos
- Plugins: Gestionar plugins instalados
- Settings: Configuración global de Grafana
- API Keys: Generar tokens API para acceso programático

Barra de Navegación Superior

Nombre de la organización: Haz clic para cambiar de organización (si tienes múltiples)
Buscar (Search): Encuentra dashboards rápidamente
Crear (Create) (+): Crear nuevo dashboard, carpeta o alerta
Ayuda (Help) (?): Acceder a documentación y soporte
Perfil (Profile): Tu configuración y preferencias de usuario

Área de Contenido Principal

Aquí es donde se muestran los dashboards, paneles y páginas de configuración.

Explorando Fuentes de Datos

Una de las primeras cosas que querrás hacer es configurar una fuente de datos.

Haz clic en Connections → Data sources en la barra lateral izquierda
Verás la página de Fuentes de Datos

Haz clic en Add data source para ver las opciones disponibles:

Prometheus
Graphite
InfluxDB
MySQL, PostgreSQL
Elasticsearch
CloudWatch
Y muchas más...

💡 Consejo: Para tu primera experiencia, puedes usar la fuente de datos integrada TestData DB de Grafana, que genera datos de muestra para probar visualizaciones.

Creando Tu Primer Dashboard

Haz clic en el ícono + en la navegación superior o selecciona Dashboards desde la barra lateral
Haz clic en Create Dashboard
Verás un dashboard vacío con una opción para Add visualization

¡Aquí es donde construirás tus primeros paneles de dashboard!

4️⃣ Conceptos Clave Explicados

Dashboards

Un dashboard es una colección de paneles organizados en un diseño de cuadrícula. Los dashboards son:

Compartibles con miembros del equipo
Exportables como JSON
Controlados por rango de tiempo (puedes ajustar el período de tiempo para todos los paneles)

Paneles (Panels)

Los paneles son los bloques de construcción de los dashboards. Cada panel:

Muestra datos de una o más consultas
Puede personalizarse con varios tipos de visualización (gráficos, tablas, estadísticas, etc.)
Tiene su propia configuración de apariencia y comportamiento

Fuentes de Datos (Data Sources)

Las fuentes de datos son los backends donde se almacenan tus métricas, logs u otros datos. Grafana soporta docenas de fuentes de datos desde el inicio.

Consultas (Queries)

Las consultas definen qué datos recuperar de tus fuentes de datos. El lenguaje de consulta depende del tipo de fuente de datos (PromQL para Prometheus, SQL para bases de datos, etc.).

Rango de Tiempo (Time Range)

Los dashboards de Grafana están centrados en el tiempo. El selector de rango de tiempo en la esquina superior derecha controla qué período de tiempo estás visualizando en todos los paneles.

5️⃣ Conclusión

¡Felicitaciones! Has completado exitosamente:

✅ Instalación de Grafana en macOS usando Homebrew
✅ Inicio del servicio de Grafana
✅ Acceso a la interfaz web de Grafana
✅ Inicio de sesión y cambio de la contraseña predeterminada
✅ Exploración de la interfaz de usuario y componentes clave de Grafana
✅ Aprendizaje sobre fuentes de datos y dashboards

Próximos Pasos

Ahora que tienes Grafana funcionando, aquí hay algunos pasos recomendados:

Agregar una Fuente de Datos
- Configura una conexión a tu base de datos de métricas
- O usa la TestData DB integrada para practicar
Crear Tu Primer Dashboard
- Agrega paneles con diferentes tipos de visualización
- Experimenta con consultas y transformaciones
- Personaliza la apariencia de los paneles
Explorar Dashboards de Muestra
- Importa dashboards pre-construidos desde grafana.com/dashboards
- Aprende de ejemplos de la comunidad
Configurar Alertas
- Configura reglas de alerta para monitorear métricas importantes
- Establece canales de notificación (email, Slack, etc.)
Aprender Lenguajes de Consulta
- Estudia el lenguaje de consulta para tu fuente de datos
- Practica la construcción de consultas complejas

Recursos Adicionales

Documentación Oficial: grafana.com/docs
Foros de la Comunidad: community.grafana.com
Tutoriales en YouTube: Busca "Grafana tutoriales" para guías en video
Repositorio de Dashboards: grafana.com/dashboards

Referencia de Comandos Comunes

# Iniciar Grafana
brew services start grafana

# Detener Grafana
brew services stop grafana

# Reiniciar Grafana
brew services restart grafana

# Verificar estado del servicio
brew services list | grep grafana

# Ver logs de Grafana
tail -f /opt/homebrew/var/log/grafana/grafana.log

# Actualizar Grafana
brew upgrade grafana

¡Feliz monitoreo con Grafana! 🎉

Practice AWS Certification Question: AWS Solutions Architect Professional — Lambda — ECR

juan jose orjuela — Sun, 03 Nov 2024 15:24:37 +0000

When studying for an AWS certification, we often encounter questions that require a deep understanding of services we might not use every day. Understanding these services only at a theoretical level can lead to confusion and mistakes during the exam. So, how can we improve our comprehension and retention of these complex topics?

One of the best ways to tackle this challenge is by practicing directly with AWS services, replicating question scenarios in a real environment.

Services Associated with the Question: Lambda and ECR

Domain: Designing New Solutions

Question: Your team is developing a new Lambda function for a microservice component. You need to package and deploy the Lambda function as a container image. The container image must be based on the python:buster image with other dependencies and libraries installed. To use the container image correctly for the Lambda function, which of the following actions is necessary?

Answer: Install the runtime interface client in the container image to make it compatible with Lambda.

Related Services

AWS Lambda:

Description: Lambda is an AWS service that allows you to run code without provisioning or managing servers. This service supports multiple programming languages and, more recently, allows the use of container images as execution environments.
Relevant Features: When using Lambda with container images, it is essential to include a "runtime interface client" in the image. This client is an API within the container that enables Lambda to interact correctly with the runtime environment.

Amazon Elastic Container Registry (ECR):

Description: ECR is a fully managed container registry service that simplifies the storage, management, and deployment of Docker container images.
Relevant Features: Although it is not necessary to install an ECR agent in the container for this question, ECR is still relevant for storing and managing the container images that will be run on Lambda.

Practical Exercise to Demonstrate the Correct Answer

To test this configuration, you can follow these steps:

Create a Dockerfile with the installation of the runtime interface client.

FROM python:buster

RUN pip install awslambdaric

WORKDIR /var/task

COPY app.py /var/task/

ENTRYPOINT ["python3", "-m", "awslambdaric"]

CMD ["app.handler"]

Create a basic handler for the Lambda execution.

import json

def handler(event, context):
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Build the Docker image locally:

docker build -t lambda-container-demo .

Authenticate to the AWS account using environment variables:

export AWS_ACCESS_KEY_ID=mi-key
export AWS_SECRET_ACCESS_KEY=mi-secret
export AWS_DEFAULT_REGION=us-east-1

Create the repository in ECR to upload our image:

aws ecr create-repository --repository-name lambda-container-demo

Verify that it has been created in the ECR service from the console:

Run the following commands to authenticate and upload the container:

#Inicia sesión en ECR, crea un repositorio y sube la imagen
aws ecr create-repository --repository-name lambda-container-demo
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <your_account_id>.dkr.ecr.us-east-1.amazonaws.com
docker tag lambda-container-demo:latest <your_account_id>.dkr.ecr.us-east-1.amazonaws.com/lambda-container-demo:latest
docker push <your_account_id>.dkr.ecr.us-east-1.amazonaws.com/lambda-container-demo:latest

Create a Lambda function based on this ECR image.

Select the previously created image.

Test the Lambda function to verify the expected behavior.

This concludes the practice for answering this AWS Architect Professional certification question.

Keep in mind that if you've never used ECR or Lambda with Docker images, in this practice you covered basic concepts that, by practicing, can help you retain them long-term.

If you've enjoyed this article, feel free to give a 👏.
🤔 Follow me on social media! ⏬

YouTube: https://www.youtube.com/jjoc007
Twitter: https://twitter.com/jjoc007
GitHub: https://github.com/jjoc007
Medium: https://jjoc007.com
LinkedIn: https://www.linkedin.com/in/jjoc007/

Thank you!

Analysis of AWS Certification Question: AWS Solutions Architect Professional — VPC Flow Logs — Kinesis Data Firehose

juan jose orjuela — Sun, 03 Nov 2024 15:13:26 +0000

One of the best ways to tackle this challenge is by practicing directly with AWS services, replicating question scenarios in a real environment. In this article, I will guide you through the analysis of a specific question about VPC Flow Logs and Kinesis Data Firehose, breaking down each component and showing how you can build a similar practice in your AWS account. This hands-on approach not only reinforces key concepts but also provides real insight into how these services work in practice, helping us turn theory into applicable knowledge.

Domain: Design Solutions for Organizational Complexity

An engineering firm has deployed a critical application on the web servers of an Amazon EC2 instance launched in a VPC. The operations team is looking for a detailed analysis of the traffic from these web servers. They have enabled VPC Flow Logs on the VPC. The logs need to be analyzed using open-source tools in near real-time and visualized to create dashboards.

Proposed Solution:

Ingest the VPC Flow Logs into Amazon Kinesis Data Firehose, which will deliver these logs to Amazon OpenSearch Service for near real-time analysis and visualization of the logs.

Problem Architecture

Involved Services

VPC Flow Logs:

Captures and stores information about the network traffic entering and leaving network interfaces in your Amazon VPC. VPC Flow Logs allow recording details about network connections for analysis and monitoring.

Amazon Kinesis Data Firehose:

A real-time data ingestion service that facilitates loading large volumes of data into storage and analysis services. In this case, Kinesis Data Firehose acts as the intermediary that collects the VPC Flow Logs and sends them to Amazon OpenSearch Service.

Amazon OpenSearch Service:

An AWS-managed platform for search, analysis, and real-time data visualization. It is commonly used to work with logs and telemetry data. OpenSearch Service allows storing and analyzing VPC Flow Logs and creating dashboards to visualize them.

Steps to Replicate the Exercise:

Objective: Configure VPC Flow Logs, send the logs to Kinesis Data Firehose, and visualize the data in Amazon OpenSearch Service.

Create an OpenSearch Service domain to store and analyze the VPC Flow Logs.
Create a Firehose delivery stream to send the flow logs to the OpenSearch Service domain.
Create a VPC Flow Log subscription to the delivery stream.
Explore the VPC Flow Logs on the OpenSearch Service dashboards.
Create a role mapping with an OpenSearch Service user to the Kinesis Data Firehose service role. Since we are using a public access domain for OpenSearch Service, we need to assign the IAM role of the delivery stream to the OpenSearch Service principal user to send bulk logs to the OpenSearch Service domain.
Create an index pattern in the OpenSearch Service dashboards to enable analysis and visualization of the VPC logs.

Prerequisites

As a prerequisite, you need to create an Amazon Simple Storage Service (Amazon S3) bucket to store Firehose delivery stream backups and failed logs.

⚠️ 💰 🤑 Before proceeding, be cautious of the potential costs that may be incurred by executing these steps and remember to delete the resources after completing the exercise ⚠️ 💰 🤑

Step-by-Step Execution of the Tutorial:

Creating an S3 Bucket to Store Backups of Kinesis Data Firehose Messages:

Creating the OpenSearch domain:

After a few minutes, we will see the created domain.

Creating the Kinesis Data Firehose:

Go to the Amazon Kinesis Data Firehose console and create a new Firehose stream.

Select the domain created previously:

Create an S3 bucket to store backups.

Creating VPC Flow Logs:

Go to the VPC service and select the VPC to which you want to add the configuration.

Create a flow log:

Select the Firehose stream that we created previously.

Go to the OpenSearch dashboard.

Since we are using a public access domain for OpenSearch Service, you need to assign the role created for the Firehose delivery stream to the OpenSearch Service dashboard user so that the delivery stream can send bulk logs to the OpenSearch Service domain.

Go to Security > Roles and select the all_access role.

Copy the ARN of the role generated by Kinesis Firehose.

Return to Home, then go to Manage and select Index Patterns.

Create one with the prefix vpcflowlogs*.

At this point, we will be able to see the logs coming from the vpcflowlogs index.

We can also go to the Kinesis Data Firehose metrics to see the flow of messages between VPC and OpenSearch.

This concludes the practice for answering this AWS Architect Professional certification question.

Keep in mind that if you've never used OpenSearch or Kinesis, in this practice you covered basic concepts that, through hands-on experience, can help you retain them long-term.

References:

Question obtained from the simulator provided by https://www.whizlabs.com/
Stream VPC flow logs to Amazon OpenSearch Service via Amazon Kinesis Data Firehose

If you've enjoyed this article, feel free to give a 👏 and ⭐ to the repository.

🤔 Follow me on social media! ⏬

YouTube: https://www.youtube.com/jjoc007
Twitter: https://twitter.com/jjoc007
GitHub: https://github.com/jjoc007
Medium: https://jjoc007.com
LinkedIn: https://www.linkedin.com/in/jjoc007/

Thank you!

Analysis of AWS Solutions Architect Professional Certification Question — EC2 Image Builder and Resource Access Manager

juan jose orjuela — Sun, 27 Oct 2024 04:33:09 +0000

When studying for an AWS certification, we often encounter questions that require in-depth knowledge of services we may not use daily. Understanding these services at a purely theoretical level can lead to confusion and mistakes during the exam. So, how can we improve our understanding and retention of these complex topics?

One of the best ways to tackle this challenge is to practice directly with AWS services, replicating question scenarios in a real environment. In this article, I will guide you through analyzing a specific question on EC2 Image Builder and AWS Resource Access Manager (RAM), breaking down each component and showing how you can set up a similar practice in your AWS account—even if you don't have access to an AWS Organization. This practical approach not only reinforces key concepts but also provides a real view of how these services function in practice, helping us turn theory into applicable knowledge.

Domain: Design for New Solutions

Question: You are assisting a team in creating multiple AMIs and Docker images through EC2 Image Builder pipelines. Other teams want to use the same EC2 Image Builder resources, including components, recipes, and images, in their image pipelines. You need to find an appropriate approach to share resources with other organizational units within the AWS Organization or specific AWS accounts. Which of the following methods is suitable?

Answer: In AWS Resource Access Manager (RAM), add the shared components, images, or recipes to shared resources and configure the principals that are permitted to access the shared resources.

Analysis, Practice, and Demonstration of the Answer

Services Involved

AWS EC2 Image Builder

This AWS service allows you to automate the creation and management of system images (such as EC2 AMIs or Docker images). You can define "pipelines" that build images according to specifications (components, recipes, tests, etc.) and update them automatically as needed.

AWS Resource Access Manager (RAM)

RAM is an AWS service that enables sharing resources across accounts within an AWS Organization without needing to duplicate them. You can use RAM to share EC2 Image Builder components, AMIs, subnets, VPCs, and more with other accounts in your organization or specific external accounts.

Related Concepts

Build Component (AWS EC2 Image Builder)

In EC2 Image Builder, Build Components are scripts or command sequences that define custom configurations and installation steps for an image. These components allow for automating the installation, configuration, and validation of software and settings in the final image you are building.

Role of Build Components

Customization: You can use components to add specific software or make custom configurations in the image, such as installing web servers, databases, monitoring agents, or any other necessary software.
Automation: Components automate complex configurations. Instead of configuring each instance manually after the image is created, you can have the final image already include everything needed, reducing errors and saving time.
Modularity: Components are modular, meaning you can create a component once and reuse it across multiple recipes or pipelines. This is useful for maintaining consistent configurations across multiple images.

Examples of Common Components

Some examples of components you might find or create include:

Installing a web server (e.g., Apache or NGINX).
Installing and configuring monitoring agents (such as CloudWatch or Datadog).
Adding extra security configurations or installing development tool packages.
Operating system configuration tweaks, such as modifying network settings or kernel parameters.

Summary of Steps to Replicate the Exercise:

Objective: Create an image with EC2 Image Builder and share it with another account using RAM.

Step 1: Set Up EC2 Image Builder

Access the EC2 Image Builder console in AWS.
Create a New Image Pipeline:
- Go to “Image pipelines” and select “Create Image Pipeline.”
- Assign a name to the pipeline (e.g., MyCustomImagePipeline).
- In “Recipe,” select or create a recipe that defines the operating system and any additional components you want in the image.
Define the Recipe:
- If creating a new recipe, select a base image (e.g., Amazon Linux 2).
- Add components (e.g., system updates, specific software installation).
Configure Tests (Optional):
- Add tests to ensure the created image meets certain requirements.
Define the Distribution Policy (Optional):
- Decide in which regions the image will be available.
Configure the Pipeline and Save.

Step 2: Build the Image

Once the pipeline is configured, start a manual build to generate the first image or wait for the pipeline to execute an automatic build based on the defined schedule.
Check the progress in “Image Pipeline” and wait for the image to be ready.

Step 3: Share the Image with AWS RAM

Access the Resource Access Manager (RAM) console.
Create a New Resource Share:
- Select “Create resource share.”
- Assign a name (e.g., ImageShareForOtherTeams).
Add Resources to the Resource Share:
- In the “Resources” section, select the resource type as “EC2 Image Builder resources.”
- Add the image, components, or recipes you want to share.
Configure Principals:
- In “Principals,” select specific AWS accounts or organizational units (OUs) within your AWS Organization with whom you want to share the resources.
Review and Create:
- Review the configuration and select “Create resource share.”

Step 4: Verify Access from Another Account

In an account with which you shared the image, access the EC2 console.
Go to “Shared AMIs” and confirm that the image is available for launch.

⚠️ 💰 🤑 Before continuing, be cautious of the potential costs incurred by executing these steps and remember to delete the resources after completing the exercise ⚠️ 💰 🤑 Approx. $2 USD

Step-by-Step Execution

EC2 Image Builder (Pipeline Creation):

Since the pipeline creation is for testing purposes, we will limit ourselves to filling in only the name and description.

The build will be manual since we are only testing the service.

It is possible to create images for both AMI and Docker; for simplicity, we will choose AMI.

We can choose an initial configuration (user data) for the instance; in this example, a server is created with a page and a message.

#!/bin/bash
yum update -y
yum install -y httpd git
systemctl start httpd
systemctl enable httpd
echo "<h1>Welcome to EC2 Image Builder</h1>" > /var/www/html/index.html

In build components, we can select one or more components; some are already predefined, and we can also create our own.

We can also select validation components, which are used to ensure that the build component has executed successfully.

Storage:

Pipeline created successfully:

EC2 Image Builder (Pipeline Execution):

We run the pipeline that we created previously.

This will create an instance where it will set up the configuration and execute the build components we selected.

After the image is successfully created, the instance is stopped, and the creation of the AMI begins.

After this, the instance is terminated.

Then an instance is created to test the image.

After completing the verification, the AMI image is created for use.

RAM (Resource Access Manager)

Resource Creation

Resources to share:

As a limitation of this exercise, I do not have another external account or an organization to share with, but as we can see in the image, we can share EC2 Builder resources seamlessly through RAM.

This concludes the analysis and verification of the answer to this AWS Solutions Architect Professional certification question.

References

Question obtained from the simulator provided by Whizlabs
What is Image Builder?
Share Image Builder resources with AWS RAM
Spanish Version

If you've enjoyed this article, feel free to give a 👏 and ⭐ to the repository.

🤔 Follow me on social media! ⏬

YouTube: https://www.youtube.com/jjoc007
Twitter: https://twitter.com/jjoc007
GitHub: https://github.com/jjoc007
Medium: https://jjoc007.com
LinkedIn: https://www.linkedin.com/in/jjoc007/

Thank you!

AWS Step Functions: Using Parallel State

juan jose orjuela — Thu, 04 Jan 2024 14:46:45 +0000

Before diving into this post, I recommend you first check out: Introduction to AWS Step Functions Using Terraform as an Infrastructure as Code Tool

The Parallel State type in AWS Step Functions is particularly relevant for scenarios where the simultaneous execution of various operations is required. This feature not only speeds up processes but also provides a flexible and scalable way to handle complex tasks. In this article, we will explain a basic example of Parallel State, providing a detailed guide on its configuration and use. Our goal is to offer a clear understanding of how to implement this powerful tool in your AWS workflows, fully leveraging its capabilities to optimize your cloud processes.

Common Use Cases

Parallel Data Processing
In situations where large volumes of data need to be processed, Parallel State allows for dividing the workload into multiple tasks that can be executed simultaneously. This is particularly useful in big data and data analysis applications, where processing and analyzing data from multiple sources at the same time is required.
Microservices and Distributed Applications
In microservices-based architectures, different components of an application may need to perform tasks in parallel. Parallel State facilitates this coordination, allowing various services to function simultaneously to complete a larger process.
Automation of Complex Workflows
In workflows that involve multiple steps or stages, such as in approval or review processes, Parallel State can be used to execute several steps in parallel, speeding up the overall process and improving operational efficiency.
Simultaneous Testing and Analysis
In development and QA environments, Parallel State can be used to execute tests or analyses in parallel, reducing the total time needed for software validation or quality analysis.
IT Operations and Infrastructure Management
For tasks such as infrastructure deployment, system updates, or security patches, Parallel State allows multiple operations to be carried out at the same time, resulting in faster and more efficient management of IT resources.

Basic Example



{
  "Comment": "Ejemplo de Step Function con parallel state",
  "StartAt": "EstadoInicial",
  "States": {
    "EstadoInicial": {
      "Type": "Pass",
      "Result": {
        "mensajeInicial": "Inicio del flujo de trabajo"
      },
      "Next": "ProcesoParalelo"
    },
    "ProcesoParalelo": {
      "Type": "Parallel",
      "ResultPath": "$.resultadosParalelos",
      "Next": "EstadoFinal",
      "Branches": [
        {
          "StartAt": "Rama1",
          "States": {
            "Rama1": {
              "Type": "Pass",
              "Result": {
                "resultadoRama1": "Dato de la Rama 1"
              },
              "End": true
            }
          }
        },
        {
          "StartAt": "Rama2",
          "States": {
            "Rama2": {
              "Type": "Pass",
              "Result": {
                "resultadoRama2": "Dato de la Rama 2"
              },
              "End": true
            }
          }
        }
      ]
    },
    "EstadoFinal": {
      "Type": "Pass",
      "ResultPath": "$.resultadoFinal",
      "InputPath": "$.resultadosParalelos",
      "Result": {
        "mensajeFinal": "Fin del flujo de trabajo"
      },
      "End": true
    }
  }
}

In this example, we present an AWS Step Function designed to demonstrate the use of a Parallel State. This workflow includes an initial state, a parallel state with two branches, and concludes with a state that converges the results of the parallel branches.

Initial State

Type: Pass. This type of state simply passes its input to its output without modification. Here, it is used to set an initial message, indicating the start of the workflow.
Result: A JSON object with an initial message is assigned, for example, {"initialMessage": "Start of the workflow"}.

Parallel Process

Type: Parallel. This state allows the execution of multiple branches in parallel.
ResultPath: $.parallelResults. This field specifies where the results of the parallel branches should be added in the input state of the next state. In this case, the results of both branches will be stored in an object called parallelResults.
Branches: Contains two branches, each with its own Pass type state.
- Branch 1 and Branch 2:
- Type: Pass. Similar to the initial state, these states pass their input to their output. Here, each branch assigns a different result to a variable, such as {"branch1Result": "Data from Branch 1"} and {"branch2Result": "Data from Branch 2"}.

Final State

Type: Pass. This state marks the end of the workflow.
ResultPath: $.finalResult. Indicates where the result of this state will be stored, in this case in finalResult.
InputPath: $.parallelResults. Specifies that this state takes the results of the parallel state as input.
Result: A JSON object with a final message is assigned, for example, {"finalMessage": "End of the workflow"}.

In this workflow, the Initial State marks the beginning and sets an initial message. Then, the Parallel Process executes two branches in parallel, each generating its own result. These results are combined and passed to the Final State, which receives them and sets a final message.

Best Practices and Recommendations

When working with AWS Step Functions, and particularly with Parallel State, there are several best practices and recommendations that can help maximize the efficiency and effectiveness of your workflows:

Careful Workflow Design

Advance Planning: Carefully think about the structure of your workflow. Ensure that the use of Parallel States is truly beneficial and does not unnecessarily complicate the process.
Task Dependencies: Avoid dependencies between tasks executed in parallel. The real value of a Parallel State lies in the ability to execute independent tasks simultaneously.

Result Management

Result Consolidation: Properly use the ResultPath field to consolidate the results of parallel tasks. Ensure that the results are combined in a way that is useful for subsequent steps.
Error Handling: Design your workflow to properly handle errors in parallel tasks. Consider how a failure in one branch could affect the others and the workflow as a whole.

Performance Optimization

Load Balancing: Distribute the workload evenly among parallel tasks to avoid bottlenecks and maximize efficiency.
Scalability: Consider the scalability of your workflow. Ensure that it can handle increases in load without degrading performance.

Security and Access Control

Permissions and Roles: Ensure that the functions and services used in your Step Function have the appropriate permissions, following the principle of least privilege.
Auditing and Monitoring: Use monitoring tools and logs to audit the performance and activity of your workflows.

Rigorous Testing

Comprehensive Testing: Perform exhaustive testing of each component of the workflow, as well as the entire workflow, to ensure that everything works as expected, especially in failure scenarios.
Load Testing: Test how your workflow behaves under different loads to identify potential performance or scalability issues.

Documentation and Maintenance

Clear Documentation: Maintain detailed documentation of your workflow and its configuration to facilitate maintenance and future updates.
Regular Updates: Keep your workflow updated with the latest practices and features offered by AWS Step Functions.

In this repository, you will find the example ready to deploy in Terraform. Feel free to download it and give it a try.

References:

Example Repository: https://github.com/jjoc007/poc_step_functions_examples/tree/main/5_parallel_state
Official Documentation: https://states-language.net/spec.html#parallel-state

Mastering Docker: Defining Health Checks in Docker Compose

juan jose orjuela — Fri, 29 Dec 2023 02:49:08 +0000

In this article, we will explore an important aspect of Docker Compose, specifically focusing on health checks, an essential tool for ensuring the reliability and robustness of containerized services. We will cover everything from the basics of what they are and why they are important, to how to practically implement them in your Docker Compose definitions.

Context on Docker and Docker Compose

In the dynamic world of information technology, virtualization and containerization have revolutionized the way we deploy and manage applications. Docker emerges as a leading solution, enabling developers and system administrators to package, distribute, and manage applications efficiently. Docker encapsulates applications in containers, lightweight and portable environments that ensure consistency across different development, testing, and production environments.

Docker Compose, a tool within the Docker ecosystem, simplifies the process of defining and sharing multi-container applications. With Docker Compose, you can define all the services that make up your application in a single docker-compose.yml file. This tool facilitates the orchestration of multiple containers, allowing them to work together harmoniously to form complex applications.

Basic Concepts

What are Health Checks?

In the context of containerization and Docker, health checks are periodic tests or verifications used to determine whether a container is functioning correctly. Essentially, a health check is a way to automatically monitor the health status of a service or application within a container. If a service fails or stops responding, the health check can detect it and take appropriate actions, such as restarting the container or alerting the operations team.

Health checks are essential in production and development environments because they ensure that applications continue to function optimally and reliably. In an environment with multiple services and containers, health checks provide an additional layer of security by ensuring that each individual service is functioning as expected.

How They Work in Docker

Docker incorporates a health check system that allows users to define commands or instructions to check the status of a container. These commands can be as simple as an HTTP request to an application endpoint or a script that checks the availability of an internal service.

When a health check fails, Docker marks the container as unhealthy. This information can be used by orchestration and monitoring tools to make automated decisions, such as restarting the container or redistributing load among healthy containers.

Docker's health checks offer flexibility in terms of configuration, allowing users to define intervals between checks, the number of attempts before considering a service unhealthy, and the maximum time a check should take before being considered failed.

This functionality is crucial for maintaining high availability and reliability of applications, especially in microservices environments where multiple interdependent services must operate in sync.

In the next section, we will delve into how health checks are integrated and configured in Docker Compose, providing practical examples for their implementation.

Health Checks in Docker Compose

Integration with Docker Compose

Docker Compose, a tool for defining and running multi-container Docker applications, natively integrates the concept of health checks into its configurations. Through docker-compose.yml files, Docker Compose enables developers and system administrators to specify how the health check for each service in their environment should be performed.

The integration of health checks in Docker Compose is crucial for automating the monitoring and management of service health. In complex environments with multiple containers, health checks become a vital tool to ensure that each service is operating correctly and to facilitate automatic recovery in case of failures.

Configuration Parameters

In a docker-compose.yml file, the configuration of a health check is done within the definition of each service. The key parameters include:

test: The command that will be executed to check the status of the service. It can be a string or a list. For example, ["CMD", "curl", "-f", "http://localhost/health"] for a web application.
interval: The time between each check. For example, 30s indicates that the health check is performed every 30 seconds.
timeout: The maximum time a health check can take before being considered failed. For example, 10s.
retries: The number of times a failed health check will be attempted before marking the service as unhealthy.
start_period: An initial period during which the results of failed health checks are not counted towards the maximum number of retries. This is useful for applications that require time to start.

These parameters allow for detailed configuration tailored to the specific needs of each service, providing precise control over how and when a service should be considered unhealthy.

Example:



 service1:
    build: ./web-server
    environment:
      - PORT=3000
    ports:
      - "3000:3000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/ping"]
      interval: 2s
      timeout: 60s
      retries: 20

This Docker Compose example illustrates the configuration of a health check for a service named service1, which represents a web server. The health check definition is made in the docker-compose.yml file and is composed of several key components:

Health Check Test (`test`):

["CMD", "curl", "-f", "http://localhost:3004/ping"]: This line defines the command that will be executed for the health check. It uses curl to make an HTTP GET request to the /ping path on port 3004 of localhost. The -f option causes curl to fail if the HTTP status code is 400 or higher, which is useful for detecting errors. This command checks if the web service is responding correctly.

Interval (`interval`):

2s: This specifies that the health check will be executed every 2 seconds. That is, every 2 seconds, Docker Compose will execute the curl command to check the status of the service.

Timeout (`timeout`):

60s: This is the maximum time Docker will wait for the health check command to complete. If the curl command does not respond within 60 seconds, it will be considered a health check failure.

Retries (`retries`):

20: Indicates the number of times Docker will retry the health check before marking the service as unhealthy. In this case, if the health check fails 20 consecutive times, Docker will consider the service1 service to be unhealthy.

Example of a Service Dependent on Another:



service2:
    build: ./web-server
    environment:
      - PORT=3001
    ports:
      - "3001:3001"
    depends_on:
      service1:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3001/ping"]
      interval: 2s
      timeout: 60s
      retries: 20

In this Docker Compose example, we have two services: service1 and service2. Both are configured with health checks, but the interesting aspect here is the dependency of service2 on the health status of service1, indicated by the depends_on clause.

Health Check-Based Dependency (`depends_on`):

service2 uses the depends_on option to establish a dependency on service1.
condition: service_healthy: This line specifies that service2 should only start after service1 has been considered "healthy" by Docker.

Operation of `service1`'s Health Check in Relation to `service2`:

Starting of service1: When the set of services is initiated with Docker Compose, service1 begins its initialization process.
Execution of service1 Health Check: While service1 is running, Docker performs health checks at intervals of 2 seconds (as per its configuration). If the service successfully responds to the health check command (in this case, a curl to http://localhost:3000/ping), it continues operating normally. If it fails, Docker will retry the health check up to 20 times, with a 60-second timeout per attempt.
Health Status of service1 and Its Effect on service2: If service1 passes the health check and is marked as healthy, then Docker proceeds to start service2. If service1 fails its health checks and is marked as unhealthy, service2 will not start until service1 is considered healthy.

Example of Starting Up When `Service1` is Healthy:

Example of Starting Up When `Service1` is Unhealthy:

This health-based dependency model is crucial in environments where services need to interact with each other and where the availability of one service depends on the state of another. It ensures that service2 will not attempt to operate until service1, on which it depends, is fully functional and ready to handle requests or interact with other services. This improves the stability and reliability of the overall application deployment.

In this repository, you will find the example ready to be deployed with Terraform. Feel free to download and try it out.

References:

Example Repository: https://github.com/jjoc007/poc_docker_compose_multiservice
Official Documentation: Docker Compose File Reference

If you liked this article, don't hesitate to give it a 👏 and ⭐ the repository.

🤔 Follow me on social media! ⏬

Thank you!

Mastering Terraform: How to Manage Multiple Environments with Dynamic S3 Backends

juan jose orjuela — Wed, 20 Dec 2023 03:42:56 +0000

What is Terraform?

Terraform is an infrastructure as code (IaC) tool developed by HashiCorp. It is used to define and provision IT infrastructure using a declarative configuration language or, in more recent versions, through JSON as well. Terraform allows users to define resources both in the cloud (such as servers, storage, and networks) and on-premises (such as virtual machines and services).

Key Features of Terraform

Idempotence: Terraform ensures that multiple executions of the same configuration files produce the same end-state, avoiding inconsistencies and potential errors in the infrastructure.
State Management: Terraform keeps a record of the current state of the infrastructure, which facilitates incremental changes and automation.
Multi-Cloud and On-Premise Support: Compatible with numerous cloud service providers, like AWS, Azure, and Google Cloud, as well as on-premise solutions, allowing users to efficiently manage a complex hybrid environment.
Modules and Templates: Enables the reuse of configurations through modules, improving efficiency and consistency in infrastructure management.

The Importance of Terraform in Infrastructure Management

Terraform has rapidly gained popularity in the world of software development and systems management due to its ability to handle large infrastructures efficiently and predictably. Its declarative nature and ability to integrate with a wide range of cloud service providers make it an indispensable tool for companies looking to adopt DevOps practices and infrastructure automation.

Backends in Terraform

Backends in Terraform play a crucial role in managing the state of the infrastructure. In Terraform, the state is a record of the managed infrastructure that maintains information about the configured resources and their properties. Backends determine both the location of this state and the locking method to prevent state conflicts.

Types of Backends in Terraform

Terraform offers various types of backends, mainly classified into two categories: local and remote.

Local Backends: Store the state in a file on the local system. They are simple and easy to use but are not suitable for team collaboration.
Remote Backends: Save the state to a remote service. These are ideal for teams as they allow state sharing and locking of files to prevent conflicts.

The AWS S3 Backend

Among remote backends, the AWS S3 backend is one of the most popular. This backend uses Amazon S3 services to store the state file and can optionally be integrated with DynamoDB for state locking and consistency.

Advantages of the S3 Backend:

Durability and Scalability: S3 offers high durability and scalability, ensuring the security and accessibility of Terraform's state.
Access Control: Integration with AWS IAM allows for detailed control of state access.
Consistency: When used with DynamoDB, it ensures state consistency and prevents conflicts in team environments.

Importance of Backends in State Management

The choice of backend directly affects how Terraform's state is managed, especially in team environments and on a large scale. An appropriate backend ensures:

Security: Protection against data loss and unauthorized access.
Effective Collaboration: Allows multiple users to work on the same infrastructure without overwriting changes.
Automation: Facilitates integration with CI/CD systems.

Example of S3 Backend Configuration in Terraform

terraform {
  backend "s3" {
    bucket = "nombre-del-bucket-s3"
    key    = "path/del/estado/terraform.tfstate"
    region = "us-east-1"
    encrypt = true
  }
}

Components of the Configuration:

bucket: The name of the Amazon S3 bucket where the Terraform state will be stored.
key: The location within the bucket where the Terraform state file (.tfstate) will be saved.
region: The AWS region where the S3 bucket is located.
encrypt: Enables encryption on the AWS server for the state file stored in S3.

Challenge in Managing Multiple Environments in Terraform

One of the most common challenges when working with Terraform, especially in large organizations or complex projects, is the efficient management of multiple infrastructure environments, such as development, testing, and production. Each of these environments may have different configurations and needs, and it is essential to keep them isolated to prevent interference and errors.

Problems with a Single Backend

When a single backend is used for all environments, several problems can arise:

State Conflict: If all environments share the same state file, there is a significant risk of conflicts and accidental overwrites.
Security and Access Control: A single backend may not provide the necessary level of differentiated access control for different environments.
Difficulties in Automation: The automation of deployments becomes more complex and error-prone when multiple environments interact with a single state.

The Need for Isolation and Flexibility

Isolation is crucial to ensure that configurations from one environment do not affect others. Moreover, each environment may require different backend configurations in terms of storage regions, access policies, and other specific security and performance settings.

Scalability and Maintenance Challenge

As a project grows, maintaining a single backend becomes unsustainable. Scalability and maintenance become significant issues, and efficient state management becomes more challenging.

Solution with Multiple S3 Backends

The solution to the challenges presented in managing multiple environments in Terraform lies in the implementation of multiple S3 backends. This strategy involves setting up a unique S3 backend for each environment (development, testing, production, etc.), using Terraform's -backend-config parameter.

Using the `-backend-config` Parameter

The -backend-config parameter allows Terraform users to specify a backend configuration file for each initialization. This enables a clear separation of the states for each environment, ensuring that operations in one environment do not affect others.

Example of Dynamic Configuration

In the main Terraform file (main.tf), a generic backend is defined:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~>4.0"
    }
  }

  backend "s3" {}
}

Then, specific configuration files for each environment are created in an env folder:

Content of an Environment Configuration File (`backend_s3_dev.hcl`):

key    = "dev-state/terraform.tfstate"
bucket = "dev-bucket"
region = "us-east-1"

Content of an Environment Configuration File (`backend_s3_prod.hcl`):

key    = "prod-state/terraform.tfstate"
bucket = "prod-bucket"
region = "us-east-1"

Initialization and Application

To initialize Terraform for a specific environment, the following command is used:

terraform init -backend-config="env/backend_s3_dev.hcl"

This command sets up the backend for the development environment, using the backend_s3_dev.hcl file.

Benefits of This Approach

State Isolation: Each environment has its own state, stored in a separate S3 bucket, preventing conflicts and overwrites.
Enhanced Security: Specific access policies can be applied for each environment, improving security.
Configuration Flexibility: Allows for individual adjustments in storage settings and regions for each environment.
Ease of Automation: This setup facilitates integration with CI/CD systems, enabling safe and efficient automated deployments.

In this repository, you will see the example ready to be deployed in Terraform. Feel free to download it and try it out.

References:

Example Repository: https://github.com/jjoc007/poc_terraform_with_multiples_s3_backend
Official Documentation: https://developer.hashicorp.com/terraform/language/settings/backends/configuration

If you liked this article, feel free to give it a 👏 and ⭐ the repository.

Thank you!

AWS Step Functions: Example HTTP Request Call"

juan jose orjuela — Mon, 18 Dec 2023 19:19:03 +0000

In the dynamic world of cloud computing, integration and automation are keys to success. AWS Step Functions stands as a powerful orchestrator of microservices, enabling developers to weave complex sequences of tasks with ease and efficiency. Among its crown jewels, HTTP Tasks shine on their own, offering unmatched versatility and integration capabilities.

What are HTTP Tasks? Essentially, they are states within an AWS Step Functions state machine designed to interact with third-party APIs. Think of them as bridges connecting your workflow in AWS with the vast universe of available web services. With HTTP Tasks, you can call external APIs like Salesforce, Stripe, or even more customized services, seamlessly integrating into your workflows.

Features:

Connection Flexibility: Whether you need to fetch data, send information, or initiate processes in external systems, HTTP Tasks allow you to do so with just a few clicks.
Robust Security: The integration of AWS IAM and EventBridge ensures that your API calls are secure and managed with AWS's best practices in terms of authentication and authorization.
Advanced Customization: From specifying HTTP methods (GET, POST, and more) to adjusting headers and parameters, you have total control over how you interact with external APIs.
Monitoring and Scalability: Benefit from integrated monitoring and the ability to scale automatically to handle workloads of any size.

This article will guide you through a fascinating journey into the world of HTTP Tasks in AWS Step Functions. We explore realistic use cases, crucial aspects of implementation, and conclude with a practical example: notifying a Slack webhook about the status of a process. Get ready to unlock new possibilities and take your cloud workflows to the next level.

Use Cases:

HTTP Tasks in AWS Step Functions open up a range of possibilities for service integration and process automation. This section explores various practical and powerful use cases where HTTP Tasks can be a transformative tool.

Integration with CRM and ERP Systems

Customer and Order Management: Automate data synchronization between your AWS system and CRM platforms like Salesforce. For instance, you can automatically update customer records in Salesforce when changes occur in your AWS system.
Business Process Automation: Connect your workflows with ERP systems to manage inventory, orders, and logistics, enabling real-time integration that optimizes business operations.

Interactions with Payment Platforms

Payment Processing: Integrate services like Stripe or PayPal to automate payment processing. HTTP Tasks can send transaction data and receive payment confirmations, streamlining the sales cycle.
Subscription Management: Automate the creation and management of subscriptions, as well as updating customer details in online payment systems.

Notifications and Alerts

Sending Notifications: Use services like Twilio or SendGrid to send SMS or emails as part of a workflow, for example, for transaction alerts or order status notifications.
Integration with Slack: Send automatic messages to Slack channels or users to notify about project updates, system alerts, or data summaries.

Data Analysis and Reporting

Integration with Analysis Tools: Extract data from AWS for analysis in external platforms like Google Analytics or BI tools, allowing for deeper understanding and data-driven action.
Automated Reporting: Generate and distribute automatic reports using data processed by your AWS workflow, sending them to reporting tools or storing them in file systems like Google Drive.

Artificial Intelligence and Machine Learning Services

Interaction with AI/ML Models: Integrate external AI services like IBM Watson or Google AI to enrich your data with machine learning capabilities, sentiment analysis, or image recognition.
Machine Learning Process Automation: Orchestrate and manage ML workflows, from data preparation to training and evaluating models, using various ML APIs.

Cloud Resource Management

Orchestration of Cloud Services: Automate the creation, update, and deletion of cloud resources, interacting with services like AWS EC2 or Google Cloud Platform.
Monitoring and Event Response: Automatically respond to cloud system events, such as launching EC2 instances or performance alerts, using workflows that include calls to relevant APIs.

Integration of Social Media and Marketing

Automated Social Media Posts: Schedule and publish content on social media platforms like Twitter or Facebook, as part of digital marketing strategies.

Basic Example:

To illustrate how an HTTP Task is implemented in AWS Step Functions, let's consider a concrete example. Imagine that we need to make a GET call to an external API to fetch information. Below is the code for the HTTP Task and an explanation of each property involved.

{
  "Comment": "Ejemplo de Tarea HTTP en AWS Step Functions",
  "StartAt": "CallExternalAPI",
  "States": {
    "CallExternalAPI": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Parameters": {
        "ApiEndpoint": "https://api.externalservice.com/data",
        "Method": "GET",
        "Headers": {
          "Content-Type": "application/json"
        },
  "Authentication": {
          "ConnectionArn": "arn:aws:events:region:account-id:event-bus/default"
        }

      },
      "End": true
    }
  }
}

Properties

Comment: A description or comment about the purpose of the state machine. Useful for documentation and clarity.
StartAt: Indicates the first state that will be executed in the state machine. In this case, it's CallExternalAPI.
States: Defines the different states of the machine. Here we only have one state, CallExternalAPI.
- CallExternalAPI: The name of the state we are defining.
- Type: Defines the type of state. In this case, it's a Task, which means it performs a specific task.
- Resource: Specifies the type of resource that the task will use. Here arn:aws:states:::http:invoke is used to indicate that it is an HTTP Task.
- Parameters: Defines the necessary parameters for the HTTP task.
  - ApiEndpoint: The URL of the external API to which the call will be made.
  - Method: The HTTP method to be used. Here it is GET.
  - Headers: Necessary HTTP headers for the request. In this case, it indicates that the content is of JSON type.
- Authentication: This field is used to specify authentication details for the API call.
  - ConnectionArn: The Amazon Resource Name (ARN) of the EventBridge connection that manages authentication credentials. This connection must be pre-configured in EventBridge and contain the necessary information to authenticate with the API (e.g., an API token).
End: Indicates if this is the last state of the state machine. If true, the state machine will end after executing this state.

Practical Example: Simple Notification to Slack via Webhook

In today's digital era, instant communication and notification are essential for efficiency and team collaboration. Slack, one of the leading platforms in business communication, offers robust functionality to integrate personalized notifications through webhooks. Combining this with AWS Step Functions allows us to automate notifications and keep teams informed about key events and processes in real time.

In this section, we will tackle a practical and highly relevant example: sending a simple notification to a Slack channel using a webhook. This task, seemingly straightforward, encapsulates fundamental concepts of integration and automation in the cloud, demonstrating how modern tools can work together efficiently.

Steps for Creating a Webhook in Slack

Create a Channel for Receiving Notifications: Start by creating a channel where you will receive the notifications. In my case, I named it slack-notification-test.

Create a Workflow in Integrations: Next, go to the integrations section and create a workflow. In this example, we'll name it bot-auto-not.

Workflow Creation with Webhook Trigger: In the workflow creation process, choose to start it via a webhook. Following this, you will be asked to define the variables that will be received when consuming the webhook. In this example, the variables are type and text.

Adding an Additional Step for Message Sending: After setting up the webhook, create an additional step to send a message to the channel using the information from the defined variables.

Finding the Webhook URL: In the webhook step, you will find the URL that will be used to send notifications. It will be in this format: https://hooks.slack.com/triggers/11111111/222222222/333333333.
Testing the Webhook: You can test the webhook by making a POST request and sending the parameters defined in the workflow in the body of the request.

curl --location 'https://hooks.slack.com/triggers/1111/22222/33333 \
--header 'Content-type: application/json' \
--data '{
  "text": "Example text",
  "type": "ERROR"
}'

Starting with the Step Function Implementation: With all these components defined, we can now begin with the implementation of the Step Function.

Steps for Creating the Step Function via Terraform:

Create a Role and Policies for the Step Function: The first step is to create a role and policies that will be assumed by the Step Function, enabling it to make HTTP calls.

Role Resource:

resource "aws_iam_role" "step_functions_role" {
  name = "step_functions_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "states.amazonaws.com"
        }
      }
    ]
  })
}

Policies Resource for role:

resource "aws_iam_policy" "sfn_logging_policy" {
  name        = "SFNLoggingPolicy"
  description = "Allow Step Functions to log to CloudWatch Logs."

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "logs:*"
        ],
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_policy" "sf_invoke_requests_policy" {
  name        = "SFSendInvokeRequestsPolicy"
  description = "Allow Step Functions to invoke requests."

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "states:InvokeHTTPEndpoint"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "events:RetrieveConnectionCredentials"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret"
        ],
        Resource = "*"
      }
    ]
  })
}

There are two key policies to set up:

Log Management Policy: This policy is for the management of logs.
HTTP Invocation and Secrets Access Policy: The second policy grants the Step Function the ability to invoke HTTP services and read secrets, which is essential in cases where the requests require authentication.

Creation of the EventBridge Connection: AWS EventBridge is a serverless event bus service that facilitates the connection of applications with data from different sources. A key feature is the ability to establish "connections" that allow AWS Step Functions to securely communicate with third-party APIs, such as Slack, Salesforce, Stripe, among others. These connections are used, for example, in HTTP Tasks to manage authentications and authorizations securely and efficiently. How it works:
- Creating a Connection: In EventBridge, you define a "connection" that encapsulates the authentication details necessary to communicate with an external API. This connection can include information such as access tokens, API keys, or basic authentication credentials.
- Secure Credential Storage: EventBridge securely stores credentials using AWS Secrets Manager. This means that the credentials are encrypted and protected, and are not exposed in the task definitions or in the code.
- Integration with Step Functions: When defining an HTTP Task in AWS Step Functions, you specify the ARN (Amazon Resource Name) of the EventBridge connection. Step Functions uses this connection to authenticate with the external API during the task execution.
- Authorization Management: The connection handles the authorization process with the external API. This frees developers from the burden of implementing and maintaining custom code for managing tokens or API keys.

Event bridge connection Resource:

resource "aws_cloudwatch_event_connection" "slack_test_webhook" {
  name               = "test-slack-notification-connection"
  description        = "test-slack-notification-connection"
  authorization_type = "API_KEY"

  auth_parameters {
    api_key {
      key   = "not-key"
      value = "None"
    }
  }
}

Necessity of the Connection Even Without Authentication for the Webhook: This connection is required even though the webhook does not handle authentication. For this reason, we input a random API key.

Aws Step function definition:

{
  "Comment": "Send a slack notification via sf",
  "StartAt": "FirstStep",
  "States": {
    "FirstStep": {
      "Type": "Pass",
      "Result": {
        "type": "INFO",
        "text": "Message from Step Functions"
      },
      "ResultPath": "$.notificationData",
      "Next": "SendNotification"
    },
    "SendNotification": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Parameters": {
        "Authentication": {
          "ConnectionArn": "${aws_cloudwatch_event_connection.slack_test_webhook.arn}"
        },
        "RequestBody": {
          "text.$": "$.notificationData.text",
          "type.$": "$.notificationData.type"
        },
        "ApiEndpoint": "https://hooks.slack.com/triggers/11111111/2222222222/33333333333",
        "Method": "POST"
      },
      "Next": "LastStep"
    },
    "LastStep": {
      "Type": "Succeed"
    }
  }
}
EOF
}

Three States in the Step Function:

FirstStep: This state generates the data that will be sent in the notification. The data is sent through the notificationData object.
SendNotification: This state is responsible for sending the notification. Notice that

After going through the process of setting up a simple notification to Slack via a webhook using AWS Step Functions, it's clear that we are dealing with a powerful combination of tools that can revolutionize the way we interact with workflows and team communication.

If you want to see the complete example to run it in Terraform, here's the repository for you to check out.

If you liked this article, don't hesitate to give it a 👏 and a ⭐ on the repository.

Thank you!!

AWS Step Functions: Sending an Email from a State

juan jose orjuela — Tue, 12 Dec 2023 04:21:59 +0000

In today's agile and always-connected world, automation and workflow orchestration are crucial for operational efficiency. AWS Step Functions is a service that simplifies the coordination of distributed application components and microservices using visual workflows. One of the most powerful features of Step Functions is its direct integration with various AWS services through AWS SDK. This functionality allows developers to quickly build complex applications with less code and maintenance. In this post, we will explore how this integration works, specifically focusing on how to send emails using Amazon Simple Email Service (SES) V2.

What is and How Does an AWS SDK Integration from Step Function Work?

The integration of AWS SDK into Step Functions allows users to make direct calls to AWS services without the need for writing a custom Lambda function for the interaction. This means you can use AWS APIs directly within your Step Functions state machine definition.

When you configure a state in your state machine to perform an AWS SDK operation, you simply specify the AWS service and the action you wish to use, along with the necessary parameters for that API call. Step Functions handles the API request and response, including retry attempts and error handling.

Official documentation for this:

Common Use Cases for This Type of Integration

Data Processing: Execute tasks such as data transformations and analytics using services like AWS Glue or Amazon EMR.
Infrastructure Automation: Orchestrate changes in AWS infrastructure, such as updating AWS CloudFormation stacks or launching Amazon EC2 instances.
Application Integrations: Communicate with other AWS services to send notifications with Amazon SNS, message queues with Amazon SQS, or store files in Amazon S3.
User Management: Automate the creation and management of users in AWS Identity and Access Management (IAM).

Example: Sending an Email with SES V2

Imagine that you need to send confirmation emails to users after they complete an action in your application. Instead of invoking a Lambda function that in turn calls SES, you can use Step Functions to call SES directly.

Here is an example of how to send an email using arn:aws:states:::aws-sdk:sesv2:sendEmail:

{
  "StartAt": "SendEmail",
  "States": {
    "SendEmail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:sesv2:sendEmail",
      "Parameters": {
        "FromEmailAddress": "sender@example.com",
        "Destination": {
          "ToAddresses": ["recipient@example.com"]
        },
        "Content": {
          "Simple": {
            "Subject": {
              "Data": "Your confirmation email",
              "Charset": "UTF-8"
            },
            "Body": {
              "Text": {
                "Data": "Thank you for your action.",
                "Charset": "UTF-8"
              }
            }
          }
        }
      },
      "End": true
    }
  }
}

In this SendEmail state, we have defined a task that utilizes the resource arn:aws:states:::aws-sdk:sesv2:sendEmail. The Parameters specify the sender's email, the recipient, and the content of the email, including both the subject and the body.

This approach simplifies the architecture of your application, as there is no need to code and maintain a Lambda function for tasks that can be handled directly by Step Functions. Furthermore, error handling and retry mechanisms can be managed within the state machine's definition, providing a robust and reliable solution.

How Can You Find the Parameters for a Specific Integration?

On this page https://docs.aws.amazon.com/step-functions/latest/dg/supported-services-awssdk.html, you will find the enabled integrations for multiple AWS services.
Then, search for the API documentation of the integration you are going to use. In this example, SES V2 was used, so we would search at https://docs.aws.amazon.com/ses/latest/APIReference-V2/API_SendEmail.html.
On the reference page, look for the action you want to perform, in this case, SendEmail.
At this point, we will see the syntax of the request, details such as the URL, headers, and body of the request. Our focus will be on the body of the request:

POST /v2/email/outbound-emails HTTP/1.1
Content-type: application/json

{
   "Content": {
      "Simple": {
      "Body": {
         "Text": {
            "Charset": "UTF-8",
            "Data": "body"
         }
      },
      "Subject": {
         "Charset": "UTF-8",
         "Data": "subject"
      }
      }
   },
   "Destination": {
      "ToAddresses": [
      "email"
      ]
   },
   "FromEmailAddress": "email"
}

These same parameters that we see in the body are what we are going to add in the state definition of our state function.

In summary, the direct integration of AWS SDK into Step Functions opens a world of possibilities for orchestrating AWS services efficiently and effectively, allowing developers to focus on business logic rather than service integration.
In this repository https://github.com/jjoc007/poc_step_functions_examples/tree/main/2_send_email_via_ses, you will see the example ready to deploy with Terraform, feel free to download it and try it out.

If you liked this article, don't hesitate to give it a 👏 and a ⭐ on the repository.

Thank you.

AWS Step Functions: Using the Wait State Type

juan jose orjuela — Fri, 01 Dec 2023 04:58:13 +0000

Before checking out the post, I recommend first looking at: Introduction to AWS Step Functions using Terraform as an infrastructure as code tool.

The "Wait" state in AWS Step Functions is a powerful and flexible tool for managing workflows in the cloud. This state is essentially a pause mechanism, allowing workflows to wait for a specified period of time or until a specific event occurs before proceeding to the next step. It is crucial in scenarios where timing and temporal coordination are important.

Operation of the "Wait" State:

Fixed Wait: You can configure the "Wait" state so that your workflow pauses for a specific amount of time, expressed in seconds or until an exact hour and date. This option is useful for predictable delays, such as waiting between attempts of an operation or to allow time for external processes to complete.

Example:



"wait_ten_seconds" : {
  "Type" : "Wait",
  "Seconds" : 10,
  "Next": "NextState"
}

Dynamic Wait: It is also possible to configure the "Wait" state to wait until a certain event occurs, such as the arrival of a message to an SQS queue or the update of a specific resource. In this mode, the workflow resumes immediately after the condition is met, making it ideal for unpredictable events.

Example:



"wait_until" : {
  "Type": "Wait",
  "TimestampPath": "$.expirydate",
  "Next": "NextState"
}

As we can see in the previous example, the waiting time is determined by an input variable $.expirydate, which should come from the previous state.

Common Use Cases for the Use of the Wait State:

Approval Processes: In workflows that require human approvals, the "Wait" state can pause the process until a response is received.

Coordination of Asynchronous Tasks: In scenarios where multiple tasks are executed in parallel and one of them must wait for the others to complete.

Scheduled Delays: Useful in cases where a delay between steps is needed, such as in batch processing or in the staggered sending of notifications.

Example:

We have a workflow that is divided into 3 stages or actions:

**Create: **creation of the process. The process will have an ID and a state, and the state must advance to the effective state before an action of completion or reversal can be taken. It is important to clarify that the process's effective state does not occur immediately; it can take from 10 to 20 seconds.

Execute Action: At this point, the process must be in an effective state to execute the action. The allowed actions are finish or rollback, depending on the case. After this, the state of the process will be finishing or rollbacking.

End: this stage will validate that the process has successfully finished or has been successfully reverted.

Example states:

Create: This state calls a lambda function that will initialize the process.



"Create": {
      "Type": "Task",
      "Resource": "${aws_lambda_function.number_validator_lambda.arn}",
      "Parameters": {
        "input.$": "$",
        "action": "create"
      },
      "Next": "WaitForExecuteAction"
}

WaitForExecuteAction: After the process is initiated, a waiting period is required before validating the current state and executing the action (finish / rollback).



"WaitForExecuteAction": {
      "Type": "Wait",
      "Seconds": 10,
      "Next": "ExecuteAction"
}

ExecuteAction: After the waiting period, a lambda function must be executed that validates the current state of the process. If it is effective, the action should be executed; otherwise, the flow should fail.



"ExecuteAction": {
      "Type": "Task",
      "Resource": "${aws_lambda_function.number_validator_lambda.arn}",
      "Parameters": {
        "number.$": "$.number",
        "update.$": "$.update",
        "status.$": "$.status",
        "action": "execute_action"
      },
      "Next": "WaitForEnd"
}

WaitForEnd: After this, an additional waiting period is required for the process to reach a finished or rollbacked state.



"WaitForEnd": {
      "Type": "Wait",
      "Seconds": 10,
      "Next": "EndDeploy"
}

EndDeploy: Finally, it must be validated that the process has indeed finished; if not, the flow should fail.



"EndDeploy": {
      "Type": "Task",
      "Resource": "${aws_lambda_function.number_validator_lambda.arn}",
      "Parameters": {
        "number.$": "$.number",
        "update.$": "$.update",
        "status.$": "$.status",
        "action": "end"
      },
      "Next": "Ended"
}

Ended: Final step of the workflow.



"Ended": {
      "Type": "Succeed"
}

In summary, the "Wait" state in AWS Step Functions is a powerful tool for efficient workflow management. Its ability to handle both fixed and event-based pauses allows developers to build smarter and more responsive applications that can effectively react to changing conditions and business needs.

Complete example: https://github.com/jjoc007/poc_step_function_validator/tree/main
If you like it, give a 👍 to the post and a ⭐ to the repo.

References:

Introduction to AWS Step Functions Using Terraform as Infrastructure-as-Code Tool

juan jose orjuela — Wed, 01 Nov 2023 17:39:31 +0000

Demo in spanish

Introduction

Microservices and serverless architectures are booming in the current information technology landscape. However, coordinating and handling errors among these services can be a challenge. This is where AWS Step Functions come into play, as they provide a robust workflow tool to orchestrate microservices components and handle workflows in AWS.

In this article, we will introduce the concept of AWS Step Functions, explaining their usefulness and advantages in creating cloud-based applications. Then, as a practical case, we will show how to implement and manage AWS Step Functions using Terraform, a very popular infrastructure-as-code tool. The purpose of this article is to provide a clear understanding of AWS Step Functions and their implementation through Terraform, so you can make the most of this service and enhance the efficiency and robustness of your applications.

What are AWS Step Functions

AWS Step Functions is a fully managed workflow orchestration service that makes it easy to coordinate the components of distributed applications. This service allows developers to visually design workflows, or 'step functions,' which coordinate the components of their applications in a specific pattern, such as sequences, branches, and merges.

Features of AWS Step Functions

State Management

AWS Step Functions keep track of the state of each workflow, maintaining its activity and data at all stages.

Retries and Error Handling

Provides automatic error handling and retries.

Visualization

Offers a graphical interface to visualize and modify workflows.

Compatibility

AWS Step Functions can interact with almost all other AWS services.

Programming

Developers can program the coordination and conditional logic in their application, instead of implementing it in the code.

Advantages of Using AWS Step Functions

Resilience

AWS Step Functions has built-in error handling capabilities, making workflows resilient to failures.

Scalability

As a managed service, AWS Step Functions can scale as needed to execute workflows.

Reduced Coding

AWS Step Functions eliminates the need to write 'glue' code to coordinate microservices.

Easy to Monitor

AWS Step Functions is integrated with CloudWatch, allowing easy monitoring of workflows.

Disadvantages of Using AWS Step Functions

Cost

While AWS Step Functions can reduce the amount of code you need to write, it is not free. The cost can add up quickly for complicated or high-volume workflows.

Complexity

AWS Step Functions introduces a new level of complexity to the application, as the Step Functions service must now be managed and understood.

Time Limitations

Each execution of a step function has a maximum duration of one year. This may not be suitable for some long-term workflows.

Vendor Lock-in

By using AWS Step Functions, you are locking yourself into the AWS platform. If you ever need to migrate to another platform in the future, this could be a limiting factor.

What is Terraform

Terraform is an open-source Infrastructure as Code (IaC) tool created by HashiCorp. It allows developers to define and provide data center infrastructure using a declarative configuration language. This includes servers, storage, and networking across a variety of cloud service providers, including AWS.

Advantages of Using Terraform

Cloud-Agnostic Platform

Unlike provider-specific IaC tools like AWS CloudFormation, Terraform is cloud-agnostic, meaning it can work with any cloud service provider, including AWS, Google Cloud, Azure, and others.

Declarative Configuration Language

Terraform uses a declarative configuration language, meaning you specify what resources you want to create without having to detail the stages to create them.

State Management

Terraform maintains a state of your infrastructure and can determine what has changed since the last execution, allowing for efficient planning of changes.

Disadvantages of Using Terraform

Complexity

Although Terraform can be very powerful, it can also be complicated to configure and use correctly.

Documentation

Sometimes, the documentation for Terraform can be lacking or confusing, especially for more complex use cases.

Deployment Speed

Terraform may be slower to support new features of cloud services compared to provider-specific IaC tools.

Basic Example in Terraform "Hello World"

Here is a basic example of what a Terraform script would look like to create a single EC2 server in AWS. This is equivalent to a 'Hello World' in Terraform:



provider "aws" {
    region = "us-west-2"
}

resource "aws_instance" "example" {
    ami           = "ami-0c55b159cbfafe1f0"
    instance_type = "t2.micro"

    tags = {
        Name = "example-instance"
    }
}

This is a very simple script. Here's what's happening:

provider "aws": This specifies that we are going to use AWS as our provider for our resource. The region is specified within this block.
resource "aws_instance" "example": This defines a resource, in this case, an EC2 instance. "example" is an arbitrary name we give to this resource.
Inside the resource block, we specify the Amazon Machine Image (AMI) ID we want to use for our instance and the instance type. In this case, we are using an AMI for a basic Ubuntu instance and t2.micro for the instance type, which is the lowest cost option.
The tags block allows adding labels to the instance, in this case, simply giving the instance the name "example-instance".

To run this script, you must have Terraform installed and AWS credentials configured in your environment. Then you can initialize Terraform with terraform init, plan the execution with terraform plan, and finally apply the changes with terraform apply.

Architecture of the Example to Follow

In this section, we will look at how we can implement a simple step flow that can be implemented with the AWS Step Functions service.

General Architecture Example AWS Step Functions

The flow starts with the execution of a lambda that creates a random number between 1 and 100. This number will be validated by the flow; if the number is even, the lambda "Even" will be executed, and if it is odd, the lambda "Odd" will be executed. After this, we end the flow.

To make the example functional, we must carry out the following steps:

Create the Logic of the Lambda “Number Generator”: This lambda will simply generate a random number between 1 and 100 and then return that generated number as a response. This lambda will be developed in Node 14.x.
Create the Logic of the Lambda “Even”: This lambda will receive the previously generated number as an input parameter and will print a message in the logs specifying that the number is even. Like the previous lambda, the logic will be developed in Node 14.x.
Create the Logic of the Lambda “Odd”: This lambda will receive the previously generated number as an input parameter and will print a message in the logs specifying that the number is odd.
Create the Base Code of the Terraform Project: We are going to create an initial Terraform project by importing the AWS provider.
Generate a Packaging Mechanism for Each of the Lambdas: In order to upload this logic as a lambda function to AWS, it is necessary that they be packaged as .zip files. We will do this through Terraform.
Create Infrastructure for the 3 Lambdas through Terraform: In the code base created in Terraform, we will add the infrastructure for the 3 lambdas mentioned above and associate the logic code of each of them.
Create Infrastructure for the Step Function That We Will Use in the Terraform Project: In the code base created in Terraform, we will implement the necessary infrastructure resources to create the step function considering the flow outlined above.
Create Infrastructure Components That Allow the Step Function to Execute the Lambdas: It is necessary to add a role to the step function so that it can execute the lambdas specified above.
Test the Created Step Function!

Implementation of the Example

Now that we have explained the example that we are going to carry out with AWS Step Functions, let's implement each step.

Create the Logic of the Lambda "Number Generator"

For the lambda we're creating, we will make a folder where we will store its logic. In this folder, we will have the files package.json and main.js.

Initial Folder Structure for Lambdas

The package.json file will contain the following:



{
    "name": "number-generator",
    "version": "1.0.0",
    "description": "Number generator lambda",
    "main": "main.js",
    "dependencies": {},
    "devDependencies": {
        "aws-sdk": "^2.1045.0"
    }
}

The main.js file will have the logic to generate a random number and return it, along with the validation of whether the number is even or not:



exports.handler = async event => {
    var number = Math.floor(Math.random() * 100) + 1;
    console.log(`Generated number is: ${number}`);
    return { number: number, is_even: number % 2 === 0 };
};

Create the Logic of the Lambda Even

For the logic of the lambda even, we will do the same as for the previous lambda, having a package.json and main.js file.

package.json:



{
    "name": "even number",
    "version": "1.0.0",
    "description": "Even number lambda",
    "main": "main.js",
    "dependencies": {},
    "devDependencies": {
        "aws-sdk": "^2.1045.0"
    }
}

main.js:



exports.handler = async event => {
    console.log(`My even number is: ${event.number}`);
    return { number: event.number };
};

Create the Logic of the Lambda Odd

Lastly, the files for the lambda Odd:

package.json:



{
    "name": "odd number",
    "version": "1.0.0",
    "description": "Odd number lambda",
    "main": "main.js",
    "dependencies": {},
    "devDependencies": {
        "aws-sdk": "^2.1045.0"
    }
}

main.js:



exports.handler = async event => {
    console.log(`My odd number is: ${event.number}`);
    return { number: event.number };
};

For creating the logic of the 3 lambdas, it's important to consider that:

The aws-sdk dependency was added; for this example, it won't be used but is included to illustrate how we can import AWS's own libraries to interact with the services.
It's crucial that in each lambda folder we run the command npm i, so that the dependencies we're adding to each of the lambdas can be installed.
The final folder structure should look similar to this:

Final folder structure for the lambda logic

Creating the Base Code for the Terraform Project

Similar to the lambda logic, we need a folder to store our entire Terraform project that will help us create the infrastructure components necessary for the example to function. Initially, we'll have a folder named terraform and a file called main.tf:

Figure: Initial Folder Structure for the Terraform Project

In the main.tf file, we define the providers we will use to create our infrastructure components. A provider in Terraform is a collection of resources we can leverage to configure and manage components:

aws: This provider supplies all the infrastructure components we can utilize from Amazon Web Services (AWS).
archive: This provider provides utilities for file handling. We will use it to generate the final files for each of the Lambda functions we've been creating.



provider "aws" {
    version = "~> 3.0"
    region  = "us-east-1"
}

provider "archive" {}

Now, we should initialize the project. For this, we execute the command terraform init:

This way, we have our Terraform project's base ready for starting the creation of infrastructure components.

Generating a Packaging Mechanism for Each Lambda

The logic for each lambda must be packaged into a .zip file to upload it to AWS. For this, we'll use the archive provider.

For this, we'll create a new file in the Terraform project called lambda_resources.tf.



data "archive_file" "number_generator" {
    type        = "zip"
    source_dir  = "../lambdas/1_number_generator/"
    output_path = "../lambdas/dist/1_number_generator.zip"
}

data "archive_file" "even" {
    type        = "zip"
    source_dir  = "../lambdas/2_even/"
    output_path = "../lambdas/dist/2_even.zip"
}

data "archive_file" "odd" {
    type        = "zip"
    source_dir  = "../lambdas/3_odd/"
    output_path = "../lambdas/dist/3_odd.zip"
}

Each code block uses the archive_file provider to create a zip file from a source directory. Let's break down each line:

data archive_file number_generator: This line defines a new archive_file resource called number_generator. The word data indicates that we are obtaining data from an existing resource, rather than creating a new one.
type = zip: This line specifies the type of output file we want. In this case, we are creating a zip file.
source_dir = ../lambdas/1_number_generator/: This line specifies the source directory we want to compress. All files and subdirectories within ../lambdas/1_number_generator/ will be included in the zip file.
output_path = ../lambdas/dist/1_number_generator.zip: This line specifies the path and name of the output file. The resulting zip file will be saved in ../lambdas/dist/1_number_generator.zip.

This type of operation is quite common in serverless applications like AWS Lambda, where you need to upload your function code in zip format to AWS. Therefore, this code block helps you prepare your Lambda function for deployment, packaging the function code into a zip file that can then be uploaded to AWS.

To see the result of this, we run terraform apply. After this, we will see that the packaged lambdas have already been created:

Figure: Packaged Lambdas

Creating the Infrastructure for the 3 Lambdas Using Terraform

Now that we have the logic of the lambdas generated and ready to upload to AWS, we need to create the infrastructure in the terraform project. For this, we need to:

Create an IAM role to be used by the lambdas, which is necessary to associate permissions with each of them.
Create the infrastructure resources for each lambda and link them to the previously created role.

IAM Role for the Lambdas

For this step, we will create a new file called iam.tf with the following content:



resource "aws_iam_role" "example_lambda_role" {
    name               = "example_lambda_role_for_numbers"
    assume_role_policy = <<EOF
    {
    "Version": "2012-10-17",
    "Statement": [
        {
        "Action": "sts:AssumeRole",
        "Principal": {
            "Service": "lambda.amazonaws.com"
        },
        "Effect": "Allow",
        "Sid": ""
        }
    ]
}
    EOF
}

In this code segment, we have:

resource aws_iam_role" "example_lambda_role: This line is declaring a Terraform resource of type aws_iam_role (an IAM role in AWS) with the local name example_lambda_role. This local name is what you would use within your Terraform configuration to refer to this resource.
name = example_lambda_role_for_numbers: This is the name the IAM role will have in AWS.
assume_role_policy = EOF: This is a policy document that defines which entities are allowed to assume the role. In this case, AWS's Lambda service.

The policy itself is a JSON document containing a list of statements, and each statement defines a rule. In this case, there's a single statement:

Action: sts:AssumeRole: This action allows entities to assume the role.
Principal: Service: lambda.amazonaws.com: This is the entity that is allowed to assume the role. In this case, it's the AWS Lambda service.
Effect: Allow: This is the decision of the policy. In this case, it's allowing (Allow) the action.
Sid: This is the Statement ID of the policy. In this case, it's empty, but you can use it to give a unique identifier to each statement.

This code is essentially creating an IAM role that allows AWS Lambda functions to assume this role. This is a common pattern in AWS when you want to allow your Lambda functions to interact with other AWS services.

Lambda Infrastructure Resources

For this step, we will create a file called lambda.tf which will contain the following content:



resource "aws_lambda_function" "number_generator_lambda" {
    filename         = data.archive_file.number_generator.output_path
    source_code_hash = data.archive_file.number_generator.output_base64sha256
    function_name    = "poc_number_generator_lambda"
    role             = aws_iam_role.example_lambda_role.arn
    handler          = "1_number_generator/main.handler"
    runtime          = "nodejs18.x"
}

resource "aws_lambda_function" "even_lambda" {
    filename         = data.archive_file.even.output_path
    source_code_hash = data.archive_file.even.output_base64sha256
    function_name    = "poc_even_lambda"
    role             = aws_iam_role.example_lambda_role.arn
    handler          = "2_even/main.handler"
    runtime          = "nodejs18.x"
}

resource "aws_lambda_function" "odd_lambda" {
    filename         = data.archive_file.odd.output_path
    source_code_hash = data.archive_file.odd.output_base64sha256
    function_name    = "poc_odd_lambda"
    role             = aws_iam_role.example_lambda_role.arn
    handler          = "3_odd/main.handler"
    runtime          = "nodejs18.x"
}

This code snippet is an example of how to use Terraform to create an AWS Lambda function. AWS Lambda is a service that allows you to run your code without provisioning or managing servers. You just upload your code (known as a Lambda function), and Lambda takes care of the rest.

We will analyze each line of the code to better understand what it is doing:

resource aws_lambda_function odd_lambda: This line declares a Terraform resource of type aws_lambda_function with the local name odd_lambda. Terraform uses this local name to refer to this resource in other parts of the configuration.
filename = data.archive_file.odd.output_path: This line specifies the path of the zip file containing the code for the Lambda function. In this case, a data archive file resource is used to generate this zip file. data.archive_file.odd.output_path refers to the output path of the generated file.
source_code_hash = data.archive_file.odd.output_base64sha256: This is a hash of the Lambda function's source code. Terraform uses this hash to determine if the source code has changed and if it needs to redeploy the Lambda function.
function_name = poc_odd_lambda: This is the name that the Lambda function will have in AWS.
role = aws_iam_role.example_lambda_role.arn: This is the ARN (Amazon Resource Name) of the IAM role that the Lambda function will assume when executed. In this case, it refers to the IAM role created in the previous example.
handler = 3_odd/main.handler: This is the handler of the Lambda function, which is the function in your code that Lambda calls when the function is executed. The format is file.function. In this case, 3_odd/main.handler means that Lambda will call the handler function in the main file of the odd directory.
runtime = nodejs18.x: This is the runtime environment in which the Lambda function will execute. In this case, the function will run in a Node.js 18.x environment.

In summary, this Terraform code is creating a Lambda function that will execute code located at the specified path in a Node.js 18.x environment, will assume a specific IAM role when executed, and will have a specific name in AWS.

To test this, we will run the command terraform apply and then confirm with yes:

Caption: Resources created
Label: fig:architecture_aws_step_function

Next, if we go to the AWS console, we will see the Lambdas already created:

Caption: Resources created
Label: fig:architecture_aws_step_function

Creating Infrastructure Components to Enable Step Function to Execute Lambdas

To allow the previously created Lambdas to be executed by a Step Function, it is necessary to create a role that the Step Function can assume to execute the Lambdas. For this, we will create the following resources inside the iam.tf file:

resource aws_iam_role step_functions_role: Role that the Step Function will assume.
data aws_iam_policy_document lambda_access_policy: IAM Policy that will be associated with the Step Function's role.
resource aws_iam_policy step_functions_policy_lambda: IAM Policy resource to associate with the Step Function's role.
resource aws_iam_role_policy_attachment step_functions_to_lambda: Explicit association of the IAM Policy with the Role.

resource aws_iam_role step_functions_role



resource "aws_iam_role" "step_functions_role" {
    name = "step_functions_role_poc_sf"

    assume_role_policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
        {
            Action = "sts:AssumeRole"
            Effect = "Allow"
            Principal = {
            Service = "states.amazonaws.com"
            }
        }
        ]
    })
}

data aws_iam_policy_document lambda_access_policy



data "aws_iam_policy_document" "lambda_access_policy" {
    statement {
        actions = [
        "lambda:*"
        ]
        resources = ["*"]
    }
}

resource aws_iam_policy step_functions_policy_lambda



resource "aws_iam_policy" "step_functions_policy_lambda" {
    name   = "step_functions_policy_lambda_policy_all_poc_sf"
    policy = data.aws_iam_policy_document.lambda_access_policy.json
}

resource aws_iam_role_policy_attachment step_functions_to_lambda



resource "aws_iam_role_policy_attachment" "step_functions_to_lambda" {
    role       = aws_iam_role.step_functions_role.name
    policy_arn = aws_iam_policy.step_functions_policy_lambda.arn
}

Creating the Step Function Infrastructure for Our Terraform Project

In this section, we will create the infrastructure components that will allow us to set up a Step Function interacting with the previously created Lambdas.

For this, we will create a file named step_function.tf in our Terraform project with the following content:



# Step Functions State Machine
resource "aws_sfn_state_machine" "number_processor_sf" {
    name     = "NumberProcessorSF"
    role_arn = aws_iam_role.step_functions_role.arn

    definition = <<EOF
{
    "Comment": "execute lambdas",
    "StartAt": "NumberGenerator",
    "States": {
    "NumberGenerator": {
        "Type": "Task",
        "Resource": "${aws_lambda_function.number_generator_lambda.arn}",
        "Next": "IsNumberEven"
    },
    "IsNumberEven": {
        "Type": "Choice",
        "Choices": [
        {
            "Variable": "$.Payload.is_even",
            "BooleanEquals": true,
            "Next": "Even"
        }
        ],
        "Default": "Odd"
    },
    "Even": {
        "Type": "Task",
        "Resource": "${aws_lambda_function.even_lambda.arn}",
        "End": true
    },
    "Odd": {
        "Type": "Task",
        "Resource": "${aws_lambda_function.odd_lambda.arn}",
        "End": true
    }
    }
}
EOF
}

To apply changes to the cloud just like in previous steps, simply run a terraform apply.

Testing the Created Step Function!

In the Amazon console, we should navigate to the Step Functions service and observe that there is a step function named: NumberProcessorSF.

Detail of the Created Step Function

In the definition section, we can see a diagram with the defined flow, which is similar to the architecture defined at the beginning of the exercise.

Definition of the Created Step Function

Now, to test it, we go to the Executions tab and click the Start Execution button:

Execute the Created Step Function

In this view, we can see the flow of our execution and each of the steps taken to complete the execution. Additionally, if we click on any of the steps, we can see logs, inputs, and outputs of what is happening, thus facilitating the traceability of each execution.

This is what a successful execution looks like:

Successful Execution

Informative Links

Spanish version: https://jjoc007.com/introducci%C3%B3n-a-aws-step-functions-usando-terraform-como-herramienta-de-infrastructura-como-c%C3%B3digo-e2add2930269
Example on GitHub: https://github.com/jjoc007/poc-step-function
AWS Step Functions: https://aws.amazon.com/es/step-functions/
Terraform: https://www.terraform.io/