DEV Community: Massimo Biagioli

RaspberryPi 3 Homelab & Vibe "IaC"

Massimo Biagioli — Wed, 24 Dec 2025 10:40:56 +0000

Per Natale vorrei regalarmi un nuovo RaspberryPi ... Ops, ma questo è un Deja-Vu!
Proprio un anno fa, in questo articolo, avevo iniziato a fare esperimenti sul mio vecchio RaspberryPi 3 per provare a mettere in piedi una Homelab.
Però, nel frattempo, qualcosa è cambiato: l'avvento del vibe coding ha modificato il modo in cui uno sviluppatore si approccia al codice.
Quindi ho tentato di ricalibrare il mio esperimento, utilizzando il vibe coding per creare un'infrastruttura "su misura" per il mio raspberry.

Il "Kickstart"
Ovviamente, non ho dato a lui "le chiavi", chiedendogli banalmente "Hey, ho bisogno che mi crei un'infrastruttura per il mio RaspberryPi 3". No, non avrebbe funzionato. E non è quello che volevo.
La "Big Picture" volevo averla bene in testa io.
Era doveroso per me pormi alcune domande:

Quale stack tecnologico usare?
Come interagire con lo strumento che avrei creato?

E dare dei vincoli importanti:

Non dovrà richiedere configurazioni complicate per farlo funzionare
Dovrà essere facile da manutenere
Dovrà essere facilmente estendibile

E quindi ho tracciato il perimetro e stabilito le regole del gioco:

Avrei usato Ansible
Avrei interagito con lo strumento tramite un semplice Makefile: è uno strumento per me, non ho bisogno di una UI. Digitando "make" lo strumento mi avrebbe fornito tutte le operazioni disponibili.
Feedback loop rapido: basta lanciare il comando ed osservare l'output, niente di più, niente di meno.

Tanti Playbooks, ma un'unica filosofia
L'input era chiaro: un playbook deve svolgere solamente un'operazione. Queste al momento quelle disponibili:

Usage: make run-playbook PLAYBOOK=update [VERBOSE=true]

📚 Available Playbooks:
  📋 deploy
  📋 git
  📋 git-cleanup
  📋 grafana
  📋 grafana-cleanup
  📋 health-check
  📋 loki
  📋 loki-cleanup
  📋 mariadb
  📋 mariadb-cleanup
  📋 mosquitto
  📋 mosquitto-cleanup
  📋 nginx
  📋 nginx-cleanup
  📋 node
  📋 node-cleanup
  📋 redis
  📋 redis-cleanup
  📋 update
  📋 uv
  📋 uv-cleanup

In questo modo, è anche facile tramite vibe coding installare un nuovo componente, perchè l'agente (nel mio caso GPT-4.1) può basarsi su degli esempi funzionanti.
Ecco un esempio di come creo un nuovo playbook:

Crea un nuovo playbook: postgresql.
Crea anche il playbook di cleanup.
Aggiorna il Makefile.
Aggiorna il Readme.

Ovviamente, quando l'agente può basarsi su degli esempi, si muoverà molto meglio rispetto a quando deve inventare.
E spesso ci prende "al primo colpo".

Obiettivi raggiunti
Ok, ma cosa avrebbe dovuto fare questo strumento?
Gli obiettivi che mi ero prefissato (e che ho raggiunto grazie al vibe coding in tempi rapidi, considerando che ci ho investito solo poche ore) sono i seguenti:

Installazione di servizi (MariaDB, Nginx, Mosquitto, ecc.)
Installazione web app (Python) come servizio linux
Abilitate/Disabilitare/Verificare stato dei servizi installati
Healthcheck sistema (produzione di un report)

Ma l'obiettivo più importante raggiunto è stato ... il non acquisto di un nuovo RaspberryPi 5!

Conclusioni
Il vibe coding può aiutare moltissimo per velocizzare la fase di sviluppo, permettendoci di concentrarci sulla "Big Picture", spostando i ragionamenti più ad alto livello. Ma non dimentichiamoci mai di controllare ciò che l'agente genera per noi, perchè non sempre è ciò che vogliamo, quindi non va persa la skill tecnica, sarebbe un errore grossolano dimenticarsi di sviluppare, e voler "disimparare". L'intento è quello di "amplificare".

Trovate il repository di questo progetto in Github a questo indirizzo.

Empower Your Development: Simulating AWS with LocalStack for Efficient Local Testing

Massimo Biagioli — Mon, 26 May 2025 12:38:42 +0000

Intro

In the development of cloud-based applications, the ability to replicate AWS services on local environments is invaluable, saving time and resources during the testing phase. LocalStack provides a comprehensive solution by emulating a wide range of AWS services locally, enabling developers to test their cloud applications without incurring the costs and complexities associated with connecting to actual AWS services. This robust tool supports services such as Lambda, S3, DynamoDB, and many more, offering a seamless integration with your existing AWS workflows. In this exploration, we will discuss how LocalStack can help developers and teams streamline their development process by providing a reliable and scalable local testing environment. Learn how to set up your LocalStack, integrate with your development pipeline, and leverage this powerful tool to enhance productivity, reduce development costs, and accelerate your go-to-market strategy. Embrace the power of local emulation with LocalStack and transform your cloud application development lifecycle.

Navigating the Complexity of Cloud Development

Cloud development presents a multifaceted challenge that significantly differs from traditional web development. Unlike monolithic web applications, which can often be built and tested in a single environment, cloud applications are inherently distributed, relying on a diverse array of services that interact with each other in complex ways. This architecture introduces additional layers of complexity, including service orchestration, network configurations, and data management across multiple cloud services. Moreover, the dynamic nature of the cloud environment necessitates considerations for scalability, latency, and security that are rarely required in conventional applications. Developers must navigate intricate dependencies and ensure that their applications remain resilient and performant under varying conditions. Understanding these complexities is crucial in adopting effective strategies and tools, such as LocalStack, that simplify local testing and streamline development workflows. Embracing these challenges is key to leveraging the full potential of cloud-native applications while mitigating risks associated with deployment and integration.

Use Case: Managing a New Booking in a Cloud-Based Booking System

Consider a cloud-based booking system where managing a new reservation is a key function. When a customer makes a new booking, the process involves several important steps supported by various AWS services. First, the booking details are saved in DynamoDB, which offers a scalable and easy-to-use NoSQL database for storing and accessing reservation information. At the same time, the system creates a confirmation PDF receipt to verify the booking, ensuring customers have a clear record of their reservation. This receipt is then stored in an S3 bucket, utilizing AWS's object storage capabilities for durability and easy access. By using these services, the platform ensures efficient data management and provides a smooth user experience. Implementing LocalStack allows developers to simulate this entire process locally, helping them test and improve each component —ultimately saving time and reducing costs while ensuring strong functionality before deployment.

A First Approach: Leveraging a Real AWS Environment

Testing in a real AWS environment can provide a valuable perspective on how cloud applications will perform in production. The primary advantage of this approach is:

Accurate Insights: Provides a true representation of service behavior, performance, and interactions within the full AWS ecosystem, allowing for the identification of potential integration issues.

However, relying solely on a real AWS environment for testing comes with notable drawbacks:

Cost: The expenses related to running resources in AWS can quickly accumulate, especially during extensive testing phases or with suboptimal resource management.
Complexity: Managing AWS environments can lead to longer setup and teardown times, making it challenging to iterate quickly on development and testing cycles.
Unpredictable Issues: External factors such as throttling limits, network latency, or availability concerns may arise in the shared environment, potentially distorting test results and making it difficult to consistently replicate scenarios.

Consequently, while using a real AWS environment has its merits, it is important to supplement this approach with localized testing solutions to enhance efficiency and control in the development process.

Introducing LocalStack: Empowering Local Development

LocalStack is an open-source tool that emulates Amazon Web Services (AWS) on your local machine, enabling developers to build, test, and deploy cloud applications without incurring the costs and complexities associated with actual cloud services. By providing a full-fledged local environment that mimics the behavior of AWS services, LocalStack supports a wide range of functionalities including Lambda, S3, DynamoDB, and many others.

The primary benefit of using LocalStack in the development lifecycle is the significant enhancement it brings to the feedback loop. Developers can quickly prototype and test new features in a controlled setting, leading to rapid iterations without the delays associated with deploying to the cloud. This accelerated feedback allows teams to identify and fix issues early in the development process, reducing the likelihood of regressions and enhancing the overall quality of releases.

Moreover, by integrating LocalStack into your development pipeline, you can simulate realistic scenarios that closely mirror production environments. This ensures that code changes are thoroughly tested before deployment, leading to more reliable applications and smoother rollouts. As a result, LocalStack not only streamlines development workflows but also fosters a culture of quality and accountability, allowing teams to deliver robust cloud-native applications with confidence.

You can easily run LocalStack using Docker, making it even more accessible for development teams. Here’s a simple example of a docker-compose.yml file to get you started:

services: 
  localstack: 
    container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}" 
    image: localstack/localstack 
    ports: 
      - "127.0.0.1:4566:4566"            # LocalStack Gateway 
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range 
    environment: 
      - DEBUG=${DEBUG:-1} 
      - LS_LOG=info 
      - LOCALSTACK_HOST=localstack 
      - HOSTNAME=localstack 
    volumes: 
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack" 
      - "/var/run/docker.sock:/var/run/docker.sock"

Structuring the Web Application: A Lambda-Based Architecture

Our web application follows a modern design pattern known as a "lambdalith," where we use AWS Lambda functions to create a scalable and cohesive application built with Python and FastAPI. This approach allows us to minimize the need for infrastructure management while maximizing scalability and responsiveness.

The application's architecture will include the following key components when deployed on AWS:

API Gateway: This acts as the entry point for incoming requests. API Gateway makes our FastAPI application accessible as RESTful endpoints, handling request routing, authentication, and response formatting. It ensures efficient and secure communication between clients and our backend functions.
Lambda Layer: To manage dependencies efficiently, we use Lambda Layers. This feature allows us to package shared libraries or code that can be used across multiple Lambda functions. By separating the core application logic from its dependencies, we can optimize deployment times and keep our functions lightweight.
Lambda Function: The core of our application is in the Lambda functions, which run the business logic defined using FastAPI. For this example, we use Mangum, a wrapper that transforms a FastAPI application into a Lambda-compatible function.

By structuring our web application in this way, we can take full advantage of serverless technologies while keeping our codebase maintainable and organized.

main.py

import os 

from fastapi import FastAPI 
from fastapi.middleware.cors import CORSMiddleware 
from mangum import Mangum 

from app.routers import health, booking 


app = FastAPI( 
   ... 

) 

app.include_router(health.router, prefix="/health", tags=["health"]) 
app.include_router(booking.router, prefix="/booking", tags=["booking"]) 


handler = Mangum(app)

Deciding What to Include in LocalStack: Focus on External Dependencies

When working with LocalStack, it's important to determine which components of your application to simulate locally. Since Mangum acts as a wrapper to transform a FastAPI application into a Lambda-compatible function, we can simplify our setup by focusing on external dependencies, such as S3 or DynamoDB, rather than trying to simulate the entire Lambda and API Gateway stack.

In this example, we chose the second approach. Here are three key advantages of this choice:

Simplified Development: By only simulating the essential AWS services that our application interacts with, we reduce complexity. This makes it easier for developers to focus on building and testing core functionality without getting sidetracked by configuring Lambda or API Gateway in a local environment.
Faster Feedback Loop: Concentrating on external dependencies allows for quicker testing and debugging. With LocalStack running only the necessary services, developers can rapidly iterate on their application, receiving immediate feedback and enhancing productivity. This streamlined approach accelerates development cycles.
Efficient Resource Usage: By limiting the number of resources that need to be simulated, we can run tests with a lighter environment. This makes it easier to manage local resources and keeps the testing setup straightforward, allowing developers to focus more on code quality and less on infrastructure concerns.

By focusing our LocalStack setup on external dependencies, we maintain an efficient local testing environment that supports development efforts while relying on Mangum to handle the conversion of our FastAPI application into a Lambda function.

Below is the Terraform code for creating the necessary S3 bucket and DynamoDB table:

s3.tf

resource "aws_s3_bucket" "booking_bucket" { 
  bucket = "booking" 
}

dynamodb.tf

resource "aws_dynamodb_table" "booking_table" { 
  name         = "booking" 
  billing_mode = "PAY_PER_REQUEST" 
  hash_key     = "customer_name" 
  range_key    = "check_in" 

  attribute { 
    name = "customer_name" 
    type = "S" 
  } 

  attribute { 
    name = "check_in" 
    type = "S" 
  } 
}

Use Case: Create Booking

When a customer creates a new booking, the application needs to seamlessly store the booking details and generate a confirmation receipt. To achieve this, we use the same library, Boto3, to interact with both S3 and DynamoDB when using LocalStack and when working with real AWS services. This creates a "transparent" development experience, as the code remains largely unchanged between local and production environments; developers simply need to adjust the connection parameters.

Here's a step-by-step breakdown of how the process works:

Storing Booking Details in DynamoDB: When a booking is created, the application saves the booking information—such as booking ID, customer details, and reservation dates—in a DynamoDB table. Using Boto3, the same code can connect to either LocalStack or the actual DynamoDB service in AWS by changing the endpoint URL.
Generating a Confirmation PDF
: After the booking details are saved, the system generates a PDF receipt confirming the booking. This receipt is then stored in an S3 bucket, again utilizing Boto3 to handle the upload.
Storing the Receipt in S3: The receipt is saved to an S3 bucket, providing customers with a downloadable file as proof of their booking. The code for interacting with S3 remains the same, requiring only a change in the connection parameters to switch between LocalStack and AWS.

create_booking.py

class CreateBookingUseCase: 
    def __init__( 
        self, 
        dynamodb_client: DynamoDBClientDependency, 
        s3_client: S3ClientDependency, 
    ): 
        self.dynamodb_client = dynamodb_client 
        self.s3_client = s3_client 

    async def execute(self, command: CreateBooking) -> Booking: 
        booking_id = uuid.uuid4() 

        booking = Booking( 
            **command.model_dump(), 
            booking_id=str(booking_id), 
        ) 

        pdf_path = generate_pdf(booking) 

        s3_key = f"{booking_id}.pdf" 

        self.s3_client.upload_file( 
            pdf_path, 
            settings.booking_bucket_name, 
            s3_key 
        ) 

        self.dynamodb_client.put_item( 
            TableName=settings.booking_table_name, 
            Item={ 
                'booking_id': {'S': str(booking_id)}, 
                'customer_name': {'S': booking.customer_name}, 
                'check_in': {'S': str(booking.check_in)}, 
                'check_out': {'S': str(booking.check_out)}, 
                'room_type': {'S': booking.room_type}, 
                'pdf_url': {'S': f"s3://{settings.booking_bucket_name}/{s3_key}"} 
            } 
        ) 

        return booking

libs/aws.py

start_local = os.environ.get("START_LOCAL", '0') 

LOCALSTACK_ENDPOINT_URL = 'http://localhost:4566' 
LOCALSTACK_AWS_REGION = 'us-east-1' 
LOCALSTACK_ACCESS_KEY_ID = 'test' 
LOCALSTACK_SECRET_ACCESS_KEY = 'test' 


def get_s3_client(): 
    if start_local == '1': 
        return boto3.client( 
            's3', 
            endpoint_url=os.getenv('LOCALSTACK_ENDPOINT', LOCALSTACK_ENDPOINT_URL), 
            region_name=os.getenv('AWS_REGION', LOCALSTACK_AWS_REGION), 
            aws_access_key_id=LOCALSTACK_ACCESS_KEY_ID, 
            aws_secret_access_key=LOCALSTACK_SECRET_ACCESS_KEY 
        ) 
    else: 
        return boto3.client('s3') 


def get_dynamodb_client(): 
    if start_local == '1': 
        return boto3.client( 
            'dynamodb', 
            endpoint_url=os.getenv('LOCALSTACK_ENDPOINT', LOCALSTACK_ENDPOINT_URL), 
            region_name=os.getenv('AWS_REGION', LOCALSTACK_AWS_REGION), 
            aws_access_key_id=LOCALSTACK_ACCESS_KEY_ID, 
            aws_secret_access_key=LOCALSTACK_SECRET_ACCESS_KEY 
        ) 
    else: 
        return boto3.client("dynamodb", region_name=settings.region) 


S3ClientDependency = Annotated[Any, Depends(get_s3_client)] 
DynamoDBClientDependency = Annotated[Any, Depends(get_dynamodb_client)]

Start the FastAPI local application with localstack

START_LOCAL=1 poetry run uvicorn app.main:app --reload

End-to-End Testing without Mocking AWS Services

With LocalStack, developers can perform true end-to-end testing without the need to mock AWS services. This allows teams to validate the entire workflow of their applications in a real-world environment before deployment. By simulating services like DynamoDB and S3, developers can verify that their code interacts correctly with these components, ensuring that functionalities such as creating bookings and storing receipts work as intended. This approach provides invaluable feedback, exposing potential issues early in the development process. Since the tests run against services that behave like their AWS counterparts, it increases confidence that the application will perform reliably in production. Overall, LocalStack helps maintain a high level of code quality by allowing thorough testing without the complexities and costs associated with using real AWS resources.

Verifying Operation Success with AWS CLI

Using the AWS Command Line Interface (CLI), developers can easily verify the success of operations performed on AWS services like DynamoDB and S3. With LocalStack running, the AWS CLI can be configured to point to the local endpoints, allowing for straightforward management and verification of resources.

For instance, to check the data within a DynamoDB table, you can run the following command:

aws --endpoint-url=http://localhost:4566 --profile localstack dynamodb scan --table-name booking

This will return a list of all items stored in the "booking" table, allowing you to confirm that your application is correctly saving booking details.

Similarly, to read the contents of an S3 bucket or download a file to your local machine, you can use:

aws --endpoint-url=http://localhost:4566 --profile localstack s3 ls s3://booking

or to download a specific file:

aws --endpoint-url=http://localhost:4566 --profile localstack cp s3://booking/$(key) ./downloads

These commands facilitate the verification of operations and help ensure that the application is functioning as expected, both locally with LocalStack and when deployed to the cloud. Leveraging the AWS CLI enhances the development workflow by providing direct access to data, making it easier to troubleshoot and validate application behavior.

As an alternative to configuring the AWS profile, developers can use awscli-local, a wrapper for the AWS CLI designed specifically for LocalStack. This tool simplifies the process by automatically setting the correct parameters and endpoints for local development.

Conclusion

Using LocalStack in your cloud application development streamlines testing and validation without relying on real AWS resources. By focusing on external dependencies and utilizing tools like Boto3 and the AWS CLI, developers can create a seamless environment that mirrors production. This approach enhances productivity, increases confidence in application reliability, and leads to higher-quality software before deployment in the cloud.

For a practical example of these concepts in action, you can explore the repository at this GitHub link.

Infusing Conversational AI in Your Business with Amazon Lex

Massimo Biagioli — Tue, 29 Apr 2025 06:20:27 +0000

Intro

In an era where instant and interactive customer service is the norm, Amazon Lex emerges as a transformative solution, enabling businesses to build sophisticated, AI-powered chatbots and voice assistants.

This technology leverages the same deep learning functionalities that power Amazon Alexa, making it possible to design conversational interfaces that genuinely understand the user's intent.

Amazon Lex automates interactions, offering a seamless, 24/7 customer engagement model that scales as your business grows.

This not only enhances user experience but also significantly reduces operational costs.

Business Needs

In today's fast-paced digital landscape, companies face a critical challenge: meeting customer demands for immediate support without straining their resources.

Long wait times and inadequate responses can lead to frustration and lost trust. There is a pressing need for solutions that enhance service efficiency while minimizing operational costs.

As customer expectations continue to rise, the inability to provide personalized interactions risks driving customers away.

Therefore, leveraging conversational AI like Amazon Lex becomes essential in effectively addressing these pain points.

Introducing Amazon Lex

Amazon Lex is a powerful service that enables businesses to build conversational interfaces using voice and text, effectively addressing the challenges of providing timely and accurate customer support.

By automating interactions, it significantly reduces response times and improves user satisfaction.

Key Features:

Natural Language Understanding (NLU): Amazon Lex uses advanced machine learning to understand user intents and context, improving the accuracy of interactions.
Multi-Channel Support: The service allows seamless integration with various platforms, including web applications, mobile apps, and messaging services, ensuring a consistent user experience.
Scalability: Lex is designed to easily scale with your business needs, efficiently handling increasing interaction volumes as your customer base grows.

Key Concepts in Amazon Lex

To effectively utilize Amazon Lex, it is essential to understand its key concepts:

Utterance: An utterance is what the user says or types to interact with the chatbot. It represents the user's input, which Lex will interpret to determine the desired action.
Intent: An intent is a mapping between what the user wants to accomplish and a specific action that the application should take. Each intent corresponds to a unique user goal, such as retrieving information or performing an action.
Slots: Slots are variables that capture specific pieces of information from the user's utterance needed to fulfill an intent. They act as placeholders that get filled in with data extracted from the user's input, such as dates or locations.

Consider the following examples of user utterances that correspond to an intent for retrieving warehouse movements:

Get all movements for warehouse {warehouse} on {date}
Give me all movements from warehouse {warehouse} on {date}

In these examples, {warehouse} and {date} are slots that will be populated with the specific warehouse name and date provided by the user, allowing the system to retrieve the relevant information.

A First, Simple Scenario

Imagine needing a chatbot integrated into an inventory management system. When a user interacts with the chatbot, it triggers Amazon Lex, which processes the user's request. Lex then invokes an AWS Lambda function that interprets the prompt and queries data stored in DynamoDB. After retrieving the necessary information, the Lambda function formats the response and sends it back to Lex. Finally, Lex delivers this information back to the user, ensuring a seamless and efficient interaction.

Let’s inspect the lambda

lambda_handler.py

from warehouse_service import get_movements_by_warehouse_and_date 


def lambda_handler(event, context): 
    slots = event['currentIntent']['slots'] 
    warehouse = slots['warehouse'] 
    date = slots['date'] 

    movements = get_movements_by_warehouse_and_date(warehouse, date) 
    if movements: 
        movement_list = ", ".join(m.__str__() for m in movements) 
        response_message = f"Sure, here is the list of movements: {movement_list}." 
    else: 
        response_message = "There are no recorded movements for the given warehouse and date." 

    return { 
        'dialogAction': { 
            'type': 'Close', 
            'fulfillmentState': 'Fulfilled', 
            'message': { 
                'contentType': 'PlainText', 
                'content': response_message 
            } 
        } 
    }

warehouse_service.py

import boto3 
from boto3.dynamodb.conditions import Key 
from botocore.exceptions import ClientError 

from warehouse_movement_item import WarehouseMovementItem 

TABLE_NAME = 'devrel-chatbot-dev-warehouse' 

def get_movements_by_warehouse_and_date(warehouse: str, operation_date: str) -> list[WarehouseMovementItem]: 
    dynamodb = boto3.resource('dynamodb') 
    table = dynamodb.Table(TABLE_NAME) 

    try: 
        response = table.query( 
            KeyConditionExpression=Key('pk').eq(warehouse) & 
                                   Key('sk').eq(operation_date) 
        ) 

        items = response.get('Items', []) 
        return [WarehouseMovementItem.from_dict(item) for item in items] 

    except ClientError as e: 
        print(f"Failed to retrieve items: {e.response['Error']['Message']}") 
        return []

This is the DynamoDB table’s data:

Setup the ChatBot with IaC with Terraform

Terraform is an open-source tool that enables Infrastructure as Code (IaC), allowing you to define and provision cloud resources through configuration files.

The entire infrastructure, including AWS Lambda and DynamoDB, has already been provisioned with Terraform.

Below are code snippets to set up a chatbot using Amazon Lex with Terraform.

resource "aws_lex_bot" "warehouse_bot" {
  name                        = "WarehouseBot"
  description                 = "Chatbot for warehouse movements queries"
  process_behavior            = "BUILD"
  idle_session_ttl_in_seconds = 300 # 5 minutes
  enable_model_improvements   = true
  locale                      = "en-US"
  child_directed              = false

  abort_statement {
    message {
      content      = "Sorry, I couldn't process your request. Please try again."
      content_type = "PlainText"
    }
  }

  clarification_prompt {
    max_attempts = 2
    message {
      content      = "I didn't understand you. Could you please rephrase your request?"
      content_type = "PlainText"
    }
  }

  intent {
    intent_name    = aws_lex_intent.warehouse_movements.name
    intent_version = aws_lex_intent.warehouse_movements.version
  }
}

resource "aws_lex_intent" "warehouse_movements" {
  name        = "GetWarehouseMovements"
  description = "Intent to get warehouse movements for a specific date"

  sample_utterances = [
    "Get all movements for warehouse {warehouse} on {date}",
    "Show me movements for warehouse {warehouse} on {date}",
    "List all warehouse {warehouse} movements for {date}",
    "Give me all movements from warehouse {warehouse} on {date}"
  ]

  fulfillment_activity {
    type = "CodeHook"
    code_hook {
      message_version = "1.0"
      uri             = aws_lambda_function.chatbot_lambda_function.arn
    }
  }

  slot {
    name           = "warehouse"
    description    = "The warehouse identifier"
    slot_constraint = "Required"
    slot_type      = "AMAZON.AlphaNumeric"
    value_elicitation_prompt {
      max_attempts = 2
      message {
        content      = "Which warehouse are you interested in?"
        content_type = "PlainText"
      }
    }
  }

  slot {
    name           = "date"
    description    = "The date of movements"
    slot_constraint = "Required"
    slot_type      = "AMAZON.AlphaNumeric"
    value_elicitation_prompt {
      max_attempts = 2
      message {
        content      = "For which date would you like to see movements?"
        content_type = "PlainText"
      }
    }
  }
}

resource "aws_lambda_permission" "lex_invoke_permission_generic" {
  statement_id  = "AllowExecutionFromLexGeneric"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.chatbot_lambda_function.function_name
  principal     = "lex.amazonaws.com"
}

Linking Lambda and the Chatbot

The connection between the AWS Lambda function and the chatbot in Amazon Lex is established through the “fulfillment_activity” section, utilizing a code hook. This code hook specifies that the Lambda function will be invoked to handle the logic required to fulfill the user's intent.

When a user interacts with the chatbot and their request is matched to a specific intent, the corresponding Lambda function is triggered, allowing it to process the necessary data and return a response back to Amazon Lex for the user.

This integration ensures dynamic and efficient handling of user interactions.

Let's See the Chatbot in Action

Using the AWS Management Console, it is possible to test the chatbot in real-time, allowing users to interact with it directly and observe its responses to different inputs.

A More Complex Scenario - Leveraging Generative AI with Bedrock

Now that the foundational components for building a chatbot have been established, it's possible to explore more advanced functionalities by integrating the Lambda function not only with DynamoDB but also with powerful services like Amazon Bedrock. Amazon Bedrock is a fully managed service that makes it easier for developers to build and scale generative AI applications using foundation models.

By leveraging Bedrock, businesses can enhance user interactions by utilizing advanced natural language processing capabilities, enabling more nuanced conversations with customers. This integration can significantly improve the chatbot's ability to understand complex queries and deliver more insightful responses.

From a business perspective, this translates to a superior customer experience, as users can engage in more meaningful dialogues and obtain relevant information quickly. Moreover, by providing a richer interaction model, companies can drive higher customer satisfaction and loyalty, ultimately leading to increased operational efficiency and effectiveness in meeting user needs.

Conclusion

In an increasingly competitive landscape, businesses face the critical challenge of meeting customer expectations for fast and effective support. By addressing these needs through innovative solutions, companies can enhance customer satisfaction and operational efficiency. Generative AI serves as a powerful tool in this regard, enabling more sophisticated interactions that go beyond basic queries.

Integrating a chatbot with advanced services, such as Amazon Bedrock, empowers businesses to create dynamic and meaningful conversations with users. These interactions not only provide immediate answers but also foster a deeper understanding of customer needs.

AWS stands out as a leader in offering a comprehensive suite of interconnected services that can be easily assembled to meet varying business demands. By leveraging AWS's powerful tools, organizations can build robust solutions that align with both user expectations and business objectives, ultimately driving growth and success in their respective markets.

For a practical example of these concepts in action, you can explore the repository at this GitHub link.

Vecchio RaspberryPi, Nuova HomeLab!

Massimo Biagioli — Sun, 29 Dec 2024 16:28:57 +0000

Per Natale vorrei regalarmi un nuovo RaspberryPi
E' iniziata così questa esperienza. La voglia di metter su una HomeLab con un nuovo RaspberryPi 5, tanto c'è Natale, quale miglior occasione per regalarselo e cestinare il vecchio RaspberryPi 3?
E invece no. Il RaspberryPi 3 non si butta. Il RaspberryPi 3 può ancora regalare delle soddisfazioni. Il RaspberryPi 3 mi ha convinto nuovamente e costruirò la mia nuova HomeLab su misura per lui.
In fondo, a Natale, siamo tutti più buoni, no?

Iniziamo
Per prima cosa, dobbiamo assicurarci di poterci collegare al nostro RaspberryPi.
Generiamo quindi una chiave ssh:

ssh-keygen -t rsa -b 2048

e copiamola dentro al RaspberryPi:

ssh-copy-id user@raspberry_pi_ip_address

Automatizziamo tutto con Ansible
Ora, vogliamo veramente installare tutto il software a mano sul nostro RaspberryPi? Oppure vogliamo pensare di automatizzare questo processo il più possibile? Ecco, io ho optato per la seconda opzione, ed ho scelto Ansible come strumento per risolvere questo problema.
Nel mio caso, per installare Ansible sul mio Macbook, ho semplicemente eseguito questo comando:

brew install ansible

Progettiamo la nostra Infrastruttura
Sicuramente la parte più "succosa" di tutto questo percorso è la creazione dell'infrastruttura.
Per prima cosa, dobbiamo creare il file inventory.ini dove indicare le "coordinate" per raggiungere il nostro RaspberryPi:

[raspberry_pi]
raspberrypi ansible_host=raspberrypi_host ansible_user=user ansible_become=true

Ora, possiamo concentrarci sulla creazione dei vari "playbooks", questi sono quelli individuati per iniziare:

update: aggiornamento dei pacchetti
install-docker
install-dockge

Playbook "update": aggiornamento dei pacchetti
Questo playbook (file: playbooks/update.yml) consente di tenere sempre aggiornato il nostro RaspberryPi.

---
- name: Update Raspberry Pi
  hosts: raspberry_pi
  gather_facts: true
  tasks:
    - name: Update the package cache
      apt:
        update_cache: true
        cache_valid_time: 3600

    - name: Upgrade all packages to the latest version
      apt:
        upgrade: dist
        autoclean: true
        autoremove: true

    - name: Check if a reboot is required
      stat:
        path: /var/run/reboot-required
      register: reboot_required

    - name: Reboot the system if a reboot is required
      reboot:
        msg: "Rebooting the system to complete the update"
        connect_timeout: 5
      when: reboot_required.stat.exists

E può essere lanciato tramite il comando:

ansible-playbook -i inventory.ini playbooks/update.yml

In questi casi, però, preferisco sempre creare un Makefile in modo da poter lanciare tutti i comandi in maniera più semplice.
In questo caso:

make update

(il comando make senza argomenti restituisce la lista di tutti i comandi disponibili, e dal mio punto di vista è davvero molto utile).

Playbook "install-docker": aggiornamento di docker
Sicuramente docker è un elemento imprescindibile nella mia HomeLab.
Il playbook playbooks/install-docker.yml consente di installare docker nel nostro RaspberryPi.

---
- name: Install Docker on Raspberry Pi
  hosts: raspberry_pi
  become: true
  vars_files:
    - vars/main.yml
  tasks:
    - name: Install prerequisites for APT
      apt:
        name:
          - apt-transport-https
          - ca-certificates
          - curl
          - software-properties-common
        state: present
        update_cache: true

    - name: Add Docker's GPG key
      ansible.builtin.shell: |
        set -o pipefail
        mkdir -p /etc/apt/keyrings
        curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
      args:
        executable: /bin/bash
        creates: /etc/apt/keyrings/docker.gpg  # Run only if the key file doesn't already exist

    - name: Add Docker's repository
      ansible.builtin.apt_repository:
        repo: "deb [arch=armhf signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable"
        state: present

    - name: Update APT cache
      apt:
        update_cache: true

    - name: Install Docker
      apt:
        name: docker-ce
        state: present

    - name: Start and enable Docker service
      systemd:
        name: docker
        enabled: true
        state: started

    - name: Add docker user user to the Docker group (optional)
      ansible.builtin.user:
        name: "{{ docker_user }}"
        groups: docker
        append: true
        state: present

    - name: Check Docker version
      ansible.builtin.shell: |
        set -o pipefail
        docker --version
      args:
        executable: /bin/bash
      register: docker_version
      changed_when: false

    - name: Display the installed Docker version
      ansible.builtin.debug:
        msg: "{{ docker_version.stdout }}"

Nel playbook viene utilizzata la variabile docker_user, che va definita nel file playbooks/vars/main.yml:

docker_user: pi

Playbook "install-dockge": aggiornamento di dockge
Per orchestrare i container utilizzeremo il software dockge.
Il playbook playbooks/install-dockge.yml consente di installare docker nel nostro RaspberryPi.

---
- name: Install Dockge using Docker Compose
  hosts: raspberry_pi
  become: true

  tasks:
    - name: Ensure the required directories exist
      file:
        path: "{{ item }}"
        state: directory
        owner: root
        group: root
        mode: '0755'
      loop:
        - /opt/stacks
        - /opt/dockge

    - name: Download compose.yaml for Dockge
      get_url:
        url: "https://dockge.kuma.pet/compose.yaml?port=5001&stacksPath=%2Fopt%2Fstacks"
        dest: /opt/dockge/compose.yaml
        owner: root
        group: root
        mode: '0644'

    - name: Start Dockge using Docker Compose
      community.docker.docker_compose_v2:
        project_src: /opt/dockge
        state: present
      register: output

It Works!
Per accedere alla console di dockge, basta collegarsi all'indirizzo:

http://<raspberrypi-host>:5001

La prima volta, occorre creare un'utenza di ammnistratore.
Questo il risultato:

Proviamo a creare uno stack di prova
Proviamo ora a creare uno stack con nginx:

Il servizio nginx ora è attivo sulla porta 9080 del nostro RaspberryPi, se proviamo ad aprire la pagina possiamo vedere che funziona:

Conclusioni
Questo vuole essere solo un punto di partenza per la mia HomeLab.
Durante questo percorso ho imparato che a volte l'obsolescenza è solo nella nostra testa, perchè semplicemente vogliamo qualcosa di nuovo. Parliamo molto spesso di valore, ma quando ci troviamo di fronte a delle scelte, a volte scegliamo di "buttar via" anzichè sfruttare bene ciò che si ha. E questo non vale solo per il software purtroppo.
Altra cosa che ho imparato, e che considero molto importante: la "DevOps" culture è anche nelle piccole cose, e spesso la pigrizia è un nemico duro da sconfiggere, perchè applica una resistenza molto forte al cambiamento. Sperimentare per migliorarsi, applicare e sbagliare per costruire: un "mantra" che mi piace applicare, e che voglio continuare a fare mio anche nei prossimi anni.

Se volete dare un'occhiata al mio repository della infra, questo è il link.

DynamoDB Single Table Design

Massimo Biagioli — Fri, 01 Mar 2024 16:52:33 +0000

One Table, Many Possibilities: Exploring Single Table Design

Dive into the realm of DynamoDB's Single Table Design to demystify the complexities of database management. Say farewell to convoluted modeling, as DynamoDB's intrinsic capabilities empower developers to effortlessly handle a myriad of entities within a unified table structure. This article guides you through the process, highlighting how embracing a straightforward approach to DynamoDB usage can significantly enhance the efficiency of your development journey. Discover the power of simplicity in database design and streamline your DynamoDB experience for optimal results.

Use Case: Modeling Entities in DynamoDB's Single Table Design

In our scenario, where entities like "Area," "Network," and "Device" coexist, we leverage DynamoDB's Single Table Design to seamlessly organize and retrieve this interconnected data.
By adopting this approach, you harness DynamoDB's inherent flexibility, allowing you to navigate through your data effortlessly. This design choice aligns with the philosophy of Single Table Design, emphasizing simplicity and ease of use in modeling complex relationships within DynamoDB.

Essential DynamoDB Knowledge: Unraveling Key Concepts

Partition Key: In DynamoDB, each item in a table is uniquely identified by a primary key, which consists of a partition key. The partition key is used to distribute data across multiple partitions for scalability. It's crucial for efficient data retrieval, especially when working with large datasets.
Sort Key (Range Key): The primary key in DynamoDB can also include a sort key (or range key). This adds a second level of organization, allowing items with the same partition key to be sorted and stored together. It enables powerful querying and sorting operations within a partition.
Global Secondary Index (GSI): GSIs in DynamoDB provide an alternative way to query your data. They allow you to create additional indexes outside the primary key, enabling different access patterns. A GSI has its own partition key and sort key, distinct from the table's primary key.
Local Secondary Index (LSI): Similar to a GSI, an LSI is an index that allows you to query data in ways other than the table's primary key. However, LSIs share the same partition key as the table but have a different sort key. LSIs can only be created when designing the table and cannot be added later.

Understanding and appropriately leveraging these DynamoDB concepts is essential for designing efficient and scalable data models based on your application's specific needs.

Introducing DynamoDB Toolbox

Unlocking the true potential of DynamoDB's Single Table Design has never been easier, thanks to DynamoDB Toolbox. With the motto "Single Table Designs have never been this easy!" this powerful library streamlines the process of implementing efficient and scalable data models.

In our example, we seamlessly integrated DynamoDB Toolbox into a Fastify application, showcasing the synergy between a robust web framework and the simplicity of Single Table Design. This integration empowers developers to focus on building features rather than wrestling with database complexities.

Explore the capabilities of DynamoDB Toolbox as we demonstrate how it aligns with the philosophy of Single Table Design, making DynamoDB development a smoother and more enjoyable experience. Whether you're a seasoned developer or just starting with DynamoDB, this toolbox is your companion for simplifying and enhancing your Single Table Design journey.

Organizing Data

Scan all areas

Query a single area

Query a single area with newtworks

Query all networks by area

Query all network devices (using GSI)

Fastify App

We kick off by creating a simple Fastify application in TypeScript.

src/app.ts

...imports

export default function createApp(
  opts?: FastifyServerOptions,
): FastifyInstance {
  const defaultOptions = {
    logger: true,
  }

  const app = fastify({ ...defaultOptions, ...opts })

  app.register(swagger, {
    ...
  })

  app.register(swaggerUI)

  app.register(cors)

  app.register(autoload, {
    dir: join(__dirname, 'core'),
  })

  app.register(autoload, {
    dir: join(__dirname, 'services'),
  })

  app.register(autoload, {
    dir: join(__dirname, 'routes'),
    options: { prefix: '/api' },
    routeParams: true
  })

  return app
}

DynamoDB Client Plugin

With the DynamoDB Client Plugin, you can effortlessly map and expose various entities within your Fastify application, following the principles of Single Table Design. This streamlined integration empowers developers to focus on building robust features, abstracting away the complexities of DynamoDB interactions.

src/core/dynamodb-client-plugin.ts

...imports...

const marshallOptions = {
  convertEmptyValues: false,
  removeUndefinedValues: false,
  convertClassInstanceToMap: false,
}

const unmarshallOptions = {
  wrapNumbers: false,
}

const translateConfig = { marshallOptions, unmarshallOptions }

const DocumentClient = DynamoDBDocumentClient.from(new DynamoDBClient({}), translateConfig)

const SingleTableDemo = new Table({
  name: 'single-table-demo',
  partitionKey: 'pk',
  sortKey: 'sk',
  indexes: {
    gsi1: { partitionKey: 'gsi1pk', sortKey: 'gsi1sk' }
  },
  DocumentClient
})

const AreaEntity = new Entity({
  name: 'AREA',

  attributes: {
    id: {
      partitionKey: true,
      default: (data: { areaId: string, _et: string }) => `${data._et}#${data.areaId}`
    },
    sk: {
      sortKey: true,
      default: (data: { areaId: string, _et: string }) => `${data._et}#${data.areaId}`
    },
    name: { type: 'string', required: true, map: 'areaName' },
    areaId: { type: 'string', required: true },
    manager: { type: 'string', required: true },
    location: { type: 'string', required: true },
  },

  table: SingleTableDemo
} as const)

const NetworkEntity = new Entity({
  name: 'NETWORK',

  attributes: {
    id: {
      partitionKey: true ,
      default: (data: { areaId: string }) => `AREA#${data.areaId}`
    },
    sk: {
      sortKey: true,
      default: (data: { _et: string, areaId: string, networkId: string }) => `AREA#${data.areaId}#${data._et}#${data.networkId}`
    },
    areaId: { type: 'string', required: true },
    networkId: { type: 'string', required: true },
    networkType: { type: 'string', required: true },
    connectionSpeed: { type: 'string', required: true },
  },

  table: SingleTableDemo
} as const)

const DeviceEntity = new Entity({
  name: 'DEVICE',

  attributes: {
    id: { partitionKey: true },
    sk: {
      sortKey: true,
      default: (data: { id: string }) => `${data.id}`
    },
    gsi1pk: { type: 'string', required: true },
    gsi1sk: { type: 'string', required: true },
    type: { type: 'string', required: true },
    deviceId: { type: 'string', required: true },
    deviceName: { type: 'string', required: true },
    deviceType: { type: 'string', required: true },
    IPAddress: { type: 'string', required: true },
  },

  table: SingleTableDemo
} as const)

async function dynamoDbClientPlugin(app: FastifyInstance): Promise<void> {
  app.decorate('dynamoDbClient', {
    Area: AreaEntity,
    Network: NetworkEntity,
    Device: DeviceEntity,
  })
}

export default fp(dynamoDbClientPlugin)

AreaService Plugin

The AreaService Plugin acts as a dedicated API gateway, exposing seamless interfaces to interact with the "Area" entity. Integrated into our Fastify application, this plugin simplifies the management of areas within the Single Table Design DynamoDB structure. Developers can effortlessly perform CRUD operations on areas, abstracting away the intricacies of database interactions and ensuring a cohesive and efficient development experience.

src/services/area.service.ts

...imports...

async function areaServicePlugin(app: FastifyInstance): Promise<void> {

  function mapItemToArea(item: Record<string, string>): Area {
    return {
      id: item.areaId,
      name: item.name,
      manager: item.manager,
      location: item.location,
    }
  }

  async function getAll(): Promise<AreaCollection> {
    const result = await app.dynamoDbClient.Area.scan({
      filters: [
        {
          attr: '_et',
          eq: 'AREA'
        },
      ],
    })

    return result?.Items?.map(mapItemToArea) || []
  }

  async function get(id: string): Promise<Area | null> {
    const pk = `AREA#${id}`
    const { Item} = await app.dynamoDbClient.Area.get({
      id: pk,
      sk: pk,
    })

    if (!Item) {
      return null
    }

    return mapItemToArea(Item)
  }

  async function put(area: Area): Promise<Area> {
    const result = await app.dynamoDbClient.Area.put({
      areaId: area.id,
      name: area.name,
      manager: area.manager,
      location: area.location,
    })

    if (result.$metadata.httpStatusCode !== 200) {
      throw new Error('Error creating area')
    }

    return area
  }

  app.decorate('areaService', {
    getAll,
    get,
    put,
  })
}

export default fp(areaServicePlugin)

Utilizing AreaService in the "GET Areas" Route

Let's dive into the practical implementation of the AreaService within the "GET Areas" route. Leveraging the Fastify application and the DynamoDB Client Plugin, our AreaService seamlessly fetches and exposes areas through a straightforward API call.

Developers can now witness the simplicity of retrieving area entities without the need for complex database queries. This integration showcases how the AreaService abstracts away the underlying DynamoDB interactions, allowing for a clean and concise implementation of the "GET Areas" route within our Fastify application. Explore the elegance of Single Table Design in action as we demonstrate the power and efficiency of the AreaService plugin.

src/routes/area/list.route.ts

import { FastifyPluginAsyncTypebox } from '@fastify/type-provider-typebox'
import { AreaCollection } from '@models/area.model'

const route: FastifyPluginAsyncTypebox = async function (app) {
  app.get<{ Reply: AreaCollection }>(
    '/',
    {
      schema: {
        tags: ['Area'],
        response: {
          200: AreaCollection,
          500: {
            type: 'null',
            description: 'Internal server error',
          },
        },
      },
    },
    async (req, reply) => {
      const areas = await app.areaService.getAll()
      reply.code(200).send(areas)
    },
  )
}

export default route

Request:

curl --location '<server-url>/api/area'

Response:

[
    {
        "id": "1000",
        "name": "WLF",
        "manager": "Clifford Smith",
        "location": "Townebury"
    },
    {
        "id": "2000",
        "name": "CRI",
        "manager": "Vera Kozey",
        "location": "North Abdiel"
    },
    {
        "id": "3000",
        "name": "DOM",
        "manager": "Philip Thompson",
        "location": "Camden"
    }
]

Conclusions

In adopting the Single Table Design approach, we unlock DynamoDB's full potential without compromising on data modeling. This streamlined methodology, exemplified through the integration of DynamoDB Toolbox and the AreaService plugin, ensures efficient data organization within our Fastify application.

By strategically structuring entities and relationships, Single Table Design proves instrumental in maximizing DynamoDB's scalability and performance. It simplifies development, offering a clear and intuitive path to fully leverage DynamoDB's capabilities. In essence, this approach signifies a harmonious blend of powerful database functionality and effective data modeling, transforming the development landscape.

The project code is in this GitHub repository: fastify-dynamodb-single-table.

Fastify Meets WireMock: External Service Mocking

Massimo Biagioli — Tue, 13 Feb 2024 16:55:20 +0000

WireMock Wonders: Boosting Fastify Testability

This article reveals how to integrate WireMock into Fastify with ease, enabling developers to effortlessly generate mock responses for external services. Join us as we explore the straightforward process of seamlessly integrating and optimizing Fastify applications using WireMock for enhanced testing capabilities.

A Use Case: Integration with an External Service

In this scenario, we'll illustrate the integration with an external service. The service in focus simulates calls to a generic "devices" management system. Our objective is to retrieve the list of devices, read a device by its ID, and create a new one. Let's delve into the practical application of WireMock within Fastify for this specific use case.

Fastify App

We kick off by creating a simple Fastify application in TypeScript, utilizing the fastify-cli.

src/app.ts

export interface AppOptions extends FastifyServerOptions, Partial<AutoloadPluginOptions> {
}

const options: AppOptions = {
}

const FastifyEnvOpts = {
  dotenv: true,
  schema: {
    type: 'object',
    required: ['DEVICE_SERVER_URL'],
    properties: {
      DEVICE_SERVER_URL: {
        type: 'string'
      }
    }
  }
}

const app: FastifyPluginAsync<AppOptions> = async (
  fastify,
  opts
): Promise<void> => {
  void fastify.register(FastifyEnv, FastifyEnvOpts)

  void fastify.register(AutoLoad, {
    dir: join(__dirname, 'plugins'),
    options: opts
  })

  void fastify.register(AutoLoad, {
    dir: join(__dirname, 'routes'),
    options: opts
  })
}

export default app
export { app, options }

Building a Service to Interact with the External System

Now, let's craft a service that interacts with the external system. In this section, we'll guide you through the process of creating a Fastify service that communicates with the simulated external 'devices' management system. This hands-on approach will showcase the seamless integration of our Fastify application with WireMock for effective testing and interaction with external services.

src/plugins/deviceService.ts

// ... imports

export default fp(async (fastify, opts) => {
  const SERVER_BASE_URL = `${fastify.config.DEVICE_SERVER_URL}/api/v1/device`

  const deviceService = {
    getDevices: async (): Promise<Device[]> => {
      const response = await fetch(SERVER_BASE_URL)
      return await response.json() as Device[]
    },
    getDeviceById: async (id: string): Promise<Device | null> => {
      const response = await fetch(`${SERVER_BASE_URL}/${id}`)
      if (response.status === 404) {
        return null
      }
      return await response.json() as Device
    },
    createDevice: async (device: DeviceRequest): Promise<Device> => {
      const response = await fetch(SERVER_BASE_URL, {
        method: 'POST',
        body: JSON.stringify(device),
        headers: {
          'Content-Type': 'application/json'
        }
      })
      return await (await response.json() as Promise<Device>)
    }
  }

  fastify.decorate('deviceService', deviceService)
})

src/model/device.ts

export interface DeviceRequest {
  name: string
  type: string
  address: string
}

export type Device = DeviceRequest & {
  id: string
}

Using the deviceService in the Controller

src/routes/device/index.ts

// ...imports

const route: FastifyPluginAsync = async (fastify, opts): Promise<void> => {
  fastify.get<{ Reply: Device[] }>('/', async function (request, reply) {
    return await fastify.deviceService.getDevices()
  })

  fastify.get<{ Params: GetDeviceByIdParams, Reply: Device | GetDeviceByIdError }>('/:id', async function (request, reply) {
    const result = await fastify.deviceService.getDeviceById(request.params.id)
    if (result === null) {
      return await reply.notFound()
    }
    return result
  })

  fastify.post<{ Body: DeviceRequest, Reply: Device }>('/', async function (request, reply) {
    const result = await fastify.deviceService.createDevice(request.body)
    return reply.code(201).send(result)
  })
}

export default route

Setting Up WireMock

Let's dive into configuring WireMock for our project. We'll employ Docker to define the WireMock service, simplifying the setup process. By using containers, we ensure a convenient and reproducible environment for running WireMock alongside our Fastify application.

docker-compose.yml

version: '3.9'
services:
  wiremock:
    image: wiremock/wiremock:3.3.1
    restart: always
    ports:
      - '8080:8080'
    volumes:
      - ./wiremock_data:/home/wiremock
    entrypoint: ["/docker-entrypoint.sh", "--global-response-templating", "--verbose"]

wiremock_data/mappings/get_devices.json

{
  "request" : {
    "url" : "/api/v1/device",
    "method" : "GET"
  },
  "response" : {
    "status" : 200,
    "bodyFileName" : "get_devices_response.json",
    "headers" : {
      "Content-Type" : "application/json"
    }
  }
}

wiremock_data/__files/get_devices_response.json

[
  {
    "id": 1,
    "name": "First Device",
    "family_id": 1,
    "address": "10.0.1.1"
  },
  ...
]

wiremock_data/mappings/get_device_by_id.json

{
  "request" : {
    "urlPattern": "^/api/v1/device/\\d*",
    "method" : "GET"
  },
  "response" : {
    "status" : 200,
    "bodyFileName" : "get_device_by_id_response.json",
    "headers" : {
      "Content-Type" : "application/json"
    }
  }
}

wiremock_data/__files/get_device_by_id_response.json

This template utilizes dynamic templating, generating a response tailored to the input parameters. In this instance, the 'id' parameter influences the response, showcasing the flexibility and adaptability achieved through WireMock's templating capabilities.

{
  "id": {{ request.pathSegments.[3] }},
  "name": "Device {{ request.pathSegments.[3] }}",
  "family_id": 1,
  "address": "10.0.1.{{ request.pathSegments.[3] }}"
}

wiremock_data/mappings/get_device_by_id_404.json

{
  "request" : {
    "urlPattern": "^/api/v1/device/\\d*404$",
    "method" : "GET"
  },
  "response" : {
    "status" : 404,
    "body" : ""
  }
}

The setup is designed to manage scenarios where the 'id' ends with '404'. When a request matches the defined URL pattern and method (GET), WireMock responds with a 404 status code, providing a straightforward mechanism to simulate and test error conditions for our Fastify application.

wiremock_data/mappings/create_device.json

{
  "request" : {
    "url" : "/api/v1/device",
    "method" : "POST",
    "headers" : {
      "Content-Type": {
        "equalTo": "application/json"
      }
    },
    "bodyPatterns": [
      {
        "matchesJsonPath": "$.name"
      },
      {
        "matchesJsonPath": "$.family_id"
      },
      {
        "matchesJsonPath": "$.address"
      }
    ]
  },
  "response" : {
    "status" : 201,
    "bodyFileName" : "create_device_response.json",
    "headers" : {
      "Content-Type" : "application/json"
    }
  }
}

wiremock_data/__files/create_device_response.json

{
  "id": {{randomValue type='NUMERIC' length=5}},
  "name": "{{jsonPath request.body '$.name'}}",
  "family_id": {{jsonPath request.body '$.family_id'}},
  "address": "{{jsonPath request.body '$.address'}}"
}

We leverage dynamic templating to generate varied responses. The 'id' field is populated with a random numeric value of length 5. The 'name', 'family_id', and 'address' fields are populated based on the corresponding values present in the incoming request body. This dynamic approach allows us to simulate diverse scenarios and handle input data flexibly within our WireMock setup.

Testing

During the testing phase, we don't need to perform service-level mocks since we are directing our requests to the WireMock URL. By utilizing WireMock as the target URL, we seamlessly integrate our Fastify application with the WireMock service, allowing for realistic simulations and comprehensive testing scenarios without the need for extensive service-level mocking.

test/routes/device.test.ts

...

test('get device by id', async (t) => {
    const app = await build(t)

    const res = await app.inject({
        url: '/device/123'
    })

    const payload = JSON.parse(res.payload)
    assert.equal(res.statusCode, 200)
    assert.deepStrictEqual(payload, { id: 123, name: 'Device 123', family_id: 1, address: '10.0.1.123' })
})

test('get device by id - 404', async (t) => {
    const app = await build(t)

    const res = await app.inject({
        url: '/device/1404'
    })

    const payload = JSON.parse(res.payload)
    assert.equal(res.statusCode, 404)
    assert.deepStrictEqual(payload, { "statusCode": 404, "error": "Not Found", "message": "Not Found" })
})

...

Conclusions

WireMock provides a comprehensive solution for building a test suite towards external services, eliminating the need for patching services at the application level. This approach enables us to conduct real tests, fostering a robust testing environment and ensuring the reliability and effectiveness of our Fastify application in interacting with external services.
The project code is in this GitHub repository: fastify-wiremock-example.

Testing Fastify with Node:Test

Massimo Biagioli — Fri, 05 Jan 2024 10:18:11 +0000

Mastering Fastify Application Testing with node:test in TypeScript

In this article, we'll explore native testing in Node.js, specifically utilizing the built-in node:test library for TypeScript-based Fastify applications. Drawing from my own experience, where I previously relied on third-party libraries like Jest, we'll delve into the simplicity and effectiveness of incorporating node:test directly into your testing toolkit. Join me in discovering a seamless testing approach tailored for TypeScript and Fastify development.

Let's start with a standard Fastify + Typescript App

src/app.ts



// ... imports ...

async function buildApp (options: Partial<typeof defaultOptions> = {}) {
  const app: FastifyInstance = Fastify({ ...defaultOptions, ...options })
    .withTypeProvider<TypeBoxTypeProvider>()

  app.register(autoload, {
    dir: join(__dirname, 'plugins')
  })

  app.register(autoload, {
    dir: join(__dirname, 'routes'),
    options: { prefix: '/api' }
  })

  return app
}

export default buildApp

This is a simple route:

src/routes/getDevices.ts



// ... imports ...

export default async function (
  fastify: FastifyInstance,
  _opts: FastifyPluginOptions
): Promise<void> {
  fastify.get<{ Reply: GetDevicesResponseType }>(
    '/device',
    async (_request, _reply) => {
      return await fastify.fetchDevices()
    }
  )
}

This is the use case as a Fastify plugin:

src/plugins/fetchDevices.ts



// ... imports ...

async function fetchDevicesPlugin (
  fastify: FastifyInstance,
  _opts: FastifyPluginOptions
): Promise<void> {
  fastify.decorate('fetchDevices', ExternalDevice.fetchDevices)
}

export default fp(fetchDevicesPlugin)

This is a point where we simulate the reading of devices through an external system:

src/external/device.ts



// ... imports ...

const fetchDevices = async (): Promise<DeviceCollectionType> => {
  return [
    {
      id: '1',
      name: 'Device 1',
      address: '10.0.0.1'
    },
    {
      id: '2',
      name: 'Device 2',
      address: '10.0.0.2'
    },
    { .... }
  ]
}

export default {
  fetchDevices
}

Now, we want to write a test using the 'node:test' library, mocking the external dependency.

test/routes/getDevices.test.ts



import buildApp from '@src/app'
import { FastifyInstance } from 'fastify'
import { describe, it, after, mock } from 'node:test'
import assert from 'assert'
import ExternalDevice from '@src/external/device'

describe('GET /device HTTP', () => {
  let app: FastifyInstance

  const mockDevices = [
    {
      id: 'test',
      name: 'Device Test',
      address: '10.0.2.12'
    }
  ]

  const mockDevicesFn = mock.fn(async () => mockDevices)

  const mockDevicesErrorFn = mock.fn(async () => {
    throw new Error('Error retrieving devices')
  })

  after(async () => {
    await app.close()
  })

  it('GET /device returns status 200', async () => {
    mock.method(ExternalDevice, 'fetchDevices', mockDevicesFn)

    app = await buildApp({ logger: false })

    const response = await app.inject({
      method: 'GET',
      url: '/api/device'
    })

    assert.strictEqual(response.statusCode, 200)
    assert.deepStrictEqual(JSON.parse(response.payload), mockDevices)
    assert.strictEqual(mockDevicesFn.call.length, 1)

    const call = mockDevicesFn.mock.calls[0]
    assert.deepEqual(call.arguments, [])

    mock.reset()
  })

})

Now, we also want to test the scenario where reading from the external system fails, still using a mock.



// ...
  it('GET /device returns status 500', async () => {
    mock.method(ExternalDevice, 'fetchDevices', mockDevicesErrorFn)

    app = await buildApp({ logger: false })

    const response = await app.inject({
      method: 'GET',
      url: '/api/device'
    })

    const expectedResult = {
      statusCode: 500,
      error: 'Internal Server Error',
      message: 'Error retrieving devices'
    }

    assert.strictEqual(response.statusCode, 500)
    assert.deepStrictEqual(JSON.parse(response.payload), expectedResult)
    assert.strictEqual(mockDevicesFn.call.length, 1)

    const call = mockDevicesFn.mock.calls[0]
    assert.deepEqual(call.arguments, [])

    mock.reset()
  })
// ...

How to set up the project the right way

We are using ts-node to utilize TypeScript.

package.json



{
    "scripts": {
        ...
        "test": "node -r ts-node/register --test test/**/*.test.ts",
        "test:watch": "node -r ts-node/register --test --watch test/**/*.test.ts",
        "test:coverage": "nyc pnpm run test",
        ...
    },
...
}

tsconfig.json



{
  "ts-node": {
    "require": ["tsconfig-paths/register"],
    "files": true
  },
  "extends": "fastify-tsconfig",
  "compilerOptions": {
    "target": "ES2022",                                  
    "lib": ["ES2022"],
    "baseUrl": "./",  
    "paths": {
      "@src/*": ["src/*"],
      "@test/*": ["test/*"],
    },                
    "outDir": "./dist"
  }
}

Launch Tests



pnpm run test

About coverage

The coverage function of node:test is still in an experimental phase. However, it is possible to achieve good results by using the nyc library in conjunction with node:test.



pnpm run test:coverage

Conclusions

The project code is in this GitHub repository: fastify-ts-nodetest.
The native node:test function serves as an excellent alternative to third-party libraries. While it is still evolving, it effectively fulfills its purpose. Developers can leverage its capabilities to replace external dependencies, benefitting from its evolving features and robust performance in testing scenarios.

CQRS with Low-Code

Massimo Biagioli — Mon, 27 Nov 2023 07:47:06 +0000

Bridging the Gap Between Two Worlds

In the realm of software design, marrying Command Query Responsibility Segregation (CQRS) with low-code might seem like an odd match. However, beneath the surface, these two approaches complement each other surprisingly well. This article explores the synergy between CQRS and low-code, demonstrating how their coexistence bridges the gap between traditional and modern software architectures. Join us as we unravel the simplicity and effectiveness of this powerful combination for building robust applications.

The Central Role of Commands in CQRS

In the realm of CQRS, a "command" represents an imperative instruction, typically responsible for altering the state of an application. Commands carry the essence of change, from updating data to triggering specific actions. When it comes to integrating CQRS with a low-code pipeline, the brilliance lies in the ability to delegate command execution. Low-code platforms excel at handling the intricacies of command processing, offering a streamlined and visual approach to define, execute, and manage these imperative actions. This synergy not only simplifies the implementation of commands but also empowers developers to harness the efficiency of low-code for rapid and visual command orchestration.

Command Handling Simplified: The Low-Code Connection

In the world of low-code, simplicity meets power as it takes on the role of a command handler through the orchestration of a dedicated pipeline. Low-code platforms serve as adept command executors, effortlessly managing the flow of instructions through a visual pipeline. This approach not only abstracts the complexities of command execution but also empowers developers with an intuitive and efficient means to design, deploy, and monitor the entire process. By embracing low-code as a command handler, developers can leverage its agility to streamline the execution of imperative actions, unlocking a new level of efficiency in the CQRS paradigm.

Scenario

As a user, the task at hand is to upload product data from a CSV file into an Airtable.

Flow

User Initiates Upload:
- The user triggers an upload action by making an API call to the backend, providing a CSV file containing product data.
Controller Dispatches Command:
- The controller prepares a 'ImportCsvCommand' command encapsulating the user's request.
- The command is dispatched into the command bus.
Command Handling:
- The command handler takes charge and performs the following operations:
  - Uploads the CSV file to an S3 bucket.
  - Invokes a low-code workflow through a webhook.
Low-Code Workflow:
- The low-code workflow follows a structured sequence:
  - Reads the CSV file from the S3 bucket.
  - Parses the CSV data.
  - Inserts the parsed data into the 'products' table in Airtable.

Demo Application

Our application is crafted with NestJS, following the principles of Domain-Driven Design (DDD) and the CQRS pattern. Using NestJs's power, we seamlessly combine these approaches for efficient handling of tasks. This mix ensures our application is not just modern but also well-organized, showcasing the synergy between user-friendly frameworks and smart design principles.

src/app.ts

export default async function bootstrap(): Promise<FastifyInstance> {
  const serverOptions = {
    logger: true,
  };
  const instance = fastify(serverOptions);

  const app = await NestFactory.create<NestFastifyApplication>(
    AppModule,
    new FastifyAdapter(instance),
  );

  app.useGlobalPipes(new ValidationPipe());
  app.enableCors();
  await app.register(FastifyMultipart);

  await app.init();

  return instance;
}

Product Controller

src/product/presentation/http/product.controller.ts

@Controller('product')
export class ProductController {
  constructor(private readonly commandBus: CommandBus) {}

  @Post('/importCsv')
  @HttpCode(200)
  async uploadFile(
    @Req() req: FastifyRequest,
    @Res() res: FastifyReply<never>,
  ): Promise<void> {
    if (!req.isMultipart()) {
      res.send(new BadRequestException());
      return;
    }

    const csvData = await req.file();

    let now = new Date();
    now = new Date(now.getTime() - now.getTimezoneOffset() * 60 * 1000);
    const storageKey = `${now.toISOString()}-${csvData.filename}`;

    await this.commandBus.execute(new ImportCsvCommand(csvData, storageKey));

    res.send();
  }
}

ImportCsv Command & Command Handler

src/product/application/command/import-csv-command.ts

export class ImportCsvCommand {
  constructor(
    public readonly data: MultipartFile,
    public readonly storageKey: string,
  ) {}
}

src/product/application/command-handler/import-csv-handler.ts

@CommandHandler(ImportCsvCommand)
export class ImportCsvHandler implements ICommandHandler<ImportCsvCommand> {
  constructor(
    @Inject(IStorageService) private readonly storageService: IStorageService,
    private readonly httpService: HttpService,
    private readonly configService: ConfigService,
  ) {}

  async execute({ data, storageKey }: ImportCsvCommand) {
    await this.storageService.uploadFile(data, storageKey);
    await this.callLowCodePipeline(storageKey);
  }

  private async callLowCodePipeline(storageKey: string) {
    // invoke webhook
  }
}

Low-Code Flow

For our low-code workflow, we harnessed the power of the Make platform. This user-friendly tool allows us to design our flow through an intuitive drag-and-drop interface. By seamlessly connecting pre-configured connectors, such as the S3 bucket and AirTable table, we effortlessly orchestrate the entire process. 'Make' empowers us to visually design and configure the workflow, ensuring a smooth integration between the S3 bucket and the AirTable table, all with a few clicks and without the need for extensive coding.

Webhook: the flow entry point

At the start of our flow is the webhook block, acting as the starting point for our process. This essential component kicks off the low-code workflow, setting the stage for the following actions.

src/product/application/command-handler/import-csv-handler.ts

@CommandHandler(ImportCsvCommand)
export class ImportCsvHandler implements ICommandHandler<ImportCsvCommand> {
  constructor(
    @Inject(IStorageService) private readonly storageService: IStorageService,
    private readonly httpService: HttpService,
    private readonly configService: ConfigService,
  ) {}

  async execute({ data, storageKey }: ImportCsvCommand) {
    await this.storageService.uploadFile(data, storageKey);
    await this.callLowCodePipeline(storageKey);
  }

  private async callLowCodePipeline(storageKey: string) {
    const result = this.httpService
      .post(this.configService.get<string>('FLOW_IMPORT_PRODUCTS_WEBHOOK'), {
        bucketName: this.configService.get<string>('PRODUCTS_BUCKET_NAME'),
        bucketKey: storageKey,
      })
      .pipe(tap((res) => console.log(res)))
      .pipe(
        catchError(() => {
          throw new ForbiddenException('API not available');
        }),
      );

    await lastValueFrom(result);
  }
}

curl --location 'http://localhost:3000/product/importCsv' \
--form 'csv=@"./sample-data/product.csv"'

sample-data/product.csv

1;First Product;10.50
2;Second Product;20.00
3;Third Product;30.00

After the execution:

From S3 Reading to CSV Parsing

Moving ahead, we shift from reading data in the S3 bucket to the CSV parsing phase. The low-code system smoothly collects the CSV file and then effortlessly processes it through parsing.

Final Step: insert into AirTable

As the final step, our low-code workflow seamlessly inserts the parsed data into the designated AirTable.

Conclusions

The project code is in this GitHub repository: cqrs-with-low-code.
Embracing a low-code approach for certain command handling in our application has proven advantageous, notably in expediting feedback loops. Delegating tasks to the low-code pipeline streamlines command execution, fostering a quicker response mechanism. This efficiency not only enhances our development process but also empowers us to adapt and iterate swiftly, ultimately contributing to a more responsive and agile system.

Serverless Layers with Monorepo

Massimo Biagioli — Fri, 17 Nov 2023 15:36:23 +0000

Introduction

In the world of cloud development, combining Serverless Framework with a Monorepo offers a smart way to handle and deploy multiple applications. This article breaks down the process of creating a CloudFormation stack using Serverless Framework. Each application, neatly packed within a Monorepo, gets its own Lambda function. Join us as we explore the easy integration of serverless tech and the practical advantages of using a Monorepo, uncovering the steps to set up a flexible and organized infrastructure.

Starting with Monolith

In the early stages of a project, it's often a good idea to begin with a monolith—a single, straightforward architecture. This simplicity is beneficial when the project is just getting started because we might not have a clear picture of the system's boundaries, and predicting how it will grow can be tricky.
By starting with a monolith, developers can handle the uncertainties of the initial phase more easily. As the project advances and we add more features, there might come a time when it makes sense to break off certain functions into smaller pieces called microservices. This step allows for a more organized and scalable system that can adapt as the project evolves.

Keyword: Modularity

Building on the foundation of a monolith, the key to success lies in a modular code organization, where "bounded contexts" are well-contained and independent.
In an ideal world, picture each module neatly residing in its own folder within our monolith. Transforming a module into a new application should be a seamless transition; if it requires excessive effort, it's a clear sign that something needs adjustment.
Embracing modularity ensures flexibility as the project evolves, promoting a codebase that is both adaptable and comprehensible.

Monorepo Overview

This monorepo is organized around the 'apps' folder, containing two Fastify applications:

Customers
Orders

Each application within the 'apps' folder follows a consistent structure:

src
- handlers
  - app.ts (Lambda entry point)
- routes
  - routeA.ts (Route definitions for specific functionalities)
  - routeB.ts (Additional route definitions)
app.ts (Application entry point)
server.ts (Server setup and configuration)

apps/customers/app.ts

import 'module-alias/register'
import fastify, {FastifyInstance, FastifyServerOptions} from 'fastify'
import autoload from '@fastify/autoload'
import {join} from 'path'

export default function createApp(
  opts?: FastifyServerOptions,
): FastifyInstance {
  const defaultOptions = {
    logger: true,
  }

  const app = fastify({ ...defaultOptions, ...opts })

  app.register(autoload, {
    dir: join(__dirname, 'routes'),
    options: { prefix: '/customers' },
  })

  return app
}

apps/customers/routes/getCustomers.ts

import { FastifyInstance } from 'fastify'

export default async function (app: FastifyInstance): Promise<void> {
  app.get(
    '/',
    async () => {
      return [
          {
              id: 1,
              name: "Johnny Walker"
          },
          {
              id: 2,
              name: "Jack White"
          }
      ]
    },
  )
}

apps/customers/handlers/app.ts

import awsLambdaFastify from '@fastify/aws-lambda'
import createApp from '../app'

const app = createApp()

const proxy = awsLambdaFastify(app)

export const handler = proxy

apps/customers/server.ts

import createApp from './app'

const app = createApp()

app.ready().then(() => {
  app.listen({ port: 4001 }, (err) => {
    if (err != null) {
      app.log.error(err)
      process.exit(1)
    }
  })
})

(Order app has the same structure).

Going Serverless

In our move to deploy the entire project on the cloud using Serverless Framework, we aim to avoid bundling all dependencies (as listed in package.json) for each Fastify application's lambda function. To achieve this, we'll create a shared layer that can be utilized by all applications in the monorepo. This layer approach streamlines deployment, ensuring efficient use of resources across multiple functions while simplifying maintenance.

Using the "serverless-layers" Plugin

Introducing the "serverless-layers" plugin (https://www.serverless.com/plugins/serverless-layers): a user-friendly tool that automates the creation of layers. By default, it utilizes the package.json found in the project's root, simplifying the layer setup process. This plugin streamlines the integration of shared dependencies across multiple functions, enhancing efficiency and reducing redundancy in your Serverless deployment.

serverless.yml

service: fastify-serverless-layer
frameworkVersion: '3'

provider:
  name: aws
  runtime: nodejs18.x
  region: eu-south-1
  deploymentBucket:
    name: fastify-serverless-layer-bucket
    serverSideEncryption: AES256

plugins:
  - serverless-plugin-typescript
  - serverless-tscpaths
  - serverless-layers
  - serverless-deployment-bucket
  - serverless-offline

custom:
  serverless-layers:
    functions: # optional
      - customers
      - orders
    dependenciesPath: ./package.json

functions:
  customers:
    handler: apps/customers/src/handlers/app.handler
    events:
      - http:
          path: /customers/
          method: ANY
      - http:
          path: /customers/{any+}
          method: ANY
  orders:
    handler: apps/orders/src/handlers/app.handler
    events:
      - http:
          path: /orders/
          method: ANY
      - http:
          path: /orders/{any+}
          method: ANY

In the provided serverless.yml, each Lambda function (e.g., 'customers' and 'orders') is configured with an entry point pointing to an application within the monorepo. For instance, the 'customers' function is linked to apps/customers/src/handlers/app.handler. These functions are subsequently associated with specific API Gateway routes, like /customers/ and /orders/. This setup streamlines the integration of Fastify applications with AWS Lambda, simplifying the routing configuration for the Serverless deployment.

Conclusions

The project code is in this GitHub repository: fastify-serverless-layer.
This journey guided us from building a CloudFormation stack based on a modular monolith to evolving into microservices, demonstrating the project's flexible design in practice.

Supercharge Your Fastify App with DynamoDB Pagination Plugin

Massimo Biagioli — Fri, 08 Sep 2023 15:06:51 +0000

Intro

Have you ever needed to read data and wondered how to paginate it effectively in DynamoDB? In this article, we'll tackle this challenge using a Fastify plugin that leverages the AWS SDK. Let's dive in!

One step back: how DynamoDB works

DynamoDB is like a giant electronic filing cabinet for your data. Imagine it as a collection of drawers (partitions), each holding folders (items) with pieces of information.

Partition Key: Think of the partition key as the label on each drawer. It helps DynamoDB decide where to put and find your data. Every item must have a unique partition key. For example, if you're storing information about people, the partition key might be their unique ID.

Sort Key: If you want to organize the folders within a drawer, you use a sort key. It's like arranging the folders by date or name. It helps you quickly find what you need inside a partition.

Query vs. Scan: Imagine you're looking for specific folders. A query is like searching for folders with a specific label in a single drawer. It's super quick because DynamoDB knows exactly where to look. On the other hand, a scan is like going through all the folders in a drawer, looking for what you need. It can be slower and less efficient because it checks everything.

In a nutshell, DynamoDB stores your data in drawers (partitions), uses labels (partition keys) to find them quickly, and can sort them using additional labels (sort keys). You can either search for specific items (query) or look through everything (scan) depending on your needs.

Example: DynamoDB Pagination with Fastify Plugin

In this example, we'll explore how to paginate data in DynamoDB using a Fastify plugin. We have a DynamoDB table called "test-pager" with the following structure:

userId: This is our partition key, a unique identifier for each user (string).
deviceId: The sort key, which helps us organize data for each user's device (string).

Each item in the table also has an attribute:

address: A string attribute that holds the device address.

Understanding the Table

To better grasp this setup, think of "test-pager" as a collection of user devices. The partition key, "userId," separates users, and the sort key, "deviceId," allows us to associate multiple devices with a user. For instance:

Item 1
- userId: "u01"
- deviceId: "d01"
- address: "10.0.1.1"
Item 2
- userId: "u01"
- deviceId: "d02"
- address: "10.0.2.1"

Limit and "Offset"

Pagination in DynamoDB using the AWS SDK for a Query operation is like flipping through a book. When you use 'Limit,' you're telling DynamoDB how many results you want to see on each page, like asking for 10 pages at a time.

'ExclusiveStartKey' is like a bookmark that remembers where you stopped reading. It helps you start from the right place on the next page. It's different from an offset because you don't count items; you rely on DynamoDB to remember.

'Limit' controls how many items per page, and 'ExclusiveStartKey' remembers where to start on the next page. This way, you can efficiently paginate through large datasets without the need to manually count or skip items.

Fastify App

In our Fastify app, we've set up an API endpoint at '/api/devices' for reading user devices. Users can provide a 'userId,' optionally set a 'limit' (defaulting to 10), and include an optional 'startKey' to resume pagination from where they left off.

route/list.ts

export default async function (
    fastify: FastifyInstance,
    _opts: FastifyPluginOptions,
): Promise<void> {
    fastify.get<{ Querystring: DeviceRequestDtoType, Reply: DynamoDBPaginatedResult<DeviceDtoType> }>(
        '/',
        {
            schema: {
                querystring: DeviceRequestDto
            }
        },
        async (request, reply) => {
            try {
                const { userId, limit, startKey } = request.query
                const result = await fastify.listDevices(userId, limit, startKey)
                return reply.send(result)
            } catch (error) {
                request.log.error(error)
                return reply.code(500).send()
            }
        },
    )
}

The feature 'listDevices', registered as a plugin, is invoked within the route:

features/devices.list.ts

async function listDevicesPlugin(
    fastify: FastifyInstance,
    _opts: FastifyPluginOptions,
): Promise<void> {
    const listDevices = async (
        userId: string,
        limit: number,
        startKey?: string
    ): Promise<DynamoDBPaginatedResult<DeviceDtoType>> => {
        const queryCommand = new QueryCommand({
            TableName: 'test-pager',
            KeyConditionExpression: 'userId = :userId',
            ExpressionAttributeValues: { ':userId': { S: userId } },
            Limit: limit,
            ExclusiveStartKey: startKey ? fastify.DynamoDBKeyUtil.decode(startKey) : undefined,
        })

        const deviceTransformer = (item: Record<string, AttributeValue>): DeviceDtoType => ({
            userId: item.userId.S as string,
            deviceId: item.deviceId.S as string,
            address: item.address.S as string,
        })

        const pager = fastify.DynamoDBPager<DeviceDtoType>(queryCommand, deviceTransformer);
        return await pager()
    }

    fastify.decorate('listDevices', listDevices)
}

export default fp(listDevicesPlugin)

plugins/dynamodb.pager.ts

async function dynamoDBPagerPlugin(
    fastify: FastifyInstance,
    _opts: FastifyPluginOptions,
): Promise<void> {
    const DynamoDBPager = <T>(
        queryCommand: QueryCommand,
        itemTransformer: (item: Record<string, AttributeValue>) => T
    ) => async (): Promise<DynamoDBPaginatedResult<T>> => {
        const queryResult = await fastify.dynamoDB.send(queryCommand)
        const items = queryResult.Items?.map(itemTransformer) ?? []

        const paginatedResult: DynamoDBPaginatedResult<T> = {
            items
        }

        if (queryResult?.LastEvaluatedKey) {
            paginatedResult.lastKey = fastify.DynamoDBKeyUtil.encode(queryResult.LastEvaluatedKey)
        }

        return paginatedResult
    }

    fastify.decorate('DynamoDBPager', DynamoDBPager)
}

export default fp(dynamoDBPagerPlugin)

Encoding and Decoding startKey and lastKey

When working with DynamoDB, you'll notice that unlike the numeric 'offset' parameter used in other databases like MySQL, DynamoDB uses 'LastEvaluatedKey' and 'ExclusiveStartKey' as objects to manage pagination. To work with these keys, you'll need encoding and decoding methods (another Fastify Plugin), which we'll illustrate with some code examples shortly:

plugins/dynamodb.keyutil.ts

async function dynamoDBkeyUtilPlugin(
    fastify: FastifyInstance,
    _opts: FastifyPluginOptions,
): Promise<void> {
    const DynamoDBKeyUtil = {
        encode: (key: Record<string, AttributeValue>): string => btoa(JSON.stringify(key)),
        decode: (key: string): Record<string, AttributeValue> => JSON.parse(atob(key))
    }
    fastify.decorate('DynamoDBKeyUtil', DynamoDBKeyUtil)
}

export default fp(dynamoDBkeyUtilPlugin)

Conclusions

DynamoDB, Fastify's plugin ecosystem, and TypeScript are powerful tools in your development arsenal. By understanding DynamoDB's nuances, leveraging Fastify's modularity, and embracing TypeScript's benefits, you can craft applications that are not only efficient but also robust and easily maintainable for the long haul.
Three interesting points:

Understanding DynamoDB: To make the most of DynamoDB, it's essential to grasp its unique characteristics. By working with its distributed data model and efficient indexing, you can harness its full potential for data storage and retrieval.
Fastify's Modular Ecosystem: Fastify's plugin-rich environment is a game-changer for building applications. With modular components readily available, development becomes more agile and efficient, allowing you to assemble your app like building blocks.
The Power of TypeScript: TypeScript plays a vital role in creating robust and maintainable applications. Its static typing helps catch errors early in the development process, making your codebase more reliable and easier to maintain over time.

Exploring Fastify and TypeScript: Mocking External Dependencies

Massimo Biagioli — Fri, 25 Aug 2023 14:15:18 +0000

Intro

In this article, I'll walk you through the process of setting up a Fastify + TypeScript project and demonstrate how to effectively mock external dependencies. I've created a demo application to showcase this concept.

Project Setup

Create new Fastify Project

mkdir fastify-ts-mock
cd fastify-ts-mock

npm init -y

Install Typescript

npm i -D typescript @types/node ts-node

tsc --init

Install Fastify

npm i fastify fastify-plugin @fastify/autoload

Folder structure

(root) fastify-ts-mock
    |-- src
         |-- lib
         |-- plugins
         |-- routes
         |-- type 
    |-- test

Path alias

npm i module-alias
npm i -D @types/module-alias tsconfig-paths

Add the following lines to tsconfig.json

{
  "ts-node": {
    "require": ["tsconfig-paths/register"]
  },
  ...
  "baseUrl": "./",
  "paths": {
      "@src/*": ["src/*"],
      "@type/*": ["src/type/*"],
      "@lib/*": ["src/lib/*"],
      "@test/*": ["test/*"],
   },     
   ...  
}

Fastify App

File: src/app.ts

import 'module-alias/register';
import fastify, { FastifyInstance, FastifyServerOptions } from 'fastify'
import autoload from '@fastify/autoload'
import { join } from 'path'

export default function createApp(
  opts?: FastifyServerOptions,
): FastifyInstance {
  const defaultOptions = {
    logger: true,
  }

  const app = fastify({ ...defaultOptions, ...opts })

  app.register(autoload, {
    dir: join(__dirname, 'plugins'),
  })

  app.register(autoload, {
    dir: join(__dirname, 'routes'),
    options: { prefix: '/api' },
  })

  return app
}

List all devices Route (`/api/devices`)

File: src/routes/devices/list.ts

import { FastifyInstance, FastifyPluginOptions } from 'fastify'
import {DeviceDtoCollectionType} from "@type/devices.type";

export default async function (
  fastify: FastifyInstance,
  _opts: FastifyPluginOptions,
): Promise<void> {
  fastify.get<{ Reply: DeviceDtoCollectionType }>(
    '/',
    async (request, reply) => {
      try {
        const devices = await fastify.listDevices()
        return reply.send(devices)
      } catch (error) {
        request.log.error(error)
        return reply.code(500).send()
      }
    },
  )
}

List all devices Feature

File: src/plugins/features/devices.list.feature.ts

import fp from 'fastify-plugin'
import { FastifyInstance, FastifyPluginOptions } from 'fastify'
import * as DeviceLib from '@lib/devices.lib'
import {DeviceDtoCollectionType} from "@type/devices.type";

declare module 'fastify' {
  interface FastifyInstance {
    listDevices: () => Promise<DeviceDtoCollectionType>
  }
}

async function listDevicesPlugin(
  fastify: FastifyInstance,
  _opts: FastifyPluginOptions,
): Promise<void> {
  const listDevices = async (): Promise<DeviceDtoCollectionType> => DeviceLib.listDevices()

  fastify.decorate('listDevices', listDevices)
}

export default fp(listDevicesPlugin)

This plugin use DeviceLib.listDevices, which is an external dependency.
In order to properly test the route, I need to mock it.

Install test libraries

npm i tap
npm i -D @types/tap

Test code (`test/routes/devices/list.test.ts`)

import {afterEach, test} from "tap"
import createApp from "@src/app";
import * as DeviceLib from '@lib/devices.lib'
import * as Fixtures from '@test/fixtures'
import {DeviceDtoCollectionType} from "@type/devices.type";

test('get all devices', async t => {
  const app = createApp({
    logger: false,
  })

  t.teardown(() => {
    app.close();
  })

  const response = await app.inject({
    method: 'GET',
    url: '/api/devices',
  })

  const deviceCollection = response.json<DeviceDtoCollectionType>()

  t.equal(response.statusCode, 200)
  t.equal(deviceCollection.length, 2)

  ... other test assertion ...
})

First approach: decorate the plugin with a factory

async function listDevicesPlugin(
  fastify: FastifyInstance,
  _opts: FastifyPluginOptions,
): Promise<void> {

  const listDevices = async (): Promise<DeviceDtoCollectionType> => {
    return await DeviceLib.listDevices()
  }

  const listDevicesTest = async (): Promise<DeviceDtoCollectionType> => {
    return [
      ... some fake data ...
    ]
  }

  function listDevicesFactory() {
    if (fastify.config.ENV === 'test') {
      return listDevicesTest()
    }
    return listDevices()
  }

  fastify.decorate('listDevices', listDevices)
}

PROS:

easy approach
no third-party libraries are needed

CONS:

more plugin complexity = more mock function complexity
code to maintain
code that is not bug-free

A better approach: using ImportMock

Install dependencies:

npm i -D sinon ts-mock-imports

Update test code (test/routes/devices/list.test.ts):

import {afterEach, test} from "tap"
import createApp from "@src/app";
import {ImportMock} from "ts-mock-imports";
import * as DeviceLib from '@lib/devices.lib'
import * as Fixtures from '@test/fixtures'
import {DeviceDtoCollectionType} from "@type/devices.type";

afterEach(() => {
  ImportMock.restore();
})

test('get all devices', async t => {
  const app = createApp({
    logger: false,
  })

  t.teardown(() => {
    app.close();
  })

  const listDevicesMock = ImportMock.mockFunction(
      DeviceLib,
      'listDevices',
      Fixtures.devices
  )

  const response = await app.inject({
    method: 'GET',
    url: '/api/devices',
  })

  const deviceCollection = response.json<DeviceDtoCollectionType>()

  t.equal(response.statusCode, 200)
  t.equal(deviceCollection.length, 2)

  ... some data assertions ...

  t.ok(listDevicesMock.calledOnce)
})

PROS:

very simple
easy to maintain

CONS:

third-party libraries are needed

We are not mocking a fastify plugin, but a library used inside the plugin.

What if we try to mock the plugin instead?

const listDevicesMock = ImportMock.mockFunction(
    app,
    'listDevices',
    Fixtures.devices
)

It doesn't work: we get this error:

Cannot stub non-existent property listDevices

Conclusion

In a nutshell, writing tests the right way isn't just important – it's crucial. Proper tests ensure our code works as intended and stays reliable. So, let's remember, good tests mean great software!

Feel free to check out the repository for a hands-on experience and deeper insights into the process. Happy coding!