DEV Community: Enri Peters

Building a Cost-Effective, Serverless Virus Scanner on AWS: A Step-by-Step Guide

Enri Peters — Fri, 17 Jan 2025 08:48:53 +0000

Introduction

In this blog post, you will learn how to create a Serverless virus scanner on AWS in a step-by-step guide.

Virus scanners are important for detecting and removing malicious software (viruses, malware, etc.). They work by scanning for patterns of code that match known malware signatures or suspicious behaviours. They can be used in safeguarding our digital environments from potential security threats.

Without a reliable virus scanner in place, our our digital environments are left vulnerable to attacks, which could result in serious consequences. Therefore, it's essential to have a trusted virus scanner to protect our systems and networks.

Why a serverless virus scanner?

A serverless virus scanner offers several benefits over traditional server-based solutions. Firstly, it requires less maintenance, which reduces costs significantly 🤑. Secondly, a serverless virus scanner is highly scalable, this makes it easy to increase or decrease the number of scans you run as needed. This makes it a cost-effective solution for protecting your digital environment from potential security threats.

Choosing the Right Services

Choosing the right services and tools is important and proper selection of services can impact cost, performance, and scalability in a positive way. We will be using the following AWS services to build our serverless virus scanner.

S3: a reliable, scalable, and cheap object storage service that can store large amounts of data.
EventBridge Scheduler: a service that enables us to schedule events based on time intervals or cron expressions, which we will be using to automate the virus definitions updating process.
Lambda: the perfect solution for executing our ClamAV virus scanner.

By using Lambda, we don't have to worry about provisioning or managing servers, and it automatically scales. Our Lambda function will be called Clambda, and it will execute the ClamAV virus scanner. We will be writing the code for Clambda in Python, making use of the libraries available for Python.

Designing the Virus Scanner Architecture

To run a serverless virus scanner we need some serverless services from AWS. As mentioned in the previous paragraph we will be using S3, EventBridge Scheduler and Lambda. In the above diagram you can see how these different component work together.

Setting up part 1 - Update virus definitions

It's essential to keep your virus scanner up to date by updating its definitions daily. ClamAV offers two ways to update its virus definitions: freshclam and cdvupdate. Freshclam can be run as a daemon or through a cronjob using the freshclam.conf config file. However, it may cause blacklisting issues with the CDN, making it difficult to develop. A better solution is cdvupdate, a tool that allows you to download and update ClamAV databases and database patch files to host your own database mirror.

Developing the cvdupdate Lambda

We opted to update virus definitions daily, but you can adjust the frequency as needed. To run the cvdupdate Lambda, we’ve set up a daily Eventbridge schedule, which executes the cvdupdate Lambda and downloads/uploads the latest virus definitions to the ClamAV virus definitions bucket.

handlers/cvdupdate/requirements.txt:

cvdupdate

handlers/cvdupdate/virus_definitions_updater.py:

"""cvdupdate Lambda"""
# Standard imports
import os
import logging

# Non standard imports
import boto3

s3 = boto3.client("s3")
logger = logging.getLogger()

logger.setLevel(logging.INFO)


def download_all_virus_definitions(definitions_folder, definitions_bucket, prefix=""):
    """Download latest virus defitions from S3."""
    logger.info("Download virus definitions from S3")
    os.system(f"mkdir -p {definitions_folder}")
    response = s3.list_objects(Bucket=definitions_bucket, Prefix=prefix)
    if not response.get("Contents"):
        update_virus_definitions(definitions_folder)
    else:
        for obj in response["Contents"]:
            s3.download_file(
                definitions_bucket, obj["Key"], definitions_folder + obj["Key"]
            )
            if obj["Key"].endswith("/"):
                download_all_virus_definitions(definitions_bucket, obj["Key"])


def update_virus_definitions(definitions_folder):
    """Update local copy of DBs."""
    logger.info("Configure cvd and run cvd update")

    os.system(f"python3 /var/task/dependencies/bin/cvd config set --dbdir {definitions_folder}")
    os.system("python3 /var/task/dependencies/bin/cvd update")


def upload_all_virus_definitions(definitions_bucket, folder_name):
    """Upload virus definitions to S3."""
    logger.info("Upload latest virus definitions to S3")
    # Traverse the directory tree and upload each file to the bucket
    for root, _, files in os.walk(folder_name):
        for file in files:
            # Construct the full file path
            local_file_path = os.path.join(root, file)
            s3_file_path = file
            # Use the S3 client to upload the file to the bucket
            s3.upload_file(local_file_path, definitions_bucket, s3_file_path)


def lambda_handler(event, context):  # pylint: disable=unused-argument
    """Lambda Handler which runs scheduled."""
    definitions_bucket = os.getenv("DEFINITIONS_BUCKET")
    definitions_folder = "/tmp/virus-definitions/"

    # Download all files to /tmp/virus-definitions
    download_all_virus_definitions(definitions_folder, definitions_bucket, "")

    # Update update update
    update_virus_definitions(definitions_folder)

    # Upload virus definitions back to S3
    upload_all_virus_definitions(definitions_bucket, definitions_folder)

AWS Login:

aws sso login --profile playground-admin
export AWS_PROFILE=playground-admin

AWS CLI:

aws s3api create-bucket \
    --bucket clamav-virus-definitions-$(openssl rand -hex 4) \
    --region eu-west-1 \
    --create-bucket-configuration LocationConstraint=eu-west-1

aws iam create-role \
    --role-name cvdupdate-lambda-role \
    --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"Service": "lambda.amazonaws.com"},"Action": "sts:AssumeRole"}]}'

aws iam put-role-policy \
    --role-name cvdupdate-lambda-role \
    --policy-name s3-access-policy \
    --policy-document '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"], "Resource": ["arn:aws:s3:::clamav-virus-definitions/*", "arn:aws:s3:::clamav-virus-definitions"]}]}'

Makefile:

# SHELL:=/bin/bash
WORKLOAD_NAME ?= my-project

clean:
    rm -rf artifacts
    mkdir -p artifacts

clean_cvdupdate_dependencies:
    rm -rf artifacts/cvdupdate.zip
    rm -rf handlers/cvdupdate/dependencies

package_cvdupdate:  clean_cvdupdate_dependencies
    pip install --target handlers/cvdupdate/dependencies -r handlers/cvdupdate/requirements.txt --upgrade
    cd handlers/cvdupdate && zip -r9 ../../artifacts/cvdupdate.zip *

deploy_cvdupdate: clean_cvdupdate_dependencies package_cvdupdate
    aws s3 cp artifacts/cvdupdate.zip s3://${WORKLOAD_NAME}-deploy/functions/cvdupdate.zip
    aws lambda update-function-code --function-name ${WORKLOAD_NAME}-cvdupdate --s3-bucket=${WORKLOAD_NAME}-deploy --s3-key=functions/cvdupdate.zip

Setting up part 2 - The virus scanner

We chose ClamAV to be our virus scanner, the reasons for this are that it is open source, lightweight, has a high detection rate, is very customisable and well supported.

Developing Clambda (ClamAV+Lambda combined 😎)

handlers/clambda/requirements.txt:

filetype

handlers/clambda/clambda.py:

"""ClamAV Lambda"""
# pylint: disable=logging-fstring-interpolation
# pylint: disable=line-too-long

# Standard imports
import os
import json
import logging
import subprocess

# Non standard imports
import boto3
import filetype  # pylint: disable=import-error

# Constants
DEFINITIONS_BUCKET = os.getenv("DEFINITIONS_BUCKET")
DEFINITIONS_FOLDER = os.getenv("DEFINITIONS_FOLDER")
TMP_FOLDER = os.getenv("TMP_FOLDER")
DEFAULT_STATUS = "FAILURE"

# Clients
s3 = boto3.client("s3")
transfer = boto3.client("transfer")

# Logger
logger = logging.getLogger()
logger.setLevel(logging.INFO)


class VirusFoundException(Exception):
    """VirusFoundException for when a virus is found."""


def download_all_virus_definitions(definitions_folder: str, bucket_name: str):
    """Download latest ClamAV virus definitions.
    Provide a bucket name on which the ClamAV virus definitions can be found.
    Provide a target folder to download them to.
    """
    # Code to download the virus definitions from the S3 bucket
    logger.info("Downloading latest virus definitions from S3!")
    os.makedirs(definitions_folder, exist_ok=True)
    response = s3.list_objects(Bucket=bucket_name)
    for obj in response.get("Contents", []):
        s3.download_file(
            bucket_name, obj["Key"], os.path.join(definitions_folder, obj["Key"])
        )


def download_file_from_s3(bucket: str, object_key: str, local_file_path: str):
    """Download object from S3 based on bucket name and object key.
    File is written to a given local file path.
    """
    logger.info(f"Downloading object: arn:aws:s3:::{bucket}/{object_key}...")
    os.makedirs(os.path.dirname(local_file_path), exist_ok=True)
    s3.download_file(bucket, object_key, local_file_path)

    return local_file_path


def guess_file_type(file_path: str):
    """Guess the filetype by using the filetype library."""
    file_type = filetype.guess(file_path)
    if file_type is None:
        logger.info("Cannot guess file type!")
        return
    logger.info(f"File MIME type is: {file_type.mime}")


def start_clamscan(definitions_folder: str, local_file_path: str):
    """Run the clamscan binary."""
    logger.info("Start scanning!")
    try:
        # Start scan and capture output
        scan_output = subprocess.check_output(
            f'/var/task/dependencies/bin/clamscan --database {definitions_folder} "{local_file_path}"',
            shell=True,
        )
        # Log scan output
        logger.info(scan_output.decode())
        logger.info("Scanning done, no viruses found!")
        return "SUCCESS"  # Overwrite default status
    except subprocess.CalledProcessError as exc:
        if exc.returncode == 1:
            logger.info(f"{exc.output.decode()}")
            raise VirusFoundException(  # pylint: disable=raise-missing-from
                "Virus Found!"
            )
        logger.error(f"Exception: {exc}")
        raise exc


def error_exit(
    metadata: dict,
    msg="",
    exception=None,
):
    """
    Exit Function. Always callback the calling AWS Transferservice.
    """
    workflow_step_state(**metadata)
    raise SystemExit(f"Error {msg}") from exception


def workflow_step_state(
    workflow_id: str, execution_id: str, token_id: str, status: str
) -> bool:
    """
    Reports the step state back to calling AWS Transfer Workflow
    """
    response = transfer.send_workflow_step_state(
        WorkflowId=workflow_id, ExecutionId=execution_id, Token=token_id, Status=status
    )

    return response


def lambda_handler(event, context):  # pylint: disable=unused-argument
    """Lambda Handler which gets an event from AWS Transfer Service.
    Example event:
    {
        "token":"secret_token",
        "serviceMetadata":{
            "executionDetails":{
                "workflowId":"w-abcdefghikjl123",
                "executionId":"423c2412-15d5-4786-a731-05d4b5f79ba6"
            },
            "transferDetails":{
                "sessionId":"abcdefghikjl123",
                "userName":"epeters",
                "serverId":"s-abcdefghikjl123"
            }
        },
        "fileLocation":{
            "domain":"S3",
            "bucket":"sftp-poc-1",
            "key":"epeters/test-file.txt",
            "eTag":"089c2c18cf2fe8979143223faeb5298e",
            "versionId":"None"
        }
    }
    This Lambda makes use of: fileLocation.bucket and fileLocation.key.
    We use these values to download the file that needs to be scanned by ClamAV.
    Before the scan starts, the latest virus definitions are downloaded from S3.
    """
    logger.info(event)

    # Extract values from the event
    sftp_bucket = event["fileLocation"]["bucket"]
    object_key = event["fileLocation"]["key"]
    workflow_id = event["serviceMetadata"]["executionDetails"]["workflowId"]
    execution_id = event["serviceMetadata"]["executionDetails"]["executionId"]
    token_id = event["token"]

    # Prepare metadata dictionary
    metadata = {
        "workflow_id": workflow_id,
        "execution_id": execution_id,
        "token_id": token_id,
        "status": DEFAULT_STATUS,
    }

    # Download file that needs to be scanned from S3
    s3_object = download_file_from_s3(
        sftp_bucket, object_key, os.path.join(TMP_FOLDER, object_key)
    )

    # Guess the file MIME type
    # TODO: Do something with this when for example the file type is unsupported
    guess_file_type(s3_object)

    # Download all virus definitions to /tmp/virus-definitions
    download_all_virus_definitions(DEFINITIONS_FOLDER, DEFINITIONS_BUCKET)

    # Run the scan
    try:
        scan_result = start_clamscan(DEFINITIONS_FOLDER, s3_object)
        metadata["status"] = scan_result
    except Exception as exc:  # pylint: disable=broad-except
        error_exit(
            metadata=metadata,
            msg="Failure during scanning file, sending error back to Transfer Service",
            exception=exc,
        )

    # Return the state to the transferservice workflow
    response = workflow_step_state(**metadata)
    logger.info(json.dumps(response))

NOTE: When I have the time I will improve this by using libclamav directly instead of the Python subprocess module, because it is not ideal to use Python to spawn other programs. However, since libclamav is written in C, integrating it with Python on AWS Lambda requires the use of ctypes.

ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap libclamav in pure Python.

An example of how this can be done can be found on this GitHub repository.

Makefile:

# SHELL:=/bin/bash
WORKLOAD_NAME ?= my-project

clean:
    rm -rf artifacts
    mkdir -p artifacts

clean_clambda_dependencies:
    rm -rf ${PWD}/usr
    rm -rf artifacts/clambda.zip
    rm -rf handlers/clambda/dependencies

package_clambda: clean_clambda_dependencies
    mkdir -p handlers/clambda/dependencies/{bin,lib}
    pip install --target handlers/clambda/dependencies -r handlers/clambda/requirements.txt --upgrade
    curl -L https://github.com/Cisco-Talos/clamav/releases/download/clamav-${CLAMAV_VERSION}/clamav-${CLAMAV_VERSION}.linux.x86_64.rpm \
        --output artifacts/clamav-${CLAMAV_VERSION}.linux.x86_64.rpm
    @if [[ ${UNAME} == 'Darwin' ]]; then \
        echo "Run macOS commands for package_clambda"; \
        tar xvf artifacts/clamav-${CLAMAV_VERSION}.linux.x86_64.rpm \
        -C handlers/clambda/dependencies/bin/ \
        --strip-components=4 \
        usr/local/bin/clamscan; \
        tar xvf artifacts/clamav-${CLAMAV_VERSION}.linux.x86_64.rpm \
        -C handlers/clambda/dependencies/lib/ \
        --strip-components=4 \
        usr/local/lib64/*.so.*; \
    else \
        echo "Run Linux commands for package_clambda"; \
        rpm2cpio artifacts/clamav-1.0.0.linux.x86_64.rpm | cpio -idmv; \
        mv ${PWD}/usr/local/bin/clamscan handlers/clambda/dependencies/bin/; \
        mv ${PWD}/usr/local/lib64/*.so.* handlers/clambda/dependencies/lib/; \
    fi
    cd handlers/clambda && zip -r9 ../../artifacts/clambda.zip *

deploy_clambda: clean_clambda_dependencies package_clambda
    aws s3 cp artifacts/clambda.zip s3://${WORKLOAD_NAME}-deploy/functions/clambda.zip
    aws lambda update-function-code --function-name ${WORKLOAD_NAME}-clambda --s3-bucket=${WORKLOAD_NAME}-deploy --s3-key=functions/clambda.zip

Part 2?

You might have noticed references to AWS Transfer Family. This is because Clambda has been integrated into an SFTP solution, making it part of a secure file transfer workflow. If you’re interested in a step-by-step guide on this integration, let me know in the comments. I might write a part 2 for this.

Conclusion

In this guide, we explored how to build a serverless virus scanner on AWS using S3 for storage, EventBridge Scheduler for automation, and Lambda for scanning with ClamAV. This solution is cost-effective, scalable, and requires minimal maintenance, perfect for keeping your systems secure.

Try following the steps to build your own serverless virus scanner, and feel free to share your feedback or ideas for improvements in the comments.

Exploring the Differences Between Amazon EC2 Instance Store and Amazon EBS

Enri Peters — Thu, 29 Dec 2022 13:30:46 +0000

Introduction

In the world of cloud computing, it's important to understand the various storage options available to you. Amazon Web Services (AWS) offers two main types of storage for its Elastic Compute Cloud (EC2) instances: instance store and Amazon Elastic Block Store (EBS). In this blog post, we'll discuss the key differences between the two and help you determine which one is the best fit for your needs.

Definition of EC2 instance store

Instance store is a type of temporary storage that is physically attached to the host computer of an EC2 instance. It is also known as ephemeral storage because it is not persisted when the instance is stopped or terminated.

Definition of Amazon EBS

Amazon EBS is a persistent storage service that allows you to create and attach one or more storage volumes to an EC2 instance. These volumes are independent from the host computer and can be detached and attached to other instances as needed.

Performance

In terms of performance, instance store tends to be faster than EBS because it is physically attached to the host computer. However, the performance of EBS has improved significantly over the years and it is now able to deliver high levels of input/output operations per second (IOPS) for workloads that require it.

Durability

One major advantage of Amazon EBS over instance store is durability. Because EBS volumes are stored on separate physical servers, they are less vulnerable to data loss due to hardware failure. On the other hand, instance store data is at risk of being lost if the host computer fails.

Instance store data is temporary and only persists for the duration of the associated instance. If the instance reboots, the data in the instance store will still be present. However, the data will be lost under the following circumstances:

The underlying disk drive fails
The instance stops
The instance hibernates
The instance terminates

It is important to note that instance store should not be relied upon for storing valuable, long-term data. Instead, consider using more durable storage options such as Amazon S3, Amazon EBS, or Amazon EFS for this purpose.

Cost

In terms of cost, instance store is generally cheaper than EBS because you don't have to pay for the storage separately. However, EBS allows you to pay for only the storage you need and scale up or down as needed, which can be more cost-effective for certain workloads.

Support

It is important to note that not all EC2 instances support instance storage, and even among those that do, the amount of storage available can vary. To get a list of all instance types that support instance storage, you can use the following AWS CLI command:

aws ec2 describe-instance-types \
    --filters "Name=instance-type,Values=*" "Name=instance-storage-supported,Values=true" \
    --query "InstanceTypes[].[InstanceType, InstanceStorageInfo.TotalSizeInGB]" \
    --output table | sort

For example, T2 and T3 instances do not support instance storage. You can find more information on EC2 instance storage in the AWS EC2 instance storage user guide.

Conclusion

In summary, both EC2 instance store and Amazon EBS offer unique benefits and trade-offs. Instance store is a fast and cost-effective option for temporary storage, but it is not durable. Amazon EBS is a more durable option that allows you to scale your storage needs, but it comes at a higher cost. Ultimately, the right choice for your needs will depend on the specific requirements of your workload.

Mastering Local AWS Lambda Development

Enri Peters — Tue, 13 Dec 2022 22:20:32 +0000

Introduction

Are you tired of deploying your AWS Lambda functions to AWS just to test them? Do you feel like you're constantly uploading your code just to see if it works?

Well, fear not my fellow Lambda enthusiasts! In this post, I will share with you two simple ways to do local AWS Lambda development. That's right, no more deployments or tedious back-and-forth just to test your code. You'll be able to run your Lambda functions right on your own computer, just like a boss.

So, grab a cup of coffee and get ready to unleash the full potential of your local Lambda development skills. Let's do this!

Prerequisites

To develop Lambda functions locally, you will need to have the following prerequisites installed on your system:

Docker to emulate the Lambda runtime on your machine
Python with a version equal or greater than 3.6
pipenv to not make a mess of your local Python installation

You can install pipenv by running the following command:

pip install pipenv

Once you have these prerequisites installed, you will be ready to start developing your lambda functions locally.

Method 1 - AWS Serverless Application Model

The first method we will discuss is using AWS SAM (Serverless Application Model). AWS SAM is an open-source framework that allows developers to quickly and easily create, build, and deploy applications that run on the AWS platform.

With AWS SAM developers can test their AWS Lambda functions locally, without having to deploy them to AWS. It does this by using a local version of the AWS Lambda runtime, which simulates the environment in which Lambda functions execute in AWS.

This is useful because it allows developers to test their Lambda functions quickly and easily, without having to incur the costs or wait for the deployment process associated with deploying to AWS. It also makes it easier to debug and troubleshoot problems with Lambda functions, since developers can see the output and errors directly on their own computers.

To install and set up the AWS SAM CLI on your local machine, follow these steps:

mkdir sam-demo
cd sam-demo
pipenv shell

NOTE: pipenv will launch a subshell in a virtual environment, to exit this subshell type 'exit'. Don't forget to do this once your done.

Now within your virtualenv created by pipenv you can install the SAM CLI by running:

pipenv install aws-sam-cli

Once the SAM CLI is installed, you can use the sam init command to initialise your SAM project. Use the sam init command as shown below to create a SAM project based on the "Hello World" template. This template includes a Lambda Function and a REST API. We will not be using the API part, but feel free to play with it.

sam init \
  --name my-sam-app \
  --app-template hello-world \
  --runtime python3.9 \
  --dependency-manager pip \
  --package-type Zip \
  --no-tracing

Make sure you change directories into the 'my-sam-app' folder created by SAM.

cd my-sam-app

You can use and edit the files in this 'hello-world' example to meet you needs. For example you can change the template.yaml file to add extra serverless resource blocks. Additionally you can change the code inside the Lambda function python file: hello_world/app.py.

Now that you know this, it's time to build and run your Lambda Function code by using the following commands:

sam build --use-container
sam local invoke

Congratulations! You have successfully learned the basics of using the SAM CLI tool to develop and test Lambda functions on your local machine.

This approach involves using the AWS SAM CLI and has the advantage of creating a streamlined development flow. The SAM CLI makes it easy to develop and test serverless applications locally, without the need to deploy them to AWS. This allows developers to iterate quickly and test their changes without incurring the cost and overhead of deploying to AWS.

However, it also has the disadvantage of that it may not be the best choice for complex or large-scale applications. While the SAM CLI is a great tool for developing and deploying simple or medium-sized serverless applications, it may not be the best choice for more complex or large-scale applications that require a more advanced deployment or management solution. In these cases, you may want to consider using other tools or frameworks that provide additional features and capabilities.

Method 2 - Docker run

There might be a moment while happily using method 1 that you think "I do not want to keep running sam build every time I make a change to my Lambda Function code".

Well, there is another way to locally test your Lambda Function, but in a slightly different way.

Before, when you ran sam build, the tool was building a Docker container image behind the scenes which is based on the public.ecr.aws/sam/build-python3.9 docker image.

We can use this base image and take full control of it, moo-hahaha!

Let's give this a try by pulling this container to our machine:

docker pull public.ecr.aws/sam/build-python3.9

Once the image has been downloaded, you can use the docker run command to start a new container based on the image. When starting the container, you will need to specify the following arguments:

The -it flag to be able to interact with the container
The --read-only flag which will create a read-only file system, this mimics the AWS Lambda behaviour.
The --tmpfs flag with /tmp argument which will create a temporary file-system on /tmp inside the container to which you can write. You also get this while running your Lambda Function inside AWS.
The -v flag which mounts a local directory containing your Lambda function code inside the container.
The -e flag for setting environment variables that the Lambda runtime can use while executing your function.

NOTE: You can only write into /tmp with a Lambda Function. In some cases (not entirely known to me) temporary storage is not deleted between warm Lambda invocations, this can help in having a faster execution of your Lambda Function.

Let's look at an example.

This example assumes you have a my_lambda_function.py file in your current directory, with the following content:

# my_lambda_function.py

def lambda_handler(event, context):
    print("Hello from Lambda!")

if __name__ == '__main__':
    lambda_handler(None, None)

Once you have created this my_lambda_function.py file you can continu by running the below command to start an interactive session with your emulated Lambda Function container.

docker run -it \
--read-only \
--tmpfs /tmp \
-v $(pwd):/var/task \
-e MY_ENV_VAR_1=hello \
-e MY_ENV_VAR_2=world \
public.ecr.aws/sam/build-python3.9

After you hit enter you will be entering the container immediately, you can verify this by looking at your PS1 prompt which should look something like this:

bash-4.2#

This prompt indicates that the container is ready to accept commands. Let's try some things to verify what we just did. Let's start by trying to write something on the root of the containers filesystem.

bash-4.2# touch /this
touch: cannot touch ‘this’: Read-only file system

Look at that! Our --read-only does indeed help in simulating a Lambda Function. You can't touch this!

Now let me explain the -v flag we passed to the docker run command. The -v flag is used with the docker run command to create a volume, which is a persistent storage location that can be accessed from the container. This flag allows you to specify a host directory to be mounted as a volume in the container. The -v flag can be used multiple times to mount multiple volumes (you could do this when you want to simulate Lambda layers). If you want to know more about this, please leave a comment.

For example, the above command uses the -v flag to mount the hosts current directory as the volume /var/task in the container.

NOTE: The $(pwd) command returns the current working directory.

This again simulates our Lambda Function perfectly. Now let's check how to run our Lambda Function. But first let's verify our Python version by running:

bash-4.2# python -V
Python 3.9.15

Nice! We are running Python 3.9.x on our local Lambda Function, let's try to run our functions code:

bash-4.2# python /var/task/my_lambda_function.py
Hello from Lambda!

Congratulations on successfully running your Lambda function code from within a Docker container! This is a great achievement, as it allows you to run your code in a consistent and predictable environment, which can be especially useful for testing and debugging.

This approach involves using Docker and has the advantage that you can now use your favourite text editor or IDE to make changes to your my_lambda_function.py file. Because the volume is mounted, any changes you make to the file on your host machine will be immediately available within the Docker container, allowing you to quickly test and iterate on your code.

However, it also has the disadvantage of that it may not provide an exact replica of the AWS Lambda runtime environment behaves. As a result, there could be differences in how the function behaves when tested locally using Docker compared to when it is actually executed in the AWS Lambda environment.

Conclusion

By using these approaches, you can test your Lambda functions locally without having to deploy them to the AWS cloud. This can be useful for development and testing purposes, as it allows you to iterate quickly and make changes to your code without incurring the overhead of deploying to AWS.

What do you think? Share your thoughts in the comments below!

Playlist: Now Go Build with Werner Vogels - Season 1, 2 & 3

Enri Peters — Thu, 06 Oct 2022 19:37:41 +0000

Season 1️⃣

Episode 1 - Jakarta 🇮🇩

Follow Dr. Werner Vogels, Amazon CTO, through Indonesia as he meets with HARA Token, a Jakarta based startup, to explore the technology that links small holder rice farmers with financial services with their innovative eco-system.

Episode 2 - Singapore 🇸🇬

Dr. Werner Vogels, Amazon CTO, travels to the hot start up scene in Singapore, meeting with Zimplistic, founded by Pranoti Nagarkar and Rishi Israni - the makers of Rotimatic, in order to find out why complex IOT and Machine Learning technology was needed to replicate a bread making process that’s been handed down from parents to their children over generations.

Episode 3 - Bergen 🇳🇴

Dr. Werner Vogels, Amazon CTO, travels to the scenic fjords in Bergen, Norway to meet with the founder of Aquabyte to learn how AI is playing a new role in salmon farming.

Episode 4 - Berlin 🇩🇪

Dr. Werner Vogels, Amazon CTO, travels to Berlin, Germany to meet with the founders of the ReDI School and Babbel to explore how technology plays a critical role in integrating refugees into the German workforce and their culture.

Episode 5 - Cape Town 🇿🇦

Dr. Werner Vogels, Amazon CTO, travels to South Africa to meet with the founder of the Africam to explore how technology plays a critical role in wildlife conservation efforts. Africam uses live streaming cameras in the African safari to create direct connections for NGO's working to curb poaching and those who want to help.

Episode 6 - Rio de Janeiro 🇧🇷

Follow Dr. Werner Vogels, Amazon CTO, through Rio de Janeiro as he meets with the founders of Dr.Consulta and Epitrack to explore how cloud technology is being applied to re-invent private health care solutions in one of the most beautiful and densely populated cities in Brazil.

Episode 7 - Porto 🇵🇹

Follow Dr. Werner Vogels, Amazon CTO, to the city of Porto, Portugal as he meets with the founders of Veniam and CEiiA and to explore how an ancient city is transforming itself into a smart city to meet the needs of a growing population within the confines of a city that dates back to 300 BC.

Episode 8 - Amsterdam ❌❌❌ 🇳🇱

Follow Dr. Werner Vogels, Amazon CTO, to Amsterdam for the Amsterdam Dance Event, the worlds largest EDM festival, to explore how cloud technology enhances the artistic process and the ability to get their music out.

Season 2️⃣

Episode 1 - Bengaluru 🙏 🕉 🇮🇳

Follow Dr. Werner Vogels, Amazon.com CTO, to the vibrant streets of Bengaluru, and meet Sujay Suresh and Innu Nevatia, founders of the company Zwende, who wanted to pair the creativity, customization, and accessibility of modern shoppers with the craft of skilled designers and makers.

Episode 2 - Tokyo 🇯🇵

Follow Dr. Werner Vogels, Amazon.com CTO, to Tokyo where taking care of older generations has long been a part of Japanese culture. As life expectancy rises and the number of elderly people far exceeds that of younger people who might take care of them, some creative minds are moving Japan towards new traditions of elderly care.

Werner meets Tatsuya Takahashi, co-CEO of Z-Works, Inc., Yoichi Ochiai, CEO of Pixie Dust Technologies, and Seigo Hara, CEO of MICIN, Inc.

Episode 3 - Poland 🇵🇱

Follow Dr. Werner Vogels, Amazon CTO, to Poland—where COVID-19 is keeping students out of the classroom and dialed in to online learning. Inspired by the idea of bringing a student study group online, see how Brainly is helping students bridge the gap between remote online classes and the 1-on-1 attention and guidance that's more accustomed to the classroom.

Werner meets Michał Borkowski, CEO & Co-Founder, and Bill Salak, CTO of Brainly.

Season 3️⃣

Episode 1 - Iceland 🇮🇸

From discovering life-saving medical treatments in the skins of Atlantic cod to reinventing batteries with advancements in nanotechnology, Iceland has proved itself to be one of the most technologically ingenuitive countries on earth. Discover Iceland’s surprisingly successful “waste nothing” approach to innovation with Amazon CTO Dr. Werner Vogels as he visits startups @Kerecis, Nanom, and ON Power.

Episode 2 - Hawai‘i 🌋

Travel to the Island of Hawai‘i with Amazon CTO Dr. Werner Vogels as he visits @Terraformation, a global native forest restoration startup working to combat climate change with the planting of one trillion trees– a task made possible with the help of solar-powered desalination, seed banking, and cloud-based technologies.

Episode 3 - Vermont 🇺🇸

Amazon CTO Dr. Werner Vogels travels to Burlington, Vermont, where the startup BETA Technologies is redefining the future of commercial, medical, and cargo aviation with its electric vertical aircrafts.

Episode 4 - Copenhagen 🇩🇰

Travel to Copenhagen with Amazon CTO Dr. Werner Vogels as he visits Veo, a company working to make sports technology accessible to all; made possible with machine learning, a love of soccer, and a mission to democratize sports for everyone.

Episode 5 - Hollywood, Los Angeles 🎥

How do you make a digital human? Hint: it starts with the cloud and digital DNA. Travel with Amazon CTO Dr. Werner Vogels as he visits Soul Machines, an AI company who uses the cloud to deliver meaningful connections between human and machine. This episode features digital capture technology, Lightstage from OTOY.

Episode 6 - Arizona ⛰️

Can autonomous trucking help with supply chain issues and give drivers an opportunity to work closer to home? Travel with Amazon CTO Dr. Werner Vogels as he visits TuSimple, a technology company who uses the cloud to help autonomous trucking work toward safer, smarter roads.

How I failed to become an AWS Community Builder

Enri Peters — Fri, 19 Aug 2022 20:36:00 +0000

But how I succeeded next time and how you could as well.

Introduction

Are you curious about the journey of someone who failed to become an AWS Community Builder? While it may seem counterintuitive, reading about someone's failure can often provide valuable insights and lessons for achieving success.

On August 17, I received an email from AWS that informed me that I had not been selected for the Community Builder program. While initially disappointing, this experience allowed me to learn and grow in ways that I never would have anticipated.

Through this blog post, I hope to share my story and the valuable lessons I learned along the way. Whether you're applying to become a Community Builder or pursuing any other goal, I believe my experience can provide valuable insight and inspiration for achieving success.

The AWS Community Builders program 🛠

The AWS Community Builders program is an exclusive program offered by AWS that provides technical resources, mentorship, and networking opportunities to individuals who are passionate about sharing their knowledge and connecting with the AWS community. The program is designed for AWS enthusiasts and emerging thought leaders who are eager to learn more about AWS and share their knowledge with the community.

The program is open to applicants worldwide and offers a diverse group of passionate and knowledgeable individuals the opportunity to connect with other AWS enthusiasts and thought leaders, as well as gain access to exclusive technical resources and mentorship from AWS experts.

For more information about the program, including eligibility requirements and application process, please visit the AWS Community Builders page. Additionally, it would be beneficial to read the frequently asked questions on the website, as they provide valuable insights into the program and the expectations of the community builders.

Key people 🔑

AWS, being the leading public cloud provider, offers a wealth of opportunities for community engagement through its Community Builders program. This program, which has been in existence since 2020, aims to provide technical resources, mentorship and networking opportunities to passionate AWS enthusiasts and emerging thought leaders who are dedicated to sharing their knowledge and connecting with the AWS community.

To fully utilize the program's knowledge and resources, it is essential to stay informed about the program by following key individuals who are involved. These individuals include:

Ross Barich, Head of AWS Community
Jason Dunn, AWS Community Builders Program Manager
María Encinar, AWS EMEA Senior Community Programs Manager
Shafraz Rahim, AWS Developer Community Program Manager for Asia Pacific, Japan
Farrah Campbell, AWS Sr. Product Marketing Manager for Containers & Serverless
Taylor Katherine Lacy, AWS Developer Community Manager
Caroline Kerns, AWS Community Manager
Albert Zhao, AWS Developer Community Manager
Mark Pergola, AWS Community Content Manager

This list is not exhaustive and if you are aware of other key individuals involved in the AWS Community Builders program, please share in the comments so the list can be updated.

Do you want to become an AWS Community Builder? 🤔

If you're someone who is driven to make an impact, wants to be part of something special, and is passionate about helping others in their AWS journey, then the AWS Community Builders program may be the perfect fit for you. The program is designed for individuals who want to build long-term relationships, improve their self-confidence, and be recognized in the AWS community. Most importantly, the program is built on the belief that the success of a group is greater than that of an individual. If you share these values and are passionate about AWS, consider applying to the program to take your community involvement to the next level.

Dont's ❌

The most important part of this blog post is about taking action on your ideas for creating and sharing content. Don't let fear or perfectionism hold you back from starting. It may seem daunting, but remember that progress is made one step at a time.

For example, instead of putting off creating a YouTube channel because you're worried about the perfect banner or intro, just start recording a video. Don't worry about whether your content will be valuable to others, just share what you know and what you're passionate about.

It's also important to not get too caught up in watching others and trying to emulate their success. While it's great to be inspired by others, remember that everyone's journey is unique and you will learn the most by taking action and creating your own content. Don't put pressure on yourself to create top-notch content right away, just start and you'll get better with time.

Do's ✅

There are many ways to get started in the AWS community and make an impact. Social media platforms like LinkedIn and Twitter are great places to start. You can increase visibility by using relevant tags such as #aws, #cloud, #awscommunity, #awscommunitybuilder, and #awscommunitybuilders. Don't forget to also like, comment, and share posts made by others, and consider mentioning companies in your posts using the "@" symbol. Joining AWS groups on social media, such as the AWS Certified Global Community, can also be beneficial.

Another great way to get involved is to join your local AWS User Groups. Meetups are a great opportunity to improve your public speaking skills and you can reach out to the organizers and offer your help with organizing meetups and community days.

Blogging is a great way to improve your writing skills and you can start today by creating an account on https://dev.to or https://medium.com. Interacting on platforms such as Twitter's AWS Developers group, AWS re:Post, Reddit, and Stack Overflow can also be beneficial.

If you're interested in contributing to open source projects, consider checking out GitHub - AWS Labs and see if you can add any value by contributing code or by improving documentation.

Connecting with and following AWS Heroes, Ambassadors, User Group Organizers/Leaders/Volunteers and Community Builders can give you insights on what they are working on and how you can contribute. You can also check out AWS Week in Review and What's New with AWS to stay up to date with the latest news and developments in the AWS community.

Finally, consider starting an AWS newsletter, in which you can summarize the latest news and developments and share your thoughts and insights on why certain features are important or amazing. You already got one subscriber (me)!

Optional do's ✅

Becoming an AWS Community Builder can be a challenging journey, but one way to increase your chances of success is by obtaining AWS certifications. Not only will these certifications help you learn more about AWS and advance your career, but they also demonstrate your knowledge and skills to others in the community.

To help you on your certification journey, here are some resources that can be useful:

Cloud Quest is a fun role-playing game that allows you to build solutions in a live AWS environment.
AWS Educate is open to any individual, regardless of where they are in their education, technical experience, or career journey.
AWS Skill Builder is an AWS learning center that helps you improve your cloud skills.
Adrian Cantrill offers some of the best AWS training available, and many of his fundamentals courses can be watched for free on his YouTube channel.
Stephane Maarek is a best-selling instructor and Kafka expert who offers AWS certification courses on Udemy.
Jon Bonso from Tutorials Dojo offers some of the best practice exams available, with explanations for correct and incorrect answers.
Neal Davis from Digital Cloud Training offers top-quality training, cheatsheets, and free resources on his website and on the YouTube channel of freeCodeCamp.
Andrew Brown who is an AWS Hero and has created a learning platform called ExamPro which has great resources for becoming AWS certified. He also has created multiple AWS certification courses which are available for free on the YouTube channel of freeCodeCamp.

A Cloud Guru, Cloud Academy, Udemy, Coursera, Whizlabs, MeasureUp and many other platforms also offer AWS certification courses and practice exams.

It's important to note that these are just a few of the many resources available, and it's important to find one that works best for you and your learning style. Additionally, it's also important to note that obtaining certifications is not the only way to become an AWS Community Builder, and it's important to also actively engage in the community by sharing your knowledge, helping others, and contributing to open-source projects. Joining and participating in local AWS user groups, speaking at meetups, and creating content such as blogs, videos, and podcasts are also great ways to build your presence in the community.

Ultimately, becoming an AWS Community Builder is about consistently putting in the effort to learn, share, and engage with others in the community. Whether it's through obtaining certifications, participating in online forums, or speaking at events, it's important to find ways to add value and make a positive impact in the community.

Tell me the benefits! 👍

There are many benefits to becoming an AWS Community Builder, including increased visibility and recognition within the community. As a member of the program, your name will be listed in the Community Builders directory, which can help you gain more visibility and recognition within the community.

You will also gain access to exclusive resources and opportunities, such as invitations to private platforms like the Community Builders Slack workspace where you can connect with other Community Builders and AWS employees, including subject matter experts.

In addition to these opportunities, you will also receive:

Access to AWS product teams and information
AWS promotional credits to use in your own AWS accounts to build projects
Access to training resources such as Cloud Academy
One certification voucher for an AWS exam

But perhaps one of the most exciting benefits is the exclusive AWS swag, such as water bottles, stickers, t-shirts, hats, and backpacks.

It's worth noting that AWS is always working to improve the benefits of the program, so you can expect them to evolve over time. Another fun fact is that if you ever attend the AWS re:Invent conference in Las Vegas, as a Community Builder you will receive special recognition, such as a Community Builder sticker on your badge and access to shorter lines for keynotes and events. This can give you some extra time to rest and recover from any previous night's activities 😉.

I am NOT convinced! Sounds like a lot of work! 📚

It's important to remember that becoming an AWS Community Builder doesn't require you to do everything on this list. The beauty of the community is that we all bring different skills and perspectives to the table. Instead of trying to do everything at once, start small by dedicating 15-30 minutes a day to building your community presence. Incorporating it into your daily work can also be helpful. For example, use the knowledge you gain from working on a project to create a blog post.

Personally, one of the biggest motivators for me is pushing myself out of my comfort zone. By becoming more visible in the community, I've learned to be more confident in my skills and abilities, which has helped me grow as a professional. Additionally, being recognized by others in the community is rewarding and helps me build my confidence. The best part is getting to meet so many amazing people in the process.

But I don't have any time ⏱

I agree it can be challenging to find time to become an AWS Community Builder. However, it is important to remember that you don't have to do everything at once. Instead, start small by setting aside 15 or 30 minutes a day to write blogs or create content. You can also make it part of your daily routine, for example, by writing a blog post about a project you are working on.

It's important to break out of your comfort zone and start taking action instead of just making plans. You can create a dev.to account in just a few minutes and start writing right away. You can even save drafts and continue working on them later.

It's also helpful to find someone who can coach and support you in your efforts. Consider having a weekly catch-up with a colleague to discuss your progress and get feedback on your content. Remember, it's never too late to start, and even small steps can make a big difference in the long run.

What has changed for me? 🧐

In the time since I began writing this blog, I have had the opportunity to speak at and lead workshops at various AWS events. I have been sharing my experiences and insights on social media platforms such as Twitter and LinkedIn. In fact, I even received a public thank you from Werner Vogels, the CTO of AWS, for creating a special birthday video for him.

Additionally, I have started my own YouTube channel, where I share tips and tutorials on various AWS-related topics. While it was tempting to spend a lot of time perfecting the channel's banner and intro, I learned that it's actually quite simple to create a professional-looking design using tools like PowerPoint.

One of the most valuable things that has helped me stay motivated and on track with my goals is having regular catch-up sessions with a mentor. These check-ins allow me to receive feedback and guidance, which helps me to stay focused and continue to grow as a community builder.

I am convinced! But how to apply? 🍀

To kickstart your journey as an AWS Community Builder, you can begin by adding your name to the waitlist for the program (this link only works when applications are closed). By doing so, you'll be notified when the next cycle of applications is open, and receive instructions on how to apply. Additionally, you can take steps to begin building your presence within the AWS community by engaging on social media platforms, joining local user groups, and contributing to community resources such as Github and Stack Overflow. Remember to focus on consistent and steady progress, rather than trying to do everything at once.

Good to know 📚

When applying to the AWS Community Builders program, it's important to choose the categories that align with the content you've created or plan to create. The categories available are Cloud Operations, Containers, Data, Dev Tools, Front-End Web & Mobile, Game Tech, Graviton Arm, Machine Learning, Networking & Content Delivery, Security & Identity, Serverless and Storage. You can choose two categories when applying, but keep in mind that the content you submit should be relevant to these categories.

AWS Community Builders are also grouped by geographic region, including Asia Pacific, Europe, Middle East, & Africa, Latin America, and USA & Canada. As of October 2022, there are around 2350 AWS Community Builders worldwide.

One way to find other Community Builders in your area is to check out the map created by @jdonboch, which uses AWS Location Services. You can find the map here and read more about it in his blog post here.

Thank you

Thank you for reading my blog post. I hope you found it informative and helpful in your journey to becoming an AWS Community Builder. If you are interested in keeping up with my content, please consider subscribing to my dev.to account and YouTube channel Obsessed By AWS. Your support will not only help me reach my goal of 100 subscribers, but also ensure that you stay updated on my latest content and updates.