DEV Community: Andrea Falzetti

AWS Lambda: Lessons learned over 5 years and 100 functions in production

Andrea Falzetti — Fri, 31 Mar 2023 04:00:12 +0000

At onoranzefunebricloud.com we provide funeral agencies with digital solutions to do their best work, and to build such complex tools we use a combination of AWS services including dockerized applications in Fargate and lot of Serverless tools such as Lambda.

Over the years, as we introduced more functionalities we've also grown our usage of Lambda, and as of today we count over 100 Lambda functions in production.

In this post I want to share the lessons learned, the good and the bad.

I've organized the content in 4 sections:

The good
The bad
Recommendations
Conclusions

The good

Zero daily operations & no downtime: Over 5 years we never had a downtime on the services built on Lambda. The product has been very reliable for us and we are always confident it will scale for us, when needed, without any manual operation on our side. Wait! Lambda doesn't scale indefinitely! There are soft and hard limits to keep in mind, for example, by default you can have up to 1000 concurrent Lambdas running in a given AWS region at any given time. We have a predictable work load, that's why I have this confidence. Check out Lambda quotas to know more.
💰 Affordable: For a long time we were paying $0! Until our startup got traction, our AWS bills have been very low thanks to using Lambda and the generous AWS free tier.
Tiny codebase ➡️ Easy to contribute to: Having services separated into each own repository and its own set of Lambdas in our experience has simplified and made easier to contribute to a project. The learning curve of the codebase is very small because the codebase is tiny!
Separation of concerns: Having services deployed as an independent Lambda means for us having decoupled components which meant individual contributors felt safe in modifying, deploying a service without the worry of breaking another one. If something went wrong, only an isolated service would be affected and it would be easier to debug.
Versatile: Lambda is very flexible in terms of what you can do with it. In a recent blog post I've even shared how I am using a runtime with multiple programming languages.

The bad

Too many Lambdas: Having too many Lambdas to update could result in a maintenance burden. For example, some of the first Lambdas we built back in 2017 were running in Node.js 14 which AWS has deprecated. We were not planning to work on those, so for us that was unplanned work that we had to undertake to avoid being stuck and blocked in making new changes to those Lambdas.
Expensive if misused: Lambda is not a silver bullet, in fact, if misused it can lead to very expensive bills. Using it for high-load or long-running tasks, it will definitely make you incur in expensive bills. Choose carefully how to use it.
Configuration Expert: Lambda has evolved a lot in the past 5 years. The initial concept: "take this code and run it" is slowly fading away over time, due to the addition of tons of configuration. You can still get started pretty quickly, but you can also get lost in fine tuning or configuring your function.

Recommendations

🎯 Choose, with care

Do not default to Lambda for everything. This is a mistake I did at the beginning, building REST APIs on Lambda might not be the best decision, especially when you grow and scale, you might regret it.

Lambda is great for gluing together components, building event-driven systems and much more, but for long-running and intense compute processes, it might make sense only in certain scenarios.

📟 Get paged when it fails

Your code will fail and you want to know when that happens. We use serverless-plugin-aws-alerts to instrument CloudWatch alarms and dashboards for our Lambdas and OpsGenie to page us on Slack or on our phones.

You need to configure OpsGenie to use SNS and then the simplest approach is alert on failures, for example serverless.yml:

custom:
  # Lambda Alarms
  alerts:
    stages:
      - prod
    dashboards: false
    topics:
      alarm:
        topic: arn:aws:sns:${self:provider.region}:${aws:accountId}:opsgenie
    definitions:
      functionErrors:
        namespace: 'AWS/Lambda'
        metric: Errors
        threshold: 1
        statistic: Sum
        period: 60
        evaluationPeriods: 1
        datapointsToAlarm: 1
        comparisonOperator: GreaterThanOrEqualToThreshold
        treatMissingData: missing
    alarms:
      - functionErrors

This alert is quite aggressive, it will page you as soon as there's an error, you can tune that to your preference.

🛠️ Debug locally

We use VSCode and Serverless to run and debug our code as if it was running in Lambda. Example .vscode/launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "console": "integratedTerminal",            
            "name": "hello",
            "env": {
                "AWS_PROFILE": "<YOUR_PROFILE>",
                "SLS_DEBUG": "*",
            },
            "program": "${workspaceFolder}/node_modules/.bin/sls",
            "args": [
                "invoke",
                "local",
                "-s",
                "dev",
                "-f",
                "hello",
                "-p",
                "./examples/event.json"
            ]
        }       
    ]
}

In ./examples/event.json you can store your test event, such as API Gateway or CloudWatch payloads.

Another approach, can be using OpenFaaS but this is a tool that I didn't have the opportunity to try yet.

📦 Keep it small!

We use serverless-esbuild to reduce the bundle size, it makes deployments and cold time faster.

📚 Stay up to date

You will find having up to date knowledge very handy the next time you're taking an important decision for example using or not Lambda or any other AWS service. I recommend following Yan's work on Twitter and his website theburningmonk.com.

Another valuable asset is AWS This Week by the folks of ACloudGuru.

Conclusions

Lambda is a great tool if leveraged correctly, at the same time it's not a silver bullet.

I suggest you to listen in to some podcasts and read few more blog posts if the concept is still blurred.

Getting started if fairly simple, especially with the Serverless Framework

What are you waiting for? Give it a try.

Cover credits: https://unsplash.com/photos/Q1p7bh3SHj8

Combining Node.js and Python in a Single AWS Lambda Function

Andrea Falzetti — Fri, 17 Mar 2023 04:29:57 +0000

Recently, I encountered a situation where I needed to run a Python script from a Node.js runtime using AWS Lambda.

AWS supports several runtimes for Lambda including Node.js, Python, Java, .Net, Go, Ruby and also Custom Runtime.

In my case I was already using nodejs16.x which of course, does not come with Python installed. After cosindering using Lambda layers or trying to compile the .py script into some stand-alone binary (which failed due to missing Python dependencies on the Lambda anyway), I've decided to give a try to the Custom Runtime.

Initially, I was put off by the possible amout of work involved into configuring a custom image, I guessed it would require at least a Dockerfile, publish the image to a registry and then instruct my Lambda to use that image, configure IAM permissions, etc.. Luckily, with Serverless.com I figured the framework itself would do most of the heavy lifting so it turned out to be easier than expected. I presume doing the same with Terraform would require a little bit more of code.

I've decided to put together this blog post to help out folks doing something similar given that it wasn't so easy to find docs about this.

Let's go through the steps required:

Create a Dockerfile
Publish the image to ECR
Instrument your Lambda to use the new custom runtime

Create a Dockerfile for a custom AWS Lambda runtime

This is simple as writing any Dockerfile, it's helpful to know that you can use AWS built images as starting point for your image. This is my example:



FROM public.ecr.aws/lambda/nodejs:16

RUN yum install -y python3 && \
    pip3 install --upgrade pip requests

COPY . .

CMD ["lib/functions/index.handler"]

To find other AWS Lambda base images, check out: Amazon ECR Public Gallery > Lambda.

You could give it a go and build the image manually, but it's not required as the Serverless Framework will be doing that for us.

Publish the image to ECR

If you haven't used ECR before, don't worry. It's a docker-compatible image registry which is available in your AWS account which can host public and private images.

All you have to do is add this snippet to your serverless.yml:



provider:
  ecr:
    images:
      node16-python3-runtime:
        path: ./

Please note:

node16-python3-runtime is an arbitrary name I gave to my image, feel free to rename it to whatever is best in your use-case.
path is the path to the directory containing the Dockerfile

Once you run serverless deploy, the tool will build the image for you, and publish the image to AWS ECR. If successful, once you open ECR in your AWS console, it should look something like the following:

Instrument your Lambda to use the new custom runtime

This is the final part, but it requires attention because it's very different from how you would usually define your Lambda and its handler in serverless.yml.

First of all: make sure to remove the handler: property that you usually have.

Second, add the property image as follows:



functions:
  yourLambdaName:
    image:
      name: node16-python3-runtime
      command:
        - lib/functions/index.handler
      entryPoint:
        - '/lambda-entrypoint.sh'

Let me explain:

yourLambdaName should be renamed of course with the name of your own lambda
name: node16-python3-runtime this should match the name of the image you created before, in provider.ecr.image
command is the command executed when the docker container starts, it should point to the handler function
entryPoint keep is as it is, to allow the Lambda service to know what to do

Disclaimer: The cold start time will most likely increase using a custom runtime especially the one I've provided as example. In my case I've noticed 1.2 seconds which is much higher that what other Lambdas only using Node.js experience. However, because this is not user-facing or its latency matter, for me it's not a big deal. Take it into account if you are building APIs for example.

That should be it, I hope this guide is helpful to someone!

Hello Gitpod 🧡

Andrea Falzetti — Tue, 26 Apr 2022 15:28:35 +0000

I meant to write this blog post a month ago, but... life happened :)

I generally write technical articles on my blog rather than talking about my personal life, but 4 years ago, I wrote Hello DAZN, and I want to keep up with the tradition, so... Hello Gitpod 🧡

At the beginning of March this year, I've joined gitpod.io ✨

I've spent the last few years at DAZN, focusing on DevX (Developer Experience), empowering fellow engineers to get their job done smoothly and efficiently.

In 2018, I was already speaking about DevX.

But, it's thanks to Cirpo and the DAZN DX team that I realised how much I liked the subject.

When I heard about Gitpod, it was the same day GitHub announced Codespaces, and my colleague Lou announced he would join Gitpod.

Since that day, I've become a user, a customer, and a community member of Gitpod because I immediately saw the benefits of a cloud development environment. I started using it for day-to-day work.

Over time, I gitpodified most of my repositories.

In fact, I am writing this very blog post which requires Jekyll to run using Gitpod.

You might think I am exaggerating, and it's overkill to gitpodify my own blog...

Assuming you have already cloned the repo locally and installed the project dependencies, starting a new blog post with Jekyll requires you to locate the project on your machine, open your favourite editor, open the terminal, launch Jekyll, etc.

With Gitpod, all I have to do is press this green button :)

And in matter of seconds, I have VS Code open in the cloud, Jekyll up and running thanks to Gitpod automatically running for me these commands:

git clone ..

bundle install ⬇️

bundle exec Jekyll serve --watch --drafts

# > open VS Code

# > open browser with live preview

😎

Enough with selling Gitpod.

In conclusion, I want to say that I am incredibly enthusiastic to have the opportunity to impact the Developer Experience of thousands of developers and shape the future of Cloud Development Environments 🧡 ✨

Thanks 🙏👋

Deploying GFPGAN to AWS using Docker and Fargate

Andrea Falzetti — Tue, 18 Jan 2022 01:26:49 +0000

This post aims to help you deploy GFPGAN on AWS using ECS Fargate.

What is GFPGAN?!

I've recently discovered this fantastic Machine Learning project trained and designed to restore faces.

The project's headline is:

GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.

In short, given a blurred, damaged, or low-quality photo of a person's face, the algorithm it's able to restore the face and improve its quality.

Example

GFPGAN is a new AI model for photo restoration. Nothing was manually adjusted in this example. 😮👍 https://t.co/AwNRxw6bZ3 #ai #ml #ArtificialIntelligence #MachineLearning #photography pic.twitter.com/EYS5c4FiWd
— Metin Seven (@metinse7en) June 17, 2021

The challenge

So, I decided to write this blog post because I recently deployed this tool on AWS, and there were a few things that I wish I had known before starting.

GFPGAN is written in Python, and its entry point assumes you're going to convert a batch of images, reading and writing those from disk.

Now, I am not sure about your use case, but this was a significant limitation for me.

If you're intimidated by Python, don't worry. The entry point is relatively simple to modify. I didn't have any Python experience before this.

My use case is to provide an HTTP API to a front-end application to call and retrieve the enhanced image.

Architecture

To provide such API, I came up with this architecture that is currently working fine in my side-project.

To better understand the flow, this is a sequence diagram:

How to Use it

Before you start getting your hands dirty with Terraform or CloudFormation code, I recommend trying out the Docker image locally.

Step 1 - Fork the repo

At the moment of writing this, the official repository has an open PR for adding the Dockerfile, shout out to Mostafa Elmenbawy for his valuable contribution!

I also want to personally thank Mostafa for helping me understand how GFPGAN Python entry point works!

I recommend forking his repo until the PR has been merged.

Step 2 - Build the image

docker build -t gfpgan .

Step 3 - Run the container

docker run \
    --name gfpgan \
    --rm \
    --volume <PROJECT_ABSOLUTE_PATH>/inputs:/app/inputs \
    --volume <PROJECT_ABSOLUTE_PATH>/results:/app/results \
    gfpgan \
    python3 inference_gfpgan.py --upscale 2 --test_path inputs --save_root results

The command above will start the container that, by running GFPGAN, will transform the default inputs photos and produce outputs in the results folder.

How to Deploy It

Before proceeding, please note that I've adapted the Python entry point to integrate with SQS to receive jobs, S3 to persist the enhanced image files, and DynamoDB as a temporary state to inform the web-app about the status of the job.

Step 1 - Customize the entry point

I have changed inference_gfpgan.py as follows:

import boto3
import cv2
import glob
import numpy as np
import os
import json
import requests
import logging
import time
import uuid

from pythonjsonlogger import jsonlogger
from basicsr.utils import imwrite
from pathlib import Path
from gfpgan import GFPGANer

sqs = boto3.resource("sqs")
queue = sqs.get_queue_by_name(QueueName=os.environ["SQS_QUEUE_NAME"])
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(os.environ["DYNAMODB_TABLE_NAME"])
s3 = boto3.client("s3")

for name in ["boto", "urllib3", "s3transfer", "boto3", "botocore", "nose"]:
    logging.getLogger(name).setLevel(logging.CRITICAL)

def setup_logging(log_level):
    logger = logging.getLogger()

    logger.setLevel(log_level)
    json_handler = logging.StreamHandler()
    formatter = jsonlogger.JsonFormatter(
        fmt="%(asctime)s %(levelname)s %(name)s %(message)s"
    )
    json_handler.setFormatter(formatter)
    logger.addHandler(json_handler)

def main(logger, message):
    [f.unlink() for f in Path("./inputs").glob("*") if f.is_file()]
    [f.unlink() for f in Path("./results").glob("*") if f.is_file()]

    model_path = "experiments/pretrained_models/GFPGANCleanv1-NoCE-C2.pth"
    upscale = 2
    arch = "clean"
    channel = 2
    ext = "auto"
    suffix = None
    save_root = "results"
    os.makedirs(save_root, exist_ok=True)

    inputs_path = os.path.join("inputs", "input.jpg")

    img_data = requests.get(url=message["imageUrl"]).content

    logger.info(f"Downloading image {message['imageUrl']} to {inputs_path}")

    try:
        with open(inputs_path, "wb") as handler:
            handler.write(img_data)
    except:
        logger.error('Cannot download file')
        return

    restorer = GFPGANer(
        model_path=model_path,
        upscale=upscale,
        arch=arch,
        channel_multiplier=channel,
        bg_upsampler=None,
    )

    img_list = sorted(glob.glob(os.path.join("inputs", "*")))
    for img_path in img_list:
        # read image
        img_name = os.path.basename(img_path)
        logger.info(f"Processing {img_name} ...")
        basename, ext = os.path.splitext(img_name)
        input_img = cv2.imread(img_path, cv2.IMREAD_COLOR)

        # restore faces and background if necessary
        cropped_faces, restored_faces, restored_img = restorer.enhance(
            input_img, has_aligned=False, only_center_face=True, paste_back=True
        )

        # save faces
        for idx, (cropped_face, restored_face) in enumerate(
            zip(cropped_faces, restored_faces)
        ):
            # save cropped face
            save_crop_path = os.path.join(
                save_root, "cropped_faces", f"{basename}_{idx:02d}.png"
            )
            imwrite(cropped_face, save_crop_path)
            # save restored face
            if suffix is not None:
                save_face_name = f"{basename}_{idx:02d}_{suffix}.png"
            else:
                save_face_name = f"{basename}_{idx:02d}.png"
            save_restore_path = os.path.join(
                save_root, "restored_faces", save_face_name
            )
            imwrite(restored_face, save_restore_path)
            # save comparison image
            cmp_img = np.concatenate((cropped_face, restored_face), axis=1)
            imwrite(
                cmp_img, os.path.join(save_root, "cmp", f"{basename}_{idx:02d}.png")
            )

        if restored_img is not None:
            if ext == "auto":
                extension = ext[1:]
            else:
                extension = ext

            if suffix is not None:
                save_restore_path = os.path.join(
                    save_root, "restored_imgs", f"{basename}_{suffix}.{extension}"
                )
            else:
                save_restore_path = os.path.join(
                    save_root, "restored_imgs", f"{basename}.{extension}"
                )

            logger.info(f"Saving restored image to {save_restore_path} ...")

            imwrite(restored_img, save_restore_path)

    s3key = f'gfpgan_{message["orgId"]}_{uuid.uuid4().hex}.jpg'
    s3.upload_file(
        "results/restored_imgs/input..jpg", os.environ["S3_BUCKET_NAME"], s3key
    )
    new_record = message.copy()
    new_record["enhancedImageS3Key"] = s3key
    new_record["updatedAt"] = round(time.time())
    new_record["timeTaken"] = new_record["updatedAt"] - new_record["createdAt"]
    update_record(new_record)
    logger.info(f"Result uploaded in S3: {s3key}")


def create_record(message):
    response = table.put_item(
        Item={
            "enhancementId": message["enhancementId"],
            "imageUrl": message["imageUrl"],
            "createdAt": message["createdAt"],
        }
    )
    return response


def update_record(message):
    response = table.put_item(
        Item={
            "enhancementId": message["enhancementId"],
            "imageUrl": message["imageUrl"],
            "createdAt": message["createdAt"],
            # Additional Props
            "enhancedImageS3Key": message["enhancedImageS3Key"],
            "updatedAt": message["updatedAt"],
            "timeTaken": message["timeTaken"],
        }
    )
    return response

def retrieve_messages():
    setup_logging(logging.INFO)
    logger = logging.getLogger()
    logger.info("Starting retrieve_messages")
    while True:
        logger.debug("Fetching SQS messages")
        messages = queue.receive_messages(MaxNumberOfMessages=1, WaitTimeSeconds=1)
        for message in messages:
            parsed_message = json.loads(message.body)
            parsed_message["createdAt"] = round(time.time())
            logger.info(parsed_message)
            create_record(parsed_message)
            main(logger, parsed_message)
            message.delete()
        time.sleep(5)


if __name__ == "__main__":
    retrieve_messages()

Note: the code above pulls new messages from SQS every 5 seconds.

The SQS message should follow this schema:

{"enhancementId": "12345", "imageUrl": "https://example.com/photo.jpg"}

Step 2 - Update the Dockerfile

I've also updated the Dockerfile as follows. Again, shout out to Mostafa Elmenbawy for creating the initial Dockerfile! 🙏

FROM nvidia/cuda:10.0-cudnn7-devel

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends --fix-missing \
    # python
    python3.8 python3-pip python3-setuptools python3-dev \
    # OpenCV deps
    libglib2.0-0 libsm6 libxext6 libxrender1 libgl1-mesa-glx \
    # c++
    g++ \
    # others
    wget unzip

# Ninja
RUN wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip && \
    unzip ninja-linux.zip -d /usr/local/bin/ && \
    update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force

# basicsr facexlib
RUN python3 -m pip install --upgrade pip && \
    pip3 install --no-cache-dir torch>=1.7 opencv-python>=4.5 && \
    pip3 install --no-cache-dir basicsr facexlib realesrgan boto3 python-json-logger uuid

# weights
RUN wget https://github.com/TencentARC/GFPGAN/releases/download/v0.2.0/GFPGANCleanv1-NoCE-C2.pth \
        -P experiments/pretrained_models &&\
    wget https://github.com/TencentARC/GFPGAN/releases/download/v0.1.0/GFPGANv1.pth \
        -P experiments/pretrained_models && \
    wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth \
        -P /usr/local/lib/python3.6/dist-packages/realesrgan/weights/ && \
    wget https://github.com/xinntao/facexlib/releases/download/v0.1.0/detection_Resnet50_Final.pth \
        -P /usr/local/lib/python3.6/dist-packages/facexlib/weights/

RUN rm -rf /var/cache/apt/* /var/lib/apt/lists/* && \
    apt-get autoremove -y && apt-get clean

COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt

COPY . .
RUN pip3 install .

CMD ["python3", "inference_gfpgan.py"]

Step 3 - Publish the image to ECR

In your CI/CD, build the Dockerfile and publish the image to ECR.

Step 4 - Deploy the ECS task using Fargate

Using Terraform or CloudFormation, create and deploy the ECS task. Please note, this service doesn't need to be exposed to the internet via Route53 or an Application Load Balancer.

If you have deployed APIs or public-facing services before to ECS, you might be familiar with configuring the Target Group's healthcheck. The healthcheck detects if the main process running in the container is still health or the container needs to be restarted.

In this case, the container doesn't expose an HTTP endpoint for the Target Group to call, so we have to use container level healthchecks.

Instead of configuring the healthcheck in the Target Group, container-level healthchecks are defined in the ECS task definition, as follows:

...
    "healthCheck": {
      "command": [ "CMD-SHELL", "exit 0" ],
      "startPeriod": 20
    },
...

Please note, the CMD-SHELL can be any bash command. Ideally, the command would help you validate if the process is healthy.

By using exit 0, you assume that the container is always healthy.

This is not ideal, but I haven't found an alternative approach yet.

Takeaways

GFPGAN:

is a ML project able to restore damaged or low-quality photos
is written in Python but the interface is fairly easy to modify, even if you're not a Python Developer
can be containerized in Docker
can be deployed to ECS Fargate, I am using 2 GB o RAM and 1 CPU
you can easily integrate it with a queue system and integrate it with your app via an API

Conclusions

Hopefully, I've provided you with a real-world scenario in which GFPGAN is used and deployed to AWS.

I didn't include the entire Terraform code to keep the blog post readable, but I am happy to share more.

Just reach out to me if you need any help with this; I am more than happy to give you a hand deploying GFPGAN to AWS.

How to stream CloudWatch logs to Elasticsearch without custom code

Andrea Falzetti — Mon, 10 Jan 2022 18:45:47 +0000

Intro

In this post, I want to share the approach I have been using to ship logs from AWS CloudWatch to Elasticsearch without writing a single line of code.

My use case

In my side-project, I use a combination of Lambdas and Fargate tasks which both write JSON-formatted logs to CloudWatch.

I use CloudWatch to debug application errors while I use Elasticsearch for storing custom events such as User Actions (e.g. User X created resource Y).

By querying Elasticsearch, I can build up a view of all actions performed by a user within the platform.

I use the User Actions timeline view to monitor the platform's usage. I also expose this information to the user, so they have historical transactions of what and when they did something within the product.

I came up with a simple JSON schema that represents actions, and I instrumented my applications to spit out those events to CloudWatch when those occur.

I mentioned above my reason for requiring CloudWatch logs to be streamed to Elasticsearch. You might have a different one, such as backing up all logs to another system or post-process logs for security or any other possible reason.

My solution: Using Functionbeat

I wanted to achieve this objective ideally without writing custom code that later I had to maintain.

Assuming I wasn't the first one with this requirement, I looked around to see how people replicated logs over Elasticsearch. The most common pattern I've seen is having a Lambda function subscribed to the CloudWatch log group you are interested in.

The complexities of building your own Lambda would be writing the code that ingests, processes, and publishes and creating a flexible and automated mechanism to subscribe this Lambda to dozens or hundreds of log groups.

After some research, I stumbled upon Functionbeat, an open-source tool from Elastic.

"Function-beat", as the name suggests, is a pre-made Lambda function, written in Go, provided by Elastic, which you can deploy to your AWS account. Using its configuration file, you can define which CloudWatch Log Groups Functionbeat should subscribe to.

You can also define processing rules for excluding a record, dropping specific fields, or applying transformations to the records.

Essentially, an off-the-shelf solution that follows a community sanctioned approach without having to worry about writing it from scratch.

Let's now see some code and example configurations!

How to use Functionbeat

Follow this link for official installation instructions. In short, all I did was:

1 - Create a private repository

You will need to host the binary and configuration of Functionbeat. Also, you might want to set up a CI/CD pipeline to deploy updates on push quickly.

2 - Download Functionbeat

Download Functionbeat from the official Elastic website. Drop the binary in your brand new repository a

3 - Configure it

Create a functionbeat.yml by looking at the provided config example in functionbeat.reference.yml start populating your Elasticsearch endpoint and credentials.

I use Elastic cloud, but you can also stream to a self-hosted instance of Elasticsearch. Check this step for more details.

In this phase, you will also define to which CloudWatch Log Groups you want to subscribe your Functionbeat

This is an example config file:



# functionbeat.yml
functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
functionbeat.provider.aws.deploy_bucket: "${YOUR_AWS_ACCOUNT_ID}-functionbeat-deploy"
functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID}'
functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
functionbeat.provider.aws.functions:
  - name: functionbeat
    enabled: true
    type: cloudwatch_logs
    description: "Ships CloudWatch logs to Elasticsearch"
    triggers:
      - log_group_name: /aws/lambda/lambda-A
      - log_group_name: /aws/lambda/lambda-B
      - log_group_name: /aws/lambda/lambda-C

    processors:
      - drop_event:
          when:
            contains:
              message: "START RequestId:"
      - drop_event:
          when:
            contains:
              message: "REPORT RequestId:"
      - drop_event:
          when:
            contains:
              message: "END RequestId:"
      - drop_event:
          when:
            contains:
              message: "ELB-HealthChecker/2.0"

      - decode_json_fields:
          fields: ["message"]
          process_array: false
          max_depth: 1
          target: ""
          overwrite_keys: false

      - drop_fields:
          fields:
            - private
            - host.name
            - hostname
            - v
            - time
            - pid
            - id
            - message

cloud.id: "${ES_ID}"
cloud.auth: "elastic:${ES_PWD}"

#============================== Setup ILM =====================================

# Configure Index Lifecycle Management Index Lifecycle Management creates a
# write alias and adds additional settings to the template.
# The elasticsearch.output.index setting will be replaced with the write alias
# if ILM is enabled.

# Enabled ILM support. Valid values are true, false, and auto. The beat will
# detect availabilty of Index Lifecycle Management in Elasticsearch and enable
# or disable ILM support.
setup.ilm.enabled: auto

# Configure the ILM write alias name.
setup.ilm.rollover_alias: "functionbeat"

# Configure rollover index pattern.
setup.ilm.pattern: "{now/d}-000001"

4 - Deploy it

Functionbeat comes with a CloudFormation config which you can simply deploy to your AWS account.



./functionbeat setup -e
./functionbeat -v -e -d "*" deploy functionbeat

If the config doesn't contain any error, Functionbeat will subscribe to the specified CloudWatch Log Groups once deployed.

Now, it's time to generate some logs and see them replicated in Elasticsearch!

Troubleshooting

If you don't see the logs in Elasticsearch, you can debug the Functionbeat execution by looking at Lambda's logs. Most likely, you're facing a configuration issue, but Functionbeat generally will tell you what's wrong.

Considerations

✅ I've been successfully using this approach for over 12 months, and I am pretty satisfied.

✨ Since it was created, additional features were added, such as support for SQS and Kinesis.

🚩 Since I've started using it, Elastic has released a few newer releases, and the upgrade wasn't always a smooth experience.

Finally, if you decide to use Functionbeat, I would recommend writing a CloudWatch alarm to monitor its health.

If you end up using Functionbeat, drop me a message and let me know :)

How to configure ECS Fargate Scheduled Autoscaling

Andrea Falzetti — Wed, 25 Aug 2021 06:31:25 +0000

You are looking at configuring ECS Fargate scheduled autoscaling, and you're feeling a bit lost?

That's how I felt a few days ago. To set this up, I struggled for a couple of days. Unfortunately, the documentation and examples I found online didn't help me much, so once I figured it out, I decided to share a working code example using Terraform.

To be on the same page, in this post, I am assuming that you know what you are doing, and you are aware that there are different methods of autoscaling. We will look at configuring scheduled autoscaling which may not be the best choice for your project.

Let's start by defining scheduled autoscaling, what it means and my use case for it. I am also very curious to know your story and how you'll leverage this feature. If you found this helpful, let me know in the comments.

What is Scheduled Autoscaling?

Scheduled autoscaling means automatically increasing or decreasing the number of computing resources at a specific time of the day.

My use case for Scheduled Autoscaling

My main goal is cost optimisation. I recently ran out of the $10,000 credits provided by AWS Activate for Startups, and the highest cost on the bill was the ECS cluster. If you're interested in that, I am currently writing another blog post describing more in detail that story.

Code Example (using Terraform)

Finished talking, let's look at some Terraform code examples.

I'll introduce the resources that you need one by one.

01. Create an Autoscaling Target

If you want to configure autoscaling for ECS, the Autoscaling Target is your ECS Service, example:

# autoscaling.tf

resource "aws_appautoscaling_target" "service" {
  service_namespace  = "ecs"
  scalable_dimension = "ecs:service:DesiredCount"
  min_capacity       = var.scheduled_min_capacity # 0
  max_capacity       = var.scheduled_max_capacity # 1
  resource_id        = "service/${local.cluster_name}/${local.service_name}"
}

The code above assumes you already have your ECS service provisioned, like:

# ecs-service.tf

resource "aws_ecs_service" "service" {
  name    = local.service_name
  cluster = local.cluster_name
  [...omitted as not relevant...]
}

02. Define your Scheduled Actions

To scale in and out the service, you will need at least two scheduled actions, one to increase the number of desired tasks, the second one to decrease it.

# autoscaling.tf

...[aws_appautoscaling_target code above omitted]

resource "aws_appautoscaling_scheduled_action" "scale_service_out" {
  name               = "${local.service_name}-scale-out"
  service_namespace  = aws_appautoscaling_target.service.service_namespace
  resource_id        = aws_appautoscaling_target.service.resource_id
  scalable_dimension = aws_appautoscaling_target.service.scalable_dimension
  schedule           = "cron(0 6 * * ? *)"
  timezone           = "Europe/London"

  scalable_target_action {
    min_capacity = var.scheduled_max_capacity # 1
    max_capacity = var.scheduled_max_capacity # 1
  }
}

resource "aws_appautoscaling_scheduled_action" "scale_service_in" {
  name               = "${local.service_name}-scale-in"
  service_namespace  = aws_appautoscaling_target.service.service_namespace
  resource_id        = aws_appautoscaling_target.service.resource_id
  scalable_dimension = aws_appautoscaling_target.service.scalable_dimension
  schedule           = "cron(0 21 * * ? *)"
  timezone           = "Europe/London"

  scalable_target_action {
    min_capacity = var.scheduled_min_capacity # 0
    max_capacity = var.scheduled_min_capacity # 0
  }

  depends_on = [aws_appautoscaling_scheduled_action.scale_service_out]
}

In my case, for the dev cluster, I shut down all tasks for most of the day, while in prod I leave at least 1 and during office hours I have up to 3.

To do that, you will also need the following variables:

# vars.tf
variable "scheduled_min_capacity" {}
variable "scheduled_max_capacity" {}

For development:

# config/dev/terraform.tfvars
scheduled_min_capacity = 0
scheduled_max_capacity = 1

And prodution:

# config/prod/terraform.tfvars
scheduled_min_capacity = 1
scheduled_max_capacity = 3

That's it. Next, let's understand how to validate and test these changes.

Let's test it

Once you deploy those resources to verify that everything works, you can, of course, stare and refresh the ECS dashboard and see the number of ECS tasks increasing and decreasing. However, if you want a more efficient way, I'd recommend using the aws-cli, because apparently, this information is not available on AWS Console.

Run this command to list the existing scheduled actions:

aws application-autoscaling describe-scheduled-actions --service-namespace ecs

The result, should look like this:

{
    "ScheduledActions": [        
        {
            "ScheduledActionName": "<SERVICE-NAME>-scale-up",
            "ScheduledActionARN": "arn:aws:autoscaling:<AWS-REGION>:<AWS-ACCOUNT-ID>:scheduledAction:<ID>:resource/ecs/service/<CLUSTER-NAME>/<SERVICE-NAME>:scheduledActionName/<SERVICE-NAME>-scale-up",
            "ServiceNamespace": "ecs",
            "Schedule": "cron(0 6 * * ? *)",
            "ResourceId": "service/<CLUSTER-NAME>/<SERVICE-NAME>",
            "ScalableDimension": "ecs:service:DesiredCount",
            "ScalableTargetAction": {
                "MinCapacity": 1,
                "MaxCapacity": 1
            }
        },
        {
            "ScheduledActionName": "<SERVICE-NAME>-scale-down",
            "ScheduledActionARN": "arn:aws:autoscaling:<AWS-REGION>:<AWS-ACCOUNT-ID>:scheduledAction:<ID>:resource/ecs/service/<CLUSTER-NAME>/<SERVICE-NAME>:scheduledActionName/<SERVICE-NAME>-scale-down",
            "ServiceNamespace": "ecs",
            "Schedule": "cron(0 23 * * ? *)",
            "ResourceId": "service/<CLUSTER-NAME>/<SERVICE-NAME>",
            "ScalableDimension": "ecs:service:DesiredCount",
            "ScalableTargetAction": {
                "MinCapacity": 0,
                "MaxCapacity": 0
            }
        } 
    ]
}

You can also check recent autoscaling activities (although this is also available on the ECS service Events page);

aws application-autoscaling describe-scaling-activities --service-namespace ecs

output:

{
    "ScalingActivities": [
        {
            "ActivityId": "a7edfe9b-b924-4468-a129-736ce0a06078",
            "ServiceNamespace": "ecs",
            "ResourceId": "service/<CLUSTER-NAME>/<SERVICE-NAME>",
            "ScalableDimension": "ecs:service:DesiredCount",
            "Description": "Setting desired count to 1.",
            "Cause": "minimum capacity was set to 1",
            "StartTime": "2021-08-24T06:00:16.520000+01:00",
            "EndTime": "2021-08-24T06:00:47.551000+01:00",
            "StatusCode": "Successful",
            "StatusMessage": "Successfully set desired count to 1. Change successfully fulfilled by ecs."
        }
]

Conclusion

Scheduled autoscaling is a simple way to introduce autoscaling in your ECS cluster.

I find the concept of saving and optimising resources exciting. The idea of turning off the machine when we don't need it sounds the right thing to do.

I wouldn't necessarily recommend using scheduled autoscaling to solve scalability challenges. For example, if you have unpredictable workloads, you might find Tracking Autoscaling more suited.

I hope you found this helpful. 🙏

Sources

I used the following docs/blogs to learn how to configure scheduled autoscaling, and you might find them helpful too: