DEV Community: Kodsama

AWS cost last resort, the killer lambda

Kodsama — Tue, 02 Jul 2024 10:37:38 +0000

Having full control over your Amazon Web Services (AWS) costs isn't that easy.

This is the continuation from my previous article Controlling AWS Lambda Costs.

Now that you have set up alerts and limit possible overflows it is time to come to the last resort, starting to kill things.

Of course, one could always manually tear down services when the budget is going too high, but many of us would rather spend hours once to automate this instead by creating a killer lambda. 😄

Here are some steps I would recommend

Setting up Billing Alarms
Using a killer lambda
- General setup
- Killer lambda
- Recovery lambda
Closing thoughts

Setting up Billing Alarms

The first and foremost thing to do is to set up budget alerts for the whole AWS cost.

Billing Alarms are the essential tool for monitoring your AWS costs. It is an easy way to get notified if your monthly AWS bill is estimated to cross a set threshold.

The official documentation is great.

Create a Budget:
- Go to the AWS Management Console.
- Navigate to the AWS Budgets dashboard.
- Click on "Create a budget."
- Follow the steps to create a budget. Set the budget amount to your desired limit.
Configure Alerts:
- Set up alert notifications for when your budget threshold is reached. You can choose to receive alerts via email or SNS (Simple Notification Service).
- To use SNS, create an SNS topic if you don't have one, and add subscribers to the topic (e.g., your email address).

Now, when you reach the budget (or expected budget), you will get an email or be able to trigger things with the alert email using SNS. 😉

Using a killer lambda

To prevent AWS API Gateway and AWS Lambda from being invoked when a specific budget is reached, we will combine multiple AWS tools:

Billing Alerts: to send an SNS notification when the budget is reached
Simple Notification Service (SNS): to handle the notification and trigger a lambda
Lambda: to do the work of killing/restoring permissions and parameters
EventBridge Scheduler: to schedule the recovery lambda
DynamoDB: to store the parameters which will be changed

The idea is this:

The Billing Alert will send a message on SNS (see Setting up Billing Alarms).
The SNS message will trigger a killer lambda.
The killer lambda will store current parameters of each lambda and EC2 instances in a DynamoDB table to restore them later.
The killer lambda will:
- Limit the API Gateway rate
- Change lambda parameters to prevent their invocation (be careful to not kill the recovery lambda!)
- Stop EC2 instances
At the beginning of each billing cycle, the recovery lambda will be triggered (via EventBridge Scheduler).
The recovery lambda will read DynamoDB and restore the lambda parameters.

NOTE: We need to prevent the killer lambda from killing the recovery lambda. For this, we need to first create the recovery lambda to get its name in the whitelist and avoid it being killed.

General setup

Step 1: Set Up AWS Budgets and send alert to SNS

See Setting up Billing Alarms

Step 2: Create the DynamoDB database

We need to ensure that the DynamoDB has the following attributes:

Table Name: LambdaAndApiSettings
Primary Key / Partition Key: ResourceID (String)

You can create the table using the AWS Management Console:

Go to the DynamoDB section.
Click on "Create table".
Set the table name to LambdaAndApiSettings.
Add a primary key/Sort key with the name FunctionName and type String.
Click "Create".

Step 3: (after creating lambdas) Give proper permissions in IAM

Ensure that the IAM role associated with your Lambda function has the necessary permissions. You need to attach a policy to the role that allows access to DynamoDB and Lambda APIs.

Required Permissions:

DynamoDB: AmazonDynamoDBFullAccess
Lambda: AWSLambda_FullAccess
EC2 instances: AmazonEC2FullAccess
API Gateway: AmazonAPIGatewayAdministrator

Recovery lambda

To re-enable the services, you can set up a scheduled Lambda function that runs at the beginning of each budget period (e.g., monthly). This function will reset the throttling limits on the API Gateway stages and re-enable the Lambda functions.

Step 1: Create the Lambda Function

In the AWS Lambda console, create a new Lambda function.
Set the Lambda type to Python.
Use the following Python code:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Author: Alexandre Martins (a.k.a Kodsama)
"""
This Lambda function restores the previous state of API Gateway stages, specified Lambda functions, 
and EC2 instances using settings saved in DynamoDB. It retrieves the saved settings from DynamoDB 
and re-applies them to the respective resources. The settings were initially saved by the disabling 
Lambda to ensure the ability to restore the original state.
"""

import boto3
import os
import logging
import json
from botocore.client import BaseClient
from botocore.exceptions import ClientError

# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Initialize clients
api_client = boto3.client("apigateway")
lambda_client = boto3.client("lambda")
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("LambdaAndApiSettings")


def lambda_handler(event: dict, context: object):
    """
    The main handler function for the Lambda function. It restores the previous state of API Gateway stages,
    specified Lambda functions using settings saved in DynamoDB.

    Args:
        event (dict): The event data that triggered the Lambda function [UNUSED].
        context (object): The context object containing metadata about the Lambda invocation.

    Returns:
        dict: A dictionary containing the status code and message.
    """
    try:
        # Get environment variables
        rest_api_id = os.environ.get("API_GATEWAY_ID")
        self_name = context.function_name  # Get the name of the current Lambda function
        simulate = os.environ.get("SIMULATE", "false").lower() == "true"
        # Lambda
        restore_all_lambdas = os.environ.get("RESTORE_ALL_LAMBDAS", "false").lower() == "true"
        lambdas_restore_list = os.environ.get("LAMBDAS_TO_RESTORE", "").split(",")
        lambdas_restore_list = [] if lambdas_restore_list == [""] else lambdas_restore_list
        lambdas_blacklist = os.environ.get("BLACKLISTED_LAMBDAS", "").split(",")
        lambdas_blacklist = [] if lambdas_blacklist == [""] else lambdas_blacklist

        # Process API Gateway stages
        if rest_api_id:
            restore_api_gateway_stages(rest_api_id, api_client, simulate)

        # Process Lambda functions
        if restore_all_lambdas:
            restore_all_lambdas_function(lambda_client, self_name, lambdas_blacklist, simulate)
        elif lambdas_restore_list:
            restore_specified_lambdas(lambda_client, lambdas_restore_list, self_name, lambdas_blacklist, simulate)
        else:
            logger.warn("No Lambda functions specified for restoration. Skipping restoring Lambda functions.")

        logger.info(
            "All done! API Gateway stages, specified Lambda functions have been restored successfully."
        )
        return {"statusCode": 200, "body": "API Gateway, Lambda functions, and EC2 instances restoration complete"}
    except Exception as e:
        logger.error(f"Error occurred: {str(e)}", exc_info=True)
        return {
            "statusCode": 500,
            "body": "Error occurred while restoring API Gateway, Lambda functions, and EC2 instances",
        }


def restore_api_gateway_stages(api_client: BaseClient, rest_api_id: str, simulate: bool):
    """
    Restore the stages of an API Gateway using settings saved in DynamoDB.

    Args:
        api_client (BaseClient): The API Gateway client.
        rest_api_id (str): The ID of the API Gateway.
        simulate (bool): Whether to simulate restoring the API Gateway stages.

    Raises:
        Exception: If an error occurs while restoring the stages.
    """
    assert isinstance(api_client, BaseClient), f"api_client must be a boto3 client instance, not {type(api_client)}"
    assert (
        isinstance(rest_api_id, str) and rest_api_id
    ), f"rest_api_id must be a non-empty string, not {type(rest_api_id)}"
    assert isinstance(simulate, bool), f"simulate must be a boolean, not {type(simulate)}"

    if not rest_api_id:
        logger.info(f"No API Gateway ID provided. Skipping.")
        return

    try:
        logger.info(f"Restoring stages for API Gateway with ID: {rest_api_id}")
        stages = api_client.get_stages(restApiId=rest_api_id)
        for stage in stages["item"]:
            stage_name = stage["stageName"]
            logger.info(f"Restoring stage: {stage_name}")

            # Retrieve saved API Gateway stage settings from DynamoDB
            stage_settings = get_saved_settings(f"api-{rest_api_id}-{stage_name}")
            if not stage_settings:
                stage_settings = {
                    "burstLimit": os.environ.get("DEFAULT_BURST_LIMIT", "1000"),
                    "rateLimit": os.environ.get("DEFAULT_RATE_LIMIT", "500"),
                }

            if simulate:
                logger.info(f"Simulation mode: API Gateway stage {stage_name} would be restored.")
                continue

            # Restore API Gateway stage
            api_client.update_stage(
                restApiId=rest_api_id,
                stageName=stage_name,
                patchOperations=[
                    {
                        "op": "replace",
                        "path": "/*/*/throttling/burstLimit",
                        "value": str(stage_settings["burstLimit"]),
                    },
                    {"op": "replace", "path": "/*/*/throttling/rateLimit", "value": str(stage_settings["rateLimit"])},
                ],
            )
            logger.info(f"Stage {stage_name} restored.")
    except api_client.exceptions.ClientError as e:
        logger.error(f"Failed to restore API Gateway stages: {e}")
        if e.response["Error"]["Code"] == "AccessDeniedException":
            logger.error(
                "Access denied. Ensure the IAM role has the following permissions: AmazonAPIGatewayAdministrator."
            )
        raise


def restore_all_lambdas_function(
    lambda_client: BaseClient,
    self_name: str,
    lambdas_blacklist: list,
    simulate: bool = True,
):
    """
    Restore all Lambda functions using settings saved in DynamoDB.

    Args:
        lambda_client (BaseClient): The Lambda client.
        self_name (str): The name of the current Lambda function.
        lambdas_blacklist (list): A list of blacklisted Lambda function names.
        simulate (bool): Whether to simulate restoring the Lambda functions.

    Raises:
        Exception: If an error occurs while restoring the Lambda functions.
    """
    assert isinstance(
        lambda_client, BaseClient
    ), f"lambda_client must be a boto3 client instance, not {type(lambda_client)}"
    assert isinstance(self_name, str) and self_name, f"self_name must be a non-empty string, not {type(self_name)}"
    assert isinstance(simulate, bool), f"simulate must be a boolean, not {type(simulate)}"

    logger.info("Restoring all Lambda functions in the account.")

    paginator = lambda_client.get_paginator("list_functions")
    all_lambdas_restore_list = []
    for page in paginator.paginate():
        for function in page["Functions"]:
            all_lambdas_restore_list.append(function["FunctionName"])

    restore_specified_lambdas(lambda_client, all_lambdas_restore_list, self_name, lambdas_blacklist, simulate)


def restore_specified_lambdas(
    lambda_client: BaseClient,
    lambdas_restore_list: list,
    self_name: str,
    lambdas_blacklist: list,
    simulate: bool = True,
):
    assert isinstance(lambda_client, BaseClient), f"lambda_client must be a boto3 client instance, not {type(lambda_client)}"
    assert isinstance(lambdas_restore_list, list), f"lambdas_restore_list must be a list, not {type(lambdas_restore_list)}"
    assert isinstance(self_name, str) and self_name, f"self_name must be a non-empty string, not {type(self_name)}"
    assert isinstance(lambdas_blacklist, list), f"lambdas_blacklist must be a list, not {type(lambdas_blacklist)}"
    assert isinstance(simulate, bool), f"simulate must be a boolean, not {type(simulate)}"

    logger.info(f"Restore Lambdas: {lambdas_restore_list}")

    if not lambdas_restore_list:
        logger.warn(f"Attempting to restore an empty list of Lambdas")
        return

    for function_name in lambdas_restore_list:
        if function_name == self_name or function_name in lambdas_blacklist:
            logger.info(f"Skipping blacklisted or current Lambda function: {function_name}")
            continue

        try:
            saved_settings = get_saved_settings(function_name)
            if not saved_settings:
                saved_settings = {"Concurrency": os.environ.get("DEFAULT_CONCURRENCY", None), "Permissions": []}

            if simulate:
                logger.info(f"Simulation mode: Lambda function {function_name} would be restored.")
                continue

            if saved_settings["Concurrency"] is not None:
                lambda_client.put_function_concurrency(
                    FunctionName=function_name, ReservedConcurrentExecutions=saved_settings["Concurrency"]
                )
                logger.info(f"Restored concurrency for Lambda function: {function_name}")

            for permission in saved_settings["Permissions"]:
                principal = permission["Principal"]
                if isinstance(principal, dict) and "Service" in principal:
                    principal = principal["Service"]

                try:
                    lambda_client.add_permission(
                        FunctionName=function_name,
                        StatementId=permission["Sid"],
                        Action=permission["Action"],
                        Principal=principal,
                    )
                    logger.info(f"Restored permission {permission['Sid']} for Lambda function: {function_name}")
                except ClientError as e:
                    if e.response['Error']['Code'] == 'ResourceConflictException':
                        logger.warning(f"Permission with StatementId {permission['Sid']} already exists for Lambda function: {function_name}. Skipping...")
                    else:
                        raise e

        except ClientError as e:
            logger.error(f"Failed to restore Lambda function {function_name}: {e}")
            if e.response["Error"]["Code"] == "AccessDeniedException":
                logger.error(
                    f"Access denied. Ensure the IAM role has the following permissions: lambda:PutFunctionConcurrency, lambda:AddPermission."
                )
            raise


def get_saved_settings(resource_id: str) -> dict:
    """
    Retrieve settings from DynamoDB. If no data is found, return the default settings.

    Args:
        resource_id (str): The unique identifier for the resource.

    Returns:
        dict: The retrieved settings or empty dict if non existing.
    """
    try:
        response = table.get_item(Key={"ResourceID": resource_id})
        if "Item" in response:
            logger.debug(f"Found settings for {resource_id}")
            return response["Item"]["Settings"]
        else:
            logger.warn(f"No saved settings found for {resource_id}, using default settings.")
            return {}
    except dynamodb.meta.client.exceptions.ClientError as e:
        logger.error(f"Failed to retrieve settings from DynamoDB for {resource_id}: {e}")
        if e.response["Error"]["Code"] == "AccessDeniedException":
            logger.error("Access denied. Ensure the IAM role has the following permissions: AmazonDynamoDBFullAccess.")
        raise
    except Exception as e:
        logger.error(f"Error retrieving saved data for {resource_id}: {e}")
        raise

Environment Variables

API_GATEWAY_ID: The ID of the API Gateway whose stages are to be restored.
SIMULATE: When set to true, the lambda will simulate actions without making actual changes.
RESTORE_ALL_LAMBDAS: When set to true, all Lambda functions will be restored, except those in the blacklist.
LAMBDAS_TO_RESTORE: A comma-separated list of specific Lambda function names to restore.
BLACKLISTED_LAMBDAS: A comma-separated list of Lambda function names that should not be restored.
RESTORE_ALL_EC2_INSTANCES: When set to true, all EC2 instances will be restored, except those in the blacklist.
EC2_TO_RESTORE: A comma-separated list of specific EC2 instance IDs to restore.
BLACKLISTED_EC2_INSTANCES: A comma-separated list of EC2 instance IDs that should not be restored.
DEFAULT_BURST_LIMIT: The default burst limit for API Gateway throttling, used if no data is found in DynamoDB.
DEFAULT_RATE_LIMIT: The default rate limit for API Gateway throttling, used if no data is found in DynamoDB.
DEFAULT_CONCURRENCY: The default Lambda concurrency value, used if no data is found in DynamoDB.

Step 2: Give the Lambda function full access in IAM

See General setup

Step 3: Create a CloudWatch Event Rule

Go to the EventBridge Scheduler.
Navigate to "EventBridge Schedule".
Click "Create Schedule".
Choose a schedule name and description.
Set to recurring schedule and CRON based schedule. 6.

If your budget period is monthly, you can use a cron expression like cron(0 0 1 * ? *) to run the function at midnight on the first day of each month.

Select no flexible time window and click Next.
Click on Invoke an AWS Lambda and select the recover lambda as the Target.
Ensure the Lambda function has the necessary permissions to be invoked by the CloudWatch Events rule. Click on Next.
Review the schedule and create it!

Killer lambda

Step 1: Create a Lambda Function

In the AWS Lambda console, create a new Lambda function.
Set the Lambda type to Python.
Use the following Python code:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Author: Alexandre Martins (a.k.a Kodsama)
"""
This Lambda disables API Gateway stages, specified Lambda functions, and stops EC2 instances, saving their settings to DynamoDB. 
It processes environment variables to determine which resources to disable and whether to simulate the action. 
The settings of the resources are saved to DynamoDB before any changes are made, ensuring that current states are preserved.
"""

import boto3
import os
import logging
import json
from botocore.client import BaseClient

# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Initialize clients
api_client = boto3.client("apigateway")
lambda_client = boto3.client("lambda")
ec2_client = boto3.client("ec2")
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("LambdaAndApiSettings")


def lambda_handler(event: dict, context: object):
    """
    The main handler function for the Lambda function. It disables API Gateway stages, specified
    Lambda functions, and stops EC2 instances, saving their settings to DynamoDB.

    Args:
        event (dict): The event data that triggered the Lambda function [UNUSED].
        context (object): The context object containing metadata about the Lambda invocation.

    Returns:
        dict: A dictionary containing the status code and message.
    """
    try:
        # Get environment variables
        rest_api_id = os.environ.get("API_GATEWAY_ID")
        self_name = context.function_name  # Get the name of the current Lambda function
        simulate = os.environ.get("SIMULATE", "false").lower() == "true"
        # Lambda
        lambdas_disable_all = os.environ.get("DISABLE_ALL_LAMBDAS", "false").lower() == "true"
        lambdas_disable_list = os.environ.get("LAMBDAS_TO_DISABLE", "").split(",")
        lambdas_disable_blacklist = os.environ.get("LAMBDAS_TO_NOT_DISABLE", "").split(",")
        lambdas_disable_list = [] if lambdas_disable_list == [""] else lambdas_disable_list
        lambdas_disable_blacklist = [] if lambdas_disable_blacklist == [""] else lambdas_disable_blacklist
        # EC2
        ec2_disable_all = os.environ.get("DISABLE_ALL_EC2_INSTANCES", "false").lower() == "true"
        ec2_disable_list = os.environ.get("EC2_TO_DISABLE", "").split(",")
        ec2_disable_blacklist = os.environ.get("EC2_TO_NOT_DISABLE", "").split(",")
        ec2_disable_list = [] if ec2_disable_list == [""] else ec2_disable_list
        ec2_disable_blacklist = [] if ec2_disable_blacklist == [""] else ec2_disable_blacklist

        # Process API Gateway stages
        if rest_api_id:
            disable_api_gateway_stages(rest_api_id, api_client, simulate)

        # Process Lambda functions
        if lambdas_disable_all:
            disable_all_lambdas(lambda_client, self_name, lambdas_disable_blacklist, simulate)
        elif lambdas_disable_list:
            disable_specified_lambdas(
                lambda_client, lambdas_disable_list, self_name, lambdas_disable_blacklist, simulate
            )
        else:
            logger.warn(
                "No Lambda functions specified and DISABLE_ALL_LAMBDAS is not true. Skipping disabling Lambda functions."
            )

        # Process EC2 instances
        if ec2_disable_all:
            disable_all_ec2(ec2_client, simulate)
        elif ec2_disable_list:
            disable_specified_ec2(ec2_client, ec2_disable_list, ec2_disable_blacklist, simulate)
        else:
            logger.warn(
                "No EC2 instances specified and DISABLE_ALL_EC2_INSTANCES is not true. Skipping disabling EC2 instances."
            )

        logger.info(
            "All done! API Gateway stages, specified Lambda functions, and EC2 instances have been processed successfully."
        )
        return {"statusCode": 200, "body": "API Gateway, Lambda functions, and EC2 instances processing complete"}
    except Exception as e:
        logger.error(f"Error occurred: {str(e)}", exc_info=True)
        return {
            "statusCode": 500,
            "body": "Error occurred while processing API Gateway, Lambda functions, and EC2 instances",
        }


def disable_api_gateway_stages(api_client: BaseClient, rest_api_id: str, simulate: bool):
    """
    Disable the stages of an API Gateway and save their current settings to DynamoDB.

    Args:
        api_client (BaseClient): The API Gateway client.
        rest_api_id (str): The ID of the API Gateway.
        simulate (bool): Whether to simulate disabling the API Gateway stages.

    Raises:
        Exception: If an error occurs while disabling the stages.
    """
    assert isinstance(api_client, BaseClient), f"api_client must be a boto3 client instance, not {type(api_client)}"
    assert (
        isinstance(rest_api_id, str) and rest_api_id
    ), f"rest_api_id must be a non-empty string, not {type(rest_api_id)}"
    assert isinstance(simulate, bool), f"simulate must be a boolean, not {type(simulate)}"

    if not rest_api_id:
        logger.info(f"No API Gateway ID provided. Skipping.")
        return

    try:
        logger.info(f"Disabling stages for API Gateway with ID: {rest_api_id}")
        stages = api_client.get_stages(restApiId=rest_api_id)
        for stage in stages["item"]:
            stage_name = stage["stageName"]
            logger.info(f"Disabling stage: {stage_name}")

            # Save current API Gateway stage settings to DynamoDB
            stage_settings = {
                "burstLimit": stage.get("methodSettings", {}).get("/*/*/throttling/burstLimit", "default"),
                "rateLimit": stage.get("methodSettings", {}).get("/*/*/throttling/rateLimit", "default"),
            }
            save_to_dynamodb(f"api-{rest_api_id}-{stage_name}", stage_settings)

            if simulate:
                logger.info(f"Simulation mode: API Gateway stage {stage_name} would be disabled.")
                continue

            # Disable API Gateway stage
            api_client.update_stage(
                restApiId=rest_api_id,
                stageName=stage_name,
                patchOperations=[
                    {"op": "replace", "path": "/*/*/throttling/burstLimit", "value": "0"},
                    {"op": "replace", "path": "/*/*/throttling/rateLimit", "value": "0"},
                ],
            )
            logger.info(f"Stage {stage_name} disabled.")
    except api_client.exceptions.ClientError as e:
        logger.error(f"Failed to disable API Gateway stages: {e}")
        if e.response["Error"]["Code"] == "AccessDeniedException":
            logger.error(
                "Access denied. Ensure the IAM role has the following permissions: AmazonAPIGatewayAdministrator."
            )
        raise


def disable_all_lambdas(
    lambda_client: BaseClient, self_name: str, lambdas_disable_blacklist: list, simulate: bool = True
):
    """
    Disable all Lambda functions except the current one and blacklisted ones, and save their settings to DynamoDB.

    Args:
        lambda_client (BaseClient): The Lambda client.
        self_name (str): The name of the current Lambda function.
        lambdas_disable_blacklist (list): A list of blacklisted Lambda function names.
        simulate (bool): Whether to simulate disabling the Lambda functions.

    Raises:
        Exception: If an error occurs while disabling the Lambda functions.
    """
    assert isinstance(
        lambda_client, BaseClient
    ), f"lambda_client must be a boto3 client instance, not {type(lambda_client)}"
    assert isinstance(self_name, str) and self_name, f"self_name must be a non-empty string, not {type(self_name)}"
    assert isinstance(
        lambdas_disable_blacklist, list
    ), f"lambdas_disable_blacklist must be a list, not {type(lambdas_disable_blacklist)}"
    assert isinstance(simulate, bool), f"simulate must be a boolean, not {type(simulate)}"

    logger.info("Disabling all Lambda functions in the account.")

    paginator = lambda_client.get_paginator("list_functions")
    all_lambdas_disable_list = []
    for page in paginator.paginate():
        for function in page["Functions"]:
            all_lambdas_disable_list.append(function["FunctionName"])

    disable_specified_lambdas(lambda_client, all_lambdas_disable_list, self_name, lambdas_disable_blacklist, simulate)


def disable_specified_lambdas(
    lambda_client: BaseClient,
    lambdas_disable_list: list,
    self_name: str,
    lambdas_disable_blacklist: list,
    simulate: bool = True,
):
    """
    Disable specified Lambda functions except the current one and blacklisted ones, and save their settings to DynamoDB.

    Args:
        lambda_client (BaseClient): The Lambda client.
        lambdas_disable_list (list): A list of Lambda function names.
        self_name (str): The name of the current Lambda function.
        lambdas_disable_blacklist (list): A list of blacklisted Lambda function names.
        simulate (bool): Whether to simulate disabling the Lambda functions.

    Raises:
        Exception: If an error occurs while disabling the Lambda functions.
    """
    assert isinstance(
        lambda_client, BaseClient
    ), f"lambda_client must be a boto3 client instance, not {type(lambda_client)}"
    assert isinstance(
        lambdas_disable_list, list
    ), f"lambdas_disable_list must be a list, not {type(lambdas_disable_list)}"
    assert isinstance(self_name, str) and self_name, f"self_name must be a non-empty string, not {type(self_name)}"
    assert isinstance(
        lambdas_disable_blacklist, list
    ), f"lambdas_disable_blacklist must be a list, not {type(lambdas_disable_blacklist)}"
    assert isinstance(simulate, bool), f"simulate must be a boolean, not {type(simulate)}"

    logger.info(f"Disable Lambdas: {lambdas_disable_list}")

    if not lambdas_disable_list:
        logger.warn(f"Attempting to disable an empty list of Lambdas")
        return

    for function_name in lambdas_disable_list:
        if function_name == self_name or function_name in lambdas_disable_blacklist:
            logger.info(f"Skipping blacklisted or current Lambda function: {function_name}")
            continue

        try:
            # Save current Lambda settings to DynamoDB
            save_lambda_settings(lambda_client, function_name)

            if simulate:
                logger.info(f"Simulation mode: Lambda function {function_name} would be disabled.")
                continue

            # Disable concurrency
            logger.info(f"Disabling Lambda function: {function_name}")
            lambda_client.put_function_concurrency(FunctionName=function_name, ReservedConcurrentExecutions=0)
            logger.info(f"Lambda function {function_name} disabled.")

        except lambda_client.exceptions.ClientError as e:
            logger.error(f"Failed to disable Lambda function {function_name}: {e}")
            if e.response["Error"]["Code"] == "AccessDeniedException":
                logger.error(
                    f"Access denied. Ensure the IAM role has the following permissions: lambda:PutFunctionConcurrency."
                )
            raise


def disable_all_ec2(ec2_client: BaseClient, simulate: bool = True):
    """
    Stop all EC2 instances and save their settings to DynamoDB.

    Args:
        ec2_client (BaseClient): The EC2 client.
        simulate (bool): Whether to simulate stopping the EC2 instances.

    Raises:
        Exception: If an error occurs while stopping the EC2 instances.
    """
    assert isinstance(ec2_client, BaseClient), f"ec2_client must be a boto3 client instance, not {type(ec2_client)}"
    assert isinstance(simulate, bool), f"simulate must be a boolean, not {type(simulate)}"

    try:
        logger.info("Stopping all EC2 instances in the account.")
        response = ec2_client.describe_instances()
        instances = [
            instance["InstanceId"] for reservation in response["Reservations"] for instance in reservation["Instances"]
        ]

        disable_specified_ec2(ec2_client, instances, [], simulate)

    except ec2_client.exceptions.ClientError as e:
        logger.error(f"Failed to stop EC2 instances: {e}")
        if e.response["Error"]["Code"] == "AccessDeniedException":
            logger.error("Access denied. Ensure the IAM role has the following permissions: AmazonEC2FullAccess.")
        raise


def disable_specified_ec2(
    ec2_client: BaseClient, ec2_disable_list: list, ec2_disable_blacklist: list, simulate: bool = True
):
    """
    Stop specified EC2 instances except blacklisted ones, and save their settings to DynamoDB.

    Args:
        ec2_client (BaseClient): The EC2 client.
        ec2_disable_list (list): A list of EC2 instance IDs to stop.
        ec2_disable_blacklist (list): A list of blacklisted EC2 instance IDs.
        simulate (bool): Whether to simulate stopping the EC2 instances.

    Raises:
        Exception: If an error occurs while stopping the EC2 instances.
    """
    assert isinstance(ec2_client, BaseClient), f"ec2_client must be a boto3 client instance, not {type(ec2_client)}"
    assert isinstance(ec2_disable_list, list), f"ec2_disable_list must be a list, not {type(ec2_disable_list)}"
    assert isinstance(
        ec2_disable_blacklist, list
    ), f"ec2_disable_blacklist must be a list, not {type(ec2_disable_blacklist)}"
    assert isinstance(simulate, bool), f"simulate must be a boolean, not {type(simulate)}"

    logger.info(f"Disable EC2 instances: {ec2_disable_list} (exempt: {ec2_disable_blacklist})")

    for instance_id in ec2_disable_list:
        if instance_id in ec2_disable_blacklist:
            logger.info(f"Skipping blacklisted EC2 instance: {instance_id}")
            continue

        try:
            # Save current EC2 settings to DynamoDB
            save_ec2_settings(ec2_client, instance_id)

            if simulate:
                logger.info(f"Simulation mode: EC2 instance {instance_id} would be stopped.")
                continue

            # Stop the EC2 instance
            logger.info(f"Stopping EC2 instance: {instance_id}")
            ec2_client.stop_instances(InstanceIds=[instance_id])
            logger.info(f"EC2 instance {instance_id} stopped.")

        except ec2_client.exceptions.ClientError as e:
            logger.error(f"Failed to stop EC2 instance {instance_id}: {e}")
            if e.response["Error"]["Code"] == "AccessDeniedException":
                logger.error("Access denied. Ensure the IAM role has the following permissions: AmazonEC2FullAccess.")
            raise


def save_lambda_settings(lambda_client: BaseClient, function_name: str):
    """
    Save the current settings of a Lambda function to DynamoDB.

    Args:
        lambda_client (BaseClient): The Lambda client.
        function_name (str): The name of the Lambda function.

    Raises:
        Exception: If an error occurs while retrieving or saving the settings.
    """
    logger.info(f"Saving settings for Lambda '{function_name}'")
    assert isinstance(
        lambda_client, BaseClient
    ), f"lambda_client must be a boto3 client instance, not {type(lambda_client)}"
    assert (
        isinstance(function_name, str) and function_name
    ), f"function_name must be a non-empty string, not {type(function_name)}"

    try:
        # Get current concurrency setting
        concurrency_response = lambda_client.get_function_concurrency(FunctionName=function_name)
        concurrency = concurrency_response.get("ReservedConcurrentExecutions", None)

        # Get current permissions
        permissions_response = lambda_client.get_policy(FunctionName=function_name)
        permissions = json.loads(permissions_response["Policy"])["Statement"]

        # Save settings to DynamoDB
        save_to_dynamodb(function_name, {"Concurrency": concurrency, "Permissions": permissions})
    except lambda_client.exceptions.ResourceNotFoundException:
        logger.info(f"No existing settings for Lambda function: {function_name}")
    except lambda_client.exceptions.ClientError as e:
        logger.error(f"Failed to retrieve settings for Lambda function {function_name}: {e}")
        if e.response["Error"]["Code"] == "AccessDeniedException":
            logger.error("Access denied. Ensure the IAM role has the following permission: AWSLambda_FullAccess.")
        raise


def save_ec2_settings(ec2_client: BaseClient, instance_id: str):
    """
    Save the current settings of an EC2 instance to DynamoDB.

    Args:
        ec2_client (BaseClient): The EC2 client.
        instance_id (str): The ID of the EC2 instance.

    Raises:
        Exception: If an error occurs while retrieving or saving the settings.
    """
    logger.info(f"Saving settings for EC2 instance {instance_id}")
    assert isinstance(ec2_client, BaseClient), f"ec2_client must be a boto3 client instance, not {type(ec2_client)}"
    assert (
        isinstance(instance_id, str) and instance_id
    ), f"instance_id must be a non-empty string, not {type(instance_id)}"

    try:
        # Get current instance details
        response = ec2_client.describe_instances(InstanceIds=[instance_id])
        instance = response["Reservations"][0]["Instances"][0]
        instance_settings = {
            "InstanceType": instance.get("InstanceType"),
            "KeyName": instance.get("KeyName"),
            "State": instance.get("State", {}).get("Name"),
            "PreviousState": instance.get("State", {}).get("Name"),
        }

        # Save settings to DynamoDB
        save_to_dynamodb(instance_id, instance_settings)
    except ec2_client.exceptions.ClientError as e:
        logger.error(f"Failed to retrieve settings for EC2 instance {instance_id}: {e}")
        if e.response["Error"]["Code"] == "AccessDeniedException":
            logger.error("Access denied. Ensure the IAM role has the following permissions: AmazonEC2FullAccess.")
        raise


def save_to_dynamodb(resource_id: str, settings: dict):
    """
    Save settings to DynamoDB and verify the saved

 data.

    Args:
        resource_id (str): The unique identifier for the resource.
        settings (dict): The settings to save.

    Raises:
        Exception: If an error occurs while saving to DynamoDB or the saved data doesn't match.
    """
    logger.info(f"Saving settings for {resource_id} to DynamoDB")
    assert (
        isinstance(resource_id, str) and resource_id
    ), f"resource_id must be a non-empty string, not {type(resource_id)}"
    assert isinstance(settings, dict), f"settings must be a dictionary, not {type(settings)}"

    try:
        table.put_item(Item={"ResourceID": resource_id, "Settings": settings})
        logger.info(f"Settings for {resource_id} saved")

        # Verify the saved data
        response = table.get_item(Key={"ResourceID": resource_id})
        if "Item" not in response or response["Item"]["Settings"] != settings:
            raise Exception(f"Verification failed: Data for {resource_id} does not match.")

        logger.info(f'Verified settings for {resource_id}: {response["Item"]}')
    except dynamodb.meta.client.exceptions.ClientError as e:
        logger.error(f"Failed to save settings to DynamoDB for {resource_id}: {e}")
        if e.response["Error"]["Code"] == "AccessDeniedException":
            logger.error("Access denied. Ensure the IAM role has the following permissions: AmazonDynamoDBFullAccess.")
        raise
    except Exception as e:
        logger.error(f"Error verifying saved data for {resource_id}: {e}")
        raise

Environment Variables

API_GATEWAY_ID: The ID of the API Gateway whose stages are to be disabled.
SIMULATE: When set to true, the lambda will simulate actions without making actual changes.
DISABLE_ALL_LAMBDAS: When set to true, all Lambda functions will be disabled, except those in the blacklist.
LAMBDAS_TO_DISABLE: A comma-separated list of specific Lambda function names to disable.
LAMBDAS_TO_NOT_DISABLE: A comma-separated list of Lambda function names that should not be disabled.
DISABLE_ALL_EC2_INSTANCES: When set to true, all EC2 instances will be disabled, except those in the blacklist.
EC2_TO_DISABLE: A comma-separated list of specific EC2 instance IDs to disable.
EC2_TO_NOT_DISABLE: A comma-separated list of EC2 instance IDs that should not be disabled.
BLACKLISTED_LAMBDAS: A comma-separated list of Lambda function names that should not be disabled.
BLACKLISTED_EC2_INSTANCES: A comma-separated list of EC2 instance IDs that should not be disabled.

Step 2: Give the Lambda function full access in IAM

See General setup

Step 3: Blacklist the restore Lambda

Don't forget to add the restore lambda name in the BLACKLISTED_LAMBDA_FUNCTIONS, otherwise it will be killed as well.

Step 4: Set Up SNS to Trigger the Lambda Function

Find the SNS topic you created in the Billing Alarm stage, in the SNS console.
Subscribe the Lambda Function to the SNS Topic:
- Add a subscription to the SNS topic.
- Choose the protocol as "AWS Lambda" and select your Lambda function.

Closing Thoughts

Here you go! By following these steps, you can ensure that your AWS API Gateway and Lambda functions are disabled when you reach a specific budget threshold, preventing further costs and limiting the possible cost overrun. This ensures that your services are only disabled when the budget is reached and are automatically restored at the beginning of the next period, keeping cost in check by killing your product when you spent too much on it. 💰💤

Controlling AWS Lambda Costs

Kodsama — Fri, 28 Jun 2024 13:36:26 +0000

Managing your Amazon Web Services (AWS) costs can be challenging. AWS provides tools to limit usage and notify you when costs rise unexpectedly. This guide focuses on scalable AWS services like AWS Lambda to help you keep costs under control.

Manually tearing down services when your budget is high is cumbersome. Instead, spend time setting up automated limits to prevent overspending.

Steps to Control Costs

Set Up Billing Alarms
Limit Lambda Concurrency
- Reduce Global Service Quotas
- Set Concurrency Limits for Individual Lambda Functions
- Set Rate Limits on API Gateway
- Use AWS WAF to Avoid HTTP Flooding
Using a Killer Lambda
Closing Thoughts

Setting Up Billing Alarms

The first step is to set up budget alerts for your AWS costs. This way, you’ll get notified if your monthly AWS bill is estimated to cross a set threshold.

Create a Budget:
- Go to the AWS Management Console.
- Navigate to the AWS Budgets dashboard.
- Click on "Create a budget."
- Follow the steps to set your budget amount.
Configure Alerts:
- Set up alert notifications for when your budget threshold is reached. You can receive alerts via email or SNS (Simple Notification Service).
- To use SNS, create an SNS topic if you don't have one, and add subscribers (e.g., your email address).

With these alerts, you’ll know when you’re approaching your budget and can take action.

Limiting Lambda Concurrency

Balancing scalability and cost is crucial. Limit Lambda invocations to prevent unexpectedly large bills.

Reduce Global AWS Service Quotas for Lambda

Lower the total concurrency of all Lambda invocations (default is 1000):

Go to the AWS Management Console.
Search for "Service Quotas."
Filter for "Lambda."
Click on "Concurrent executions."
Click the "Request quota increase" button.
Fill in the new limit and submit the request.

Set Concurrency Limits for Individual Lambda Functions

Limit concurrent invocations of specific Lambda functions:

Navigate to the desired Lambda function in the AWS Console.
Go to Configuration > Concurrency.
Click "Edit" in the Concurrency section.
Select "Reserve concurrency" and set the limit.
Click "Save."

Set Rate Limits on API Gateway

Control the number of requests sent to your backend:

Go to the AWS Management Console.
Search for "API Gateway."
Select your API.
Go to "Stages" and select the stage (e.g., prod, dev).
Scroll to the "Throttle" section in the "Stage Editor" tab.
Set the Rate Limit and Burst Limit.
Click "Save Changes."

Use AWS WAF to Avoid HTTP Flooding

Protect against DDoS attacks by setting rate limits and blocking excessive requests:

Go to the AWS Management Console.
Search for "WAF."
Click on "Create web ACL."
Add a rate-based rule with your desired rate limit.
Set the action to "Block" when the rate limit is exceeded.
Select the resources you want to protect.
Click "Create web ACL."

Using a Killer Lambda

To prevent AWS API Gateway and AWS Lambda from being invoked when a specific budget is reached, we can combine multiple AWS tools. For the sake of readability, I moved this part to a new article: AWS cost control last resort, the killer lambda.

Closing Thoughts

By following these steps, you can ensure that your AWS API Gateway and Lambda functions are managed effectively, preventing further costs and limiting possible cost overruns. This approach ensures your services are controlled and automatically restored at the beginning of the next period, keeping costs in check.

Managing AWS costs can be tough, but with the right setup, you can avoid unexpected expenses. Start with billing alarms and budgets, then use concurrency limits and API Gateway rate limits to control usage. AWS WAF can also protect against unexpected spikes.

Implementing these steps takes effort but will save you from manual intervention and unexpected costs in the long run. Focus on development, knowing your AWS costs are under control.

Address cost explosions in the development stage, but at least you can sleep peacefully until then. 😴

Let ChatGPT write your best LinkedIn summary & posts

Kodsama — Thu, 16 Mar 2023 10:07:52 +0000

A lot of people are jumping the wagon of generative AI and discovering the good and bad of the technique.

While everything should be taken with a grain of rough salt, the ChatGPT tool is by design great for summarizing big amount of data.

During my previous post Job application on steroids with ChatGPT, we walked through how to include your professional experience to ✨automagically write✨ a cover letter and answer simple questions related to a job position.

Here I would like to briefly show how you could use it to create a the best professional summary you will have on your LinkedIn profile or CV.

A very brief note: I recommend to keep a text document with an extensive description of what you did in your different jobs and what you want to highlight. This will serve as a great source for ChatGPT to answer requests.

How?

The syntax is pretty simple, just write a prompt for chat GPT in the form of:

Write a passionate short text about the kind of person I am using this professional experience: [Personal experience]

The way to extract your professional experience is explain for beginners in a previous post.

It will then summarize the kind of jobs you did, and you can adjust on the fly to get to the result you want. You can have a look at an example in the A practical example part below.

It is quite scary and still very helpful to use such a tool as it provides a fast and somewhat accurate way of finding the essence of a question you have with a lot of attached data.

For a human like me it is difficult to summarize who I think I am, but for ChatGPT it barely takes a few tokens...

A concrete example

I thought I could give a concrete example as an illustration of a command I wrote (I shortened it with [...]). Click on the arrows below to see them.

I started with:

Write a passionate short text about the kind of person I am using this professional experience: [Personal experience]

the complete ChatGPT request
Write a passionate short text about the kind of person I am using this professional experience:

Aaa
Digital Product Leader
[...]
I addressed customer needs and helped them focus on their core business. I
oversaw the entire process from defining the problem to delivering a solution
and providing support [...]
Bbb
Video lead engineer
[...]
As the lead developer for video-related projects, I worked [...]
Ccc
Experienced Video Software Engineer
[...]
I developed new video functionalities in the new generation of embedded [...]
Ddd
Flight Attendant
[...]
Being a flight attendant means [...]
Eee
Software Engineer
[...]
I wrote a VNC client software on PC (Windows) from scratch [...]
Fff
Cast Member
[...]
Ensuring that the guests have a pleasant day [...]

Education:
Zzz
Doctor of Philosophy - PhD, Automatic Control, Game theory, Applied
mathematics · (April 2016 - June 2022)
Yyy
Master of Business Administration - MBA, Business Administration and
Management, General · (September 2019 - July 2021)
Xxx
Master of Engineering - MEng, Electrical, Electronics and Communications
Engineering · (September 2010 - September 2013)
Www
Bachelor of Engineering - BE, Electrical and Electronics
Engineering · (September 2008 - September 2010)

And ChatGPT compiled this text:

the ChatGPT answer
Hello! I am a digital product leader with extensive experience in product management, project management, cross-functional team leadership, and product development. Currently, I am working at Aaa as a Digital Product Leader, where I am responsible for solving customer problems, managing an eCommerce platform, and an IoT web application. My workday includes everything from identifying problems, proposing solutions, ensuring timely and budget delivery, and handling project execution. Besides this, I have experience as a Video lead engineer at Bbb, where I worked with the backend team to enhance the cloud video architecture performance and deliver high-quality products while reducing overall system costs. Moreover, I worked for eight years at Ccc as an Experienced Video Software Engineer, where I developed new video functionalities for the new generation of embedded cameras, and my team was responsible for video and codec for all company products. My experience includes working with Amazon Web Services, video compression, video quality, embedded software, machine learning, and cloud. Overall, I have a track record of delivering innovative solutions addressing today and tomorrow's challenges related to video and bandwidth.

It was too crude and detailed to I corrected using this query:

Make it shorter, remove reference to previous work. Just write about what kind of person I am and make it more passionate.

ChatGPT correction
You are a passionate and driven individual who approaches life with a positive and optimistic outlook. You have a natural curiosity and a desire to learn and grow, which has led you to explore a variety of interests and pursue diverse experiences. Your strong work ethic and determination have helped you achieve success in both your personal and professional life. You value authenticity, honesty, and kindness, and strive to embody these qualities in your interactions with others. Overall, you are a compassionate and dynamic person who is always seeking to better yourself and the world around you.

And finally it needs to be more personal, so you need to change the pronom:

Rephrase this at the first person.

ChatGPT correction
I am a passionate and driven individual who approaches life with a positive and optimistic outlook. I have a natural curiosity and a desire to learn and grow, which has led me to explore a variety of interests and pursue diverse experiences. My strong work ethic and determination have helped me achieve success in both my personal and professional life. I value authenticity, honesty, and kindness, and strive to embody these qualities in my interactions with others. Overall, I am a compassionate and dynamic person who is always seeking to better myself and the world around me.

and as I bonus, I asked:

Rewrite it a linkedin post

And got a post which looks a lot like some colleagues:
the LinkedIn post
Hello LinkedIn fellows! 🌟

I'm a passionate and driven individual who believes in embracing every moment with positivity and optimism. My natural curiosity and love for learning have led me to explore a range of interests and pursue diverse experiences. With a strong work ethic and unwavering determination, I have achieved success both personally and professionally.

I prioritize authenticity, honesty, and kindness in all my interactions with others, making sure to embody these qualities in everything I do. As a compassionate and dynamic person, I am always seeking ways to better myself and the world around me.

Let's connect to explore opportunities to learn and grow together! 🤝

Pretty impressive I would say.

The cover image is under Creative Commons 3 license and is provided by picpedia.

Job application on steroids with ChatGPT

Kodsama — Tue, 14 Mar 2023 20:28:13 +0000

Applying to job positions is a pain, first because you repeat yourself, second because the amount of work required for the final result isn't often worth the effort.

Instead of spending your time applying, spend your time filling your LinkedIn profile and let ChatGPT (or any other language model AI) do most of the hard work after that.

The simple steps

It is pretty easy to do, you just need 4 steps to succeed at it:
1. Export your LinkedIn information
2. Compile a ChatGPT request
3. From then on, ask it what you need

Let's dig in a little more in details and look at an example

Of course to be able to do this you already need to have a working ChatGPT account set-up.

1. Export your LinkedIn information

Simply follow the steps here to get a pdf of your profile
Then select the content of the right column to get most of the data.

Do some cleaning and add anything you think would be interesting (remember you only need to do that from time to time, so you can spend some time on that).
Keep it in a text editor for later.

2. Compile a ChatGPT request

Now it gets interesting, do some formatting and compile the chat GPT command with the information you provided.

My generic pattern (different combo can be done of course!) is (replace the brackets to tweak as you please):

[Write a cover letter] for the job offer below using according to my experience.
The job offer:
[Paste the information about the job position you are interested in]

My experience:
[Paste my profile data (step 1)]

Here you go, now ChatGPT is doing the magic for you!

3. Explore

From then on, ask it what you need ! The pattern in 3 allows to do whatever you want now. You can replace "Write a cover letter" with "Write why I am the perfect candidate for the job" or "Answer why I would be a good leader", etc.
As soon as you keep the for the job offer below using according to experience and keep the data below, ChatGPT should do some proper magic!**

A concrete example

I thought I could give a concrete example as an illustration of a command I wrote (I shortened it with [...]). Click on the arrows below to see them.

the ChatGPT request
Write a cover letter for the job offer below using according to my experience.

The job offer:

We are actually not looking for a manager. We are looking for a leader that will help us guide one of our team to become the best version of themself. Join the journey! We are a people company. Our experienced, skilled and passionate people empower customers and partners to meet their challenges through best-in-class, user-friendly solutions. And we are a technology company. We help our customers and partners optimize processes through our software, hardware and services and through our support. Connecting people and technologies provide the means to accomplish goals and ambitions and gain a competitive edge in today’s digitalized world. [...] Xxx is a multinational, cross-industry innovator that connects people and technologies to optimize processes for business-critical applications. Our offer includes operator communication, automation solutions, digitalization, display solutions and support. As experts in user-friendly software, hardware and services for the Industrial Internet of Things, we empower our customers to meet their challenges through leading-edge solutions. Below are some bullets that we believe you have previous experience in:
Leading software engineers in product development teams
Developing an organization to the next level
Agile methodologies
Great communication skills, English is a must-have

My experience:

And ChatGPT compiled this cover letter:

the ChatGPT answer
Dear Hiring Manager,

I am writing to express my interest in the leadership position that you recently posted. Your description of the role, which involves guiding and empowering team members to achieve their full potential, is precisely the kind of leadership that I believe in. Based on my experience as a Digital Product Leader at Aaa, a Video Lead Engineer at Bbb, and an Experienced Video Software Engineer at Ccc, I believe that I have the skills and expertise that you are looking for.

In my current role at Aaa, I have been working with teams across different countries to develop and maintain an eCommerce platform and IoT web application. I have been responsible for overseeing the entire process, from defining the problem to delivering the solution, and providing support to ensure customer satisfaction. In doing so, I have worked closely with teams of 4 or more people and have collaborated with other teams within the Xxx group to bring our customers' ideas to life.

I have experience in product management, project management, and Scrum master roles, ensuring that projects are delivered on time and within budget. I am also skilled in problem management, addressing issues and challenges as they arise and working to resolve them efficiently. Additionally, I have focused on team development, coaching and mentoring team members to help them reach their full potential and fostering an environment of collaboration and continuous improvement.

In my previous role as a Video Lead Engineer at Bbb, I collaborated with the backend team to enhance the performance of the video architecture. I also partnered with key stakeholders to ensure that our products aligned with their needs and requirements. I have extensive experience working with Amazon Web Services, video compression, and video quality and algorithms.

My experience at Ccc as an Experienced Video Software Engineer has provided me with a deep understanding of the tech industry. I have experience working with agile methodologies, developing organizations to the next level, and leading software engineers in product development teams. My communication skills in English are strong, which is a must-have for the role you are offering.

I believe that I can bring something new to the table and contribute to your company's success. I am excited about the opportunity to join your team and help drive innovation and change within the R&D organization. Thank you for considering my application.

Sincerely,
[Your name]

Pretty cool no? And it took (after step 1 is done) basically 30 seconds!

Then you can adjust the text with simple commands as you please, some examples could be Emphasize my skill at xxx or highlight my leadership competence or be more humble...

Don't underestimate the magic of chain prompting:

Example Chained prompting
Write an article about using ChatGPT for boosting my LinkedIn. First give me the outline, which consists of a headline, a teaser, and several subparts.
[Output.]
Now write one key messages for each subpart.
[Output.]
Add three keywords to the key messages for each subpart.
[Output.]
...

And you can close the circle by having ChatGPT filling your LinkedIn for you as well.

As Marvin Phillips would say:

The difference between try and triumph is just a little umph!

So, happy (lightfast) job hunting!

The cover image is under Creative Commons 4 license and is provided by PNGAll.

How good is my video? A (traditional) video quality metrics survey

Kodsama — Fri, 29 Apr 2022 09:41:54 +0000

Human perception of video/image quality does not correlate directly with the amount of compression in a video (or image) and the perception of quality is influenced by physiological factors, cultural background or social trends (for example the amount color saturation or sharpness liked differs between people).

Image quality metrics is an active research topic which involves many adjacent areas such as biology and machine learning, as modeling human visual perception (or human visual system, HVS) is a very complex topic.

This post aims to provide pointers for anyone looking into image or video quality comparison or evaluation.

If you want to keep reading on the topic, sources are present at the end.

DISCLAIMER:
I only go through the "traditional" metrics (the ones using traditional mathematical approaches), note that there are a lot not listed which are deep-learning based. You can find a list of those HERE.

Executive summary

The most common visual metrics used in the industry for now are by far the objective full-reference PSNR and SSIM. The growing and as of today state-of-the-art metric is VMAF.

If you want to use no-reference metrics, BRISQUE is a simple and often used metric (even if less common than the full-reference).

Two metrics groups

We can isolate two types of video quality metrics families ¹:

Subjective, that present the images to groups of people and ask them to grade their view of the quality.
Objective, that use image analysis to extract a value or set of values which can be compared.

Subjective tests

Subjective quality assessment methods are the most direct method to know how the observers feel about the video quality. These tests use groups of people to calculate the Mean Opinion Scores (MOS) . MOS is a measure used in the domain of Quality of Experience and telecommunications engineering, representing overall quality of videos with different properties.

The Difference Mean Opinion Score (DMOS) is the same idea but indicates how much the differences introduced in test videos degrade subjective picture quality.

Organizations such as International Telecommunication Union (ITU) or Video Quality Experts Group (VQEG) have recommended varieties of subjective assessment methods on images and videos.

These methods require very restrictive conditions (such as wall colors, time to display images, etc) and specialized hardware (color calibrated screens) in order for the tests to be comparable. International recommendations for subjective video quality assessment ITU-R BT.500-11 for example include specifications for how to perform many different types of subjective tests ².

A subjective test looks like this:

Tests can be:

double stimulus, where viewers rate the quality or change in quality between two video streams (reference and impaired)
single stimulus, where viewers rate the quality of just one video stream (the impaired).

Some double stimulus tests are:

DSCQS: double stimulus continuous quality scale (claims to be less sensitive to context than others).
DSCS: double stimulus comparison scale.

For single stimulus:

SSCQE: single stimulus continuous quality evaluation (claims to yield more representative quality estimates for quality monitoring applications).

Objective tests

The goal of objective tests research is to design algorithms that can automatically assess the quality of images or videos in a consistent manner and with minimal human intervention.

Objective tests can be split into three method groups¹:

Full reference method (FR-IQA), where one is assumed to have access to the "ground truth" image that is not altered (often called reference image).
Reduced reference method (RR-IQA), where one has access to information (reduced image, features information, etc) about the "ground truth" image ³.
No reference method (NR-IQA),where one has no access to the original image ⁴.

The Full reference method being the most used in the broadcast industry.

Full reference methods

These algorithms generally interpret image quality as fidelity or similarity with a "reference" image. Full-Reference methods often attempt to model physiological and psycho-visual features of the human visual system (HVS). These methods give an idea of the extent of deviation of a distorted image from the reference image.

The main methods are listed below. The state of the art as of today (2022-05-01) being VMAF and VQM.

PSNR: Peak signal-to-noise ratio ⁵

PSNR calculates the difference (mean square error) between the original image and its compressed version.
PSNR represents the ratio between the maximum possible power of a signal (the original image) and the power of corrupting noise (the corruption introduced by the compression) that affects the fidelity of its representation. It is expressed as a logarithmic quantity in decibel. The higher the PSNR is, the better (the higher the fidelity of the signal should be, note that an identical image should have an infinite PSNR).

PSNR has been the most used metric to measure losses from video compression, however it does not include the characteristics of the human visual system and thus is a good mathematical representation of degradation but not a good practical measure as most corruption would not be perceived by viewers.
You can find how it is calculated here and in ⁵.

SSIM: Structural similarity index measure ⁶

SSIM is a perception-based model that considers image degradation as perceived change in structural information, while also incorporating perceptual phenomena, including both luminance masking and contrast masking terms.
Structural information is the idea that the pixels have strong inter-dependencies especially when they are spatially close (where PSNR only considers the difference between values independently of its placement). These spatial dependencies carry information about the structure of the objects in the visual scene. It is built on the assumption the measure of structural similarity can provide a good approximation to perceived image quality.

The basic model was developed in the Laboratory for Image and Video Engineering (LIVE) at The University of Texas at Austin and further developed jointly with the Laboratory for Computational Vision (LCV) at New York University.

SSIM is building on the limitations of PSNR by starting to add human visual system characteristics regarding structure perception and has been widely adopted in the industry. You can find how it is calculated here and in ⁶. You can see in the picture above that SSIM is capturing the difference in visual quality while the PSNR value stays equal.

MS-SSIM: Multi-scale Structural Similarity for Image Quality Assessment ⁷

This is a variant of SSIM. It supplies more flexibility than
previous single-scale methods in incorporating the variations of
viewing conditions. it calibrates the parameters that define the relative importance of different scales of structures.

FSIM: Feature Similarity Index for Image Quality Assessment ⁸

FSIM is yet-another variant of SSIM, it maps the features and measures the similarities between two images. FSIM goes from the assumption that the human visual system (HVS) understands an image mainly according to its low-level features. The phase congruency (PC), a dimensionless measure of the significance of a local structure, is used as the primary feature and is invariant to light changes in the image. The image gradient magnitude (GM) represents the gradient of the image and is employed as the secondary feature.
PC and GM play complementary roles in characterizing the image quality. After obtaining the local quality map, PC is again used as a weighting function to derive a single quality score.

IFC: Information Fidelity Criterion for Image Quality Assessment Using Natural Scene Statistics ⁹

In IFC, they approach the problem of image quality by proposing an information fidelity criterion that is based on natural scene statistics.
Researchers have developed sophisticated models to capture the statistics of pictures and videos of the visual environment. Using these statistical models in an information-theoretic setting, they developped a parameter-less algorithm that provides clear advantages over the traditional approaches. IFC is not a distortion metric, but a fidelity criterion. It theoretically ranges from zero (no fidelity) to infinity (perfect fidelity).

NQM: Noise Quality Measure [^16]

NQM decouples linear frequency distortion from noise and measures their effect on the human visual system. It takes into account the following:

variation in contrast sensitivity with distance, image dimensions, and spatial frequency
variation in the local luminance mean
contrast interaction between spatial frequencies;
contrast masking effects.

The distortion metric is calculated in three steps:

find the frequency distortion in the degraded image.
compute the deviation of this frequency distortion.
weight the deviation by a model of the frequency response of the human visual system and integrate over the visible frequencies.

UQI: Universal Quality Index ¹⁰

In UQI, instead of using error summation it models the image distortion as the sum of three factors:

loss of correlation
luminance distortion
contrast distortion

This metric is similar to PSNR but captures some distortions which are not picked up by the latter.

VSNR: Visual signal-to-noise ratio ¹¹

VSNR uses a three-stages:

It calculates contrast thresholds for detection of distortions via wavelet-based models of visual masking (to estimate if the difference with the original would be visible).
If deemed visible (above threshold) it then it calculates the Euclidean distances in the distortion-contrast space (in the context of a wavelet-based decomposition).
Calculates the linear sum of the distances in 2. as a global metric.

VSNR operates based on physical luminances and visual angle (rather than on digital pixel values and pixel-based dimensions) on both low-level and
mid-level properties of human visual system (HVS) and has provided good correlate with subjective studies (as you can see below)¹¹.DMOS definition

MPQM: Moving Pictures Quality Metric ¹²

MPQM is based on a multi-channel model of the human visual system (HVS) that has been parameterized for video coding applications by psycho-physical experiments. The model claims to account for spatial and temporal aspects of the HSV as well as their interactions.
The visual mechanisms of vision are simulated by a spatio-temporal filter bank. A decomposition is then used to account for phenomena as contrast sensitivity and masking. Once the amount of distortions perceived is known, quality estimation can be assessed at various levels.

VIF: Visual Information Fidelity ¹³

VIF is built on the premise that quality is complementary to the measure of information fidelity loss. The VIF score is measured as a loss of fidelity combining four different spatial scales. It quantifies the loss of image information to the distortion process, and explore the relationship between image information and visual quality. VIF quantifies the information that is present in the reference image, and also quantify how much of this reference information can be extracted from the distorted image.

DLM: Detail Loss Metric ¹⁴

DLM separately evaluates detail losses (loss of useful visual information which affects the content visibility) and the additive impairment (the redundant visual information whose appearance in the test image will distract viewer’s attention from the useful contents causing unpleasant viewing experience). The outputs of detail loss and additive impairment are adaptively combined to yield the overall DLM quality measure. DLM had better if not equal results than PSNR, SSIM or VIF during comparisons ¹⁴.

MOVIE: Motion tuned spatio-temporal quality assessment of natural videos ¹⁵

This metric aims to evaluate the effect of motion on visual quality.
It considers two separate components. It evaluates the spatial quality with a method loosely inspired by the SSIM index and emporal quality using motion information from the reference video sequence (via pixel changes). The spatial and temporal quality scores are combined to obtain an overall video integrity score known as the MOVIE index. Note that it only considers the luminance domain.

ST-MAD: A spatiotemporal most apparent distortion model for video quality assessment ¹⁶

Spatiotemporal Most Apparent Distortion (MAD)¹⁷ extends a previous paper from the same author¹⁷. ST-MAD estimates motion-based distortion by applying MAD’s appearance-based model (see below).
Local luminance and contrast masking are used to estimate detection-based perceived distortion in high-quality images, whereas changes in the local statistics of spatial-frequency components are used to estimate appearance-based perceived distortion in low-quality images.

ST-MAD uses multiple strategies to determine image quality:

For images containing near-threshold distortions, the image is most apparent, and thus the human visual system (HVS) attempts to look past the image and look for the distortions (a detection-based strategy).
For images containing clearly visible distortions, the distortions are most apparent, and thus the human visual system (HVS) attempts to look past the distortion and look for the image’s subject matter (an appearance-based strategy).

It is then adjusted with weights which take in account the human visual system (HVS) tendency to be more affected by fast-moving regions located toward the center of the video.

VQM: Video quality metric ¹⁸

VQM is an algorithm that uses a neural network model to fuse low-level features into a final metric.

The Video Quality Metric (VQM) has been developed by Institute for Telecommunication Sciences (ITS), the research laboratory of the National Telecommunications and Information Administration (NTIA) (USA). It was standardized by the ITU for SDTV (ITU-T Rec. J.144), but has shown very good performance, especially the updated version VQM for variable frame delay (VQM-VFD). However, Netflix's evaluation put VMAF slightly ahead of VQM-VFD. The problem for both of these models (and the other top models) is that distribution quality video spans a small range of quality.

VMAF: Video multi-method assessment fusion ¹⁹

VMAF is a quality metric developed by Netflix in cooperation with the University of Southern California, The IPI/LS2N lab University of Nantes, and the Laboratory for Image and Video Engineering (LIVE) at The University of Texas at Austin.
VMAF uses existing image quality metrics and other features to predict video quality:

Visual Information Fidelity (VIF)
Detail Loss Metric (DLM)
Mean Co-Located Pixel Difference, it measures temporal difference (motion) between frames on the luminance component.

The metrics are combined with SVM-based regression to provide a single output score in the range of 0–100 per video frame (100 being quality identical to the reference video). These scores are then combined for the whole video to provide an overall differential mean opinion score (DMOS).

VMAF is similar to VQM in spirit, but VQM extracts features at lower levels such as spatial and temporal gradients. Since the VMAF approach allows for incorporation of new elementary metrics into its framework, VQM could serve as an elementary metric for VMAF as well.¹⁹

Reduced reference methods

In these methids only limited features from a reference image instead of the complete image are used to evaluate the quality of distorted image.

FSI: Free-energy principle and Sparse Representation ²⁰

FSI is based on the neuroscience free-energy principle and tries to model the perception and understanding of the scene as an active inference process.
FSI is an image quality metric where the reference and distorted images are predicted by the sparse representation at first. Then, the difference between the entropies of the prediction discrepancies is defined to measure the image quality. It claims to have comparative results to PSNR and SSIM compared to subjective testing.

SRR: SSIM Reduced-Reference ²¹

The SRR value is calculated as the ratio of received (target) video signal SSIM with reference video pattern SSIM values. It uses a ratio (SRR) based on the SSIM measured at the source (provider, where we have the reference) divided by the SSIM at the user side (where we have a reduced reference). It claims to have higher fidelity than SSIM even without reference of 2.56%.

ST-RRED: ST-Reduced Reference Entropic Differencing ²²

ST-RRED is an extension of Reduced Reference Entropic Differencing (RRED) and computes wavelet coefficients of consecutive frame differences (temporal RRED) which in conjunction with spatial RRED indices provide a quality value. The spatial and temporal information differences are combined to obtain the reduced reference differences. They adopt a hybrid approach of combining statistical models and perceptual principles.

No reference methods

These algorithms aim to provide a quality metric of an image without the need of any reference image or its features. Due to absence of reference image, one needs to model the statistics of reference image, the nature of the human visual system (HVS) and effect of distortions on the image. It is difficult to evaluate the effectiveness of a quality measure with a specific distorted image in absence of a reference image.

The metrics can be further split up into two broad categories: phenomenological, that analyze features of the image without attempting to understand the fundamentals of the interaction between the human visual system (HVS) and the image, and model-based, that incorporate some more or less accurate representation of the human visual system (HVS) in the assessment algorithm ²³.

BRISQUE: No-reference Image Quality Assessment in the Spatial Domain ²⁴

This model doesn't compute distortion-specific features, such as ringing, blur, or blocking, but instead uses scene statistics of locally normalized luminance coefficients to quantify possible losses of “naturalness” in the image due to the presence of distortions, thereby leading to a holistic measure of quality.
No transformation to another coordinate frame (DCT, wavelet, etc.) is required, distinguishing it from prior non-reference approaches.
They claim to achieve better results than full-reference PSNR and SSIM.

NIQE: A Feature-Enriched Completely Blind Image Quality Evaluator ²⁵

In NIQE the authors used statistical features from the images and created a gaussian model of reference images. They calculate the distance between the features of the image to be tested with the model to determine its quality.

NBIQA: A Novel Blind Image Quality Assessment Method Based on Refined Natural Scene Statistics ²⁶

NBIAQ uses selected features from both the spatial and transform domains and Natural Scenes Statistics (NSS) model by selecting competitive features from existing NSS models and adding three new features. Then the refined NSS is fed into Support Vector Machine (SVM) tool to learn a simple regression model.

PSS: Blind Quality Assessment of Compressed Images via Pseudo Structural Similarity ²⁷

The assumption is that distortion would change the structural similarity in the image if heavily distorted. The idea in PSS is to be able to compare the highest compressed image (with theoratically the most compression artifacts) to the image we want to determine the quality level and estimate the distance. Unfortunately I did not find concrete benefits when comparing to other No-reference approaches.

BMPRI: Blind Image Quality Estimation via Distortion Aggravation ²⁸

In BMPRI, multiple pseudo reference images (MPRIs) are created by further degrading the distorted image in several ways and to certain degrees, and then compare the similarities between the distorted image and the MPRIs. Via such distortion aggravation, they create some kind of reference image, i.e., the MPRIs, and utilize the full-reference to extract local binary pattern features to "grade" the the similarities between the distorted image and the MPRIs. The similarity scores are then utilized to
estimate the overall quality.

DIIVINE: Distortion Identification-based Image Verity and Integrity Evaluation ²⁹

DIIVINE operates based on real-valued wavelet coefficients, whereas the visual appearance of an image can be strongly determined by both the magnitude and phase information.
It assesses image quality based on the complex Gaussian scale mixture model corresponding to the complex version of the steerable pyramid wavelet transform as these distributions have characteristic shapes that are consistent across different natural images but change significantly in the presence of distortions.

BLIINDS: No-reference image quality assessment based on local

magnitude and phase statistics of natural scenes ³⁰

The approach relies on a spatio-temporal model of video scenes in
the discrete cosine transform domain, and on a model that characterizes the type of motion occurring in the scenes, to predict video quality.
BLIINDS outperforms the NIQE index and the full-reference PSNR and SSIM measures. Video BLIINDS does not quite attain the performance level of state-of-the-art full-reference VQA measures, (MOVIE and ST-MAD), but its performance is nearly as good and with much less computational cost.

Sources

A nice collection of sources

011006, 2010. PDF

Brunnström, K., A. Djupsjöbacka, and B. Andren (2021). Objective video quality assessment methods for Video assistant refereeing (VAR) Systems. Tech. rep. RISE report 2021:30. RISE Research Institute of Sweden AB. PDF ↩
Pinson, M. H., & Wolf, S. (2003, June). Comparing subjective video quality testing methodologies. In Visual Communications and Image Processing 2003 (Vol. 5150, pp. 573-582). SPIE. PDF ↩
Dost, S., F. Saud, M. Shabbir, M. G. Khan, M. Shahid, and B. Lovstrom (2022). “Reduced reference image and video quality assessments: review of methods”. EURASIP Journal on Image and Video Processing 2022:1, pp. 1–31. PDF ↩
Lin, X., H. Ma, L. Luo, and Y. Chen (2012). “No-reference video quality assessment in the compressed domain”. IEEE Transactions on Consumer Electronics 58:2, pp. 505–512. PDF ↩
Teo, P. and D. Heeger (1994). “Perceptual image distortion”. In: IEEE International Conference on Image Processing, Austin, TX, 982–986 vol.2. PDF ↩
Wang, Z., A. Bovik, H. Sheikh, and E. Simoncelli (2004). “Image quality assessment: from error visibility to structural similarity”. IEEE Transactions on Image Processing 13:4, pp. 600–612. PDF ↩
Multi-scale Structural Similarity for Image Quality Assessment (MS-SSIM), IEEE Asilomar Conference on Signals, Systems and Computers, 2003, Wang Z et al. PDF ↩
FSIM: A Feature Similarity Index for Image Quality Assessment (FSIM), IEEE TIP, 2011, Zhang L et al. PDF ↩
Information Content Weighting for Perceptual Image Quality Assessment (IW-SSIM), IEEE TIP, 2010, Wang Z et al. PDF ↩
. Wang and A. Bovik, “A universal image quality index,” IEEE Signal Proces. Lett., vol. 9, no. 3, pp. 81–84, Mar. 2002. PDF ↩
. Chandler and S. Hemami, “VSNR: A wavelet-based visual signal-to-noise ratio for natural images,” IEEE transactions on image processing, vol. 16, no. 9, pp. 2284–2298, 2007. PDF ↩
C. J. Branden Lam brecht and O. Verscheu re. “Perceptual Quality Measure using a Spatio-Temporal Model of the Human Visual System,” Proc. SPIE Vol. 266 8, p. 450-461, March, 1996. PDF ↩
H. Sheikh and A. Bovik, “Image information and visual quality,” IEEE Transactions on image processing, vol. 15, no. 2, pp. 430–444, 2006. PDF ↩
S. Li, F. Zhang, L. Ma, and K. Ngan, “Image Quality Assessment by Separately Evaluating Detail Losses and Additive Impairments,” IEEE Transactions on Multimedia, vol. 13, no. 5, pp. 935–949, Oct. 2011. PDF ↩
K. Seshadrinathan and A. C. Bovik, “Motion tuned spatio-temporal quality assessment of natural videos” IEEE Trans. Image Process., vol. 19, no. 2, pp. 335–350, Feb. 2010. PDF ↩
P. V. Vu, C. T. Vu, and D. M. Chandler, “A spatiotemporal most apparent distortion model for video quality assessment” in Proc. IEEE ICIP, Sep. 2011, pp. 2505–2508. PDF ↩
Eric C. Larson and Damon M. Chandler, “Most apparent distortion: full-reference image quality assessment and the role of strategy,” Journal of Electronic Imaging, vol. 19, no. 1, pp. ↩
ITU-T (2004). Objective perceptual video quality measurement techniques for digital cable television in the presence of full reference (itu-t rec. j.144). https://www.itu.int/rec/T-REC-J.144-200403-I. ↩
Li, Z., A. Aaron, I. Katsavounidis, A. Moorthy, and M. Manohara (2016b). Toward a practical perceptual video quality metric. url: https: //netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric- 653f208b9652. ↩
Reduced-reference Image Quality Assessment in Free-energy principle and Sparse Representation (FSI), IEEE TMM, 2017, Liu Y et al. PDF ↩
Kourtis, M.-A.; Koumaras, H.; Liberal, F. (July–August 2016). "Reduced-reference video quality assessment using a static video pattern". Journal of Electronic Imaging. 25 (4): 043011. PDF ↩
Soundararajan, R.; Bovik, A.C. (2013-04-04). "Video Quality Assessment by Reduced Reference Spatio-Temporal Entropic Differencing". IEEE Transactions on Circuits and Systems for Video Technology. 23 (4): 684–694. PDF ↩
Zhai, G. and X. Min (2020). “Perceptual image quality assessment: a survey”. Science China Information Sciences 63:11, pp. 1–52. PDF ↩
No-reference Image Quality Assessment in the Spatial Domain (BRISQUE), IEEE TIP, 2012, Mittal A et al. PDF ↩
A Feature-Enriched Completely Blind Image Quality Evaluator (IL-NIQE), IEEE TIP, 2015, Zhang L et al. PDF ↩
A Novel Blind Image Quality Assessment Method Based on Refined Natural Scene Statistics (NBIQA), IEEE ICIP, 2019, Ou F Z et al. PDF ↩
Blind Quality Assessment of Compressed Images via Pseudo Structural Similarity (PSS), IEEE ICME, 2016, Min X et al. PDF ↩
Blind Image Quality Estimation via Distortion Aggravation (BMPRI), IEEE TBC, 2018, Min X et al. PDF ↩
Zhang, Moorthy, A. K., Chandler, D., & Bovik, A. (2014). C-DIIVINE: No-reference image quality assessment based on local magnitude and phase statistics of natural scenes. Sig. Proc. Image Commun., 29(7). PDF ↩
Saad, M., Bovik, A. C., & Charrier, C. (2014). Blind prediction of natural video quality. IEEE Trans. on Image Proc., 23(3), 1352-1365. PDF ↩

How to live broadcast from my sofa to my grandma like a pro? (1) From FFmpeg to YouTube.

Kodsama — Wed, 27 Apr 2022 12:39:36 +0000

How to live broadcast from my sofa to my grandma like a pro? (1) From FFmpeg to YouTube.

In a short set of articles I will explain how to stream video from your sofa to end users like a pro using FFmpeg and different services. We will go briefly through the technical details and get our hands dirty with some command lines.

Here I will introduce the protocols supported by YouTube and use FFmpeg (a fantastic open source media toolbox) to stream content from point A (my computer) to the world (of which your grandma is part of!).

YouTube is an American online video sharing and social media platform owned by Google. It is by far the biggest video sharing site on the internet, but most of what is below applies to other websites such as Facebook, Vimeo, Dailymotion, Twitch, etc.

Our goal is to broadcast live content from our computer to our grandma using YouTube live.

Set up YouTube for the live broadcast

1. Get the link for your grandma

First things first, we need to find the direct link to our live so that we can send it to our grandma:

Go to YouTube.com
Click “View your channel” and select the appropriate channel
Go to youtube.com/account_advanced
Copy your Channel ID
Now, type in youtube.com/channel/PASTE YOUR CHANNEL ID HERE/live

Your permanent YouTube Live Page should look like this: https://www.youtube.com/channel/UC7qJ6z7MVPtqP8/live

2. Set up the live in YouTube

We need to set up the live streaming from the receiver side. Again, each streaming platform is slightly different, here I will walk you through YouTube

It is fairly easy:

Go to your YouTube account
From the top right, click Create (the small camera with a + sign) and “Go live”.
Follow the prompts to verify your channel (if not done yet).
You will arrive on YouTube creator, select if you want to stream now or later.
Select “streaming software” as source

On YouTube Creator we find the information of interest:

Stream URL (something of the form rtmp://[…].youtube.com/live)
Stream Key (a serie of characters)

Copy those and save them somewhere for later use.

How to Encode Videos for Video Sharing Sites

1. Install FFmpeg

First of we need to have FFmpeg installed on the computer, to install it we can follow the tutorial available online: Windows Linux MacOS

When it is installed you then need to open a terminal/Powershell session and verify it is properly installed, type in the first line below and you should see text with information about FFmpeg like in the second line:

    ffmpeg -version

    ffmpeg version x.x.x Copyright (c) 2000-20xx the FFmpeg developers [...]

The syntax of FFmpeg commands is (almost) always like this:

    ffmpeg [input options] -i [input] [output options] [output]

Below I will go through the different parts in brackets and explain how they are used.

2. Encode with FFmpeg: the [input] and [input options]

Now, we need to locate the video file we want to stream (if any) or open the device we want to stream from (our input), transform its content to suit YouTube and send it there.

If you are using a file then you should point to the file (here an mp4 file, this could be avi, mkv, etc):

    ffmpeg /myfolder/myfile.mp4

[input options] here is empty and [input] is “/myfolder/myfile.mp4”

If you want to use your computer camera (here a macbook pro) then first find a list of devices (first line) and its answer below:

    ffmpeg -f avfoundation -list_devices true -i ""

    [AVFoundation indev @ 0x139f04ff0] AVFoundation video devices
    [AVFoundation indev @ 0x139f04ff0] [0] FaceTime HD Camera
    [AVFoundation indev @ 0x139f04ff0] [1] Capture screen 0
    [AVFoundation indev @ 0x139f04ff0] AVFoundation audio devices:
    [AVFoundation indev @ 0x139f04ff0] [0] MacBook Pro Microphone

And then use the device you want as input (here 0):

    ffmpeg -f avfoundation -i "0"

[input options] here is “-f avfoundation” and [input] is “-i “0””

3. Encode with FFmpeg: the [output options]

We need to make sure that the content we input is compatible for YouTube. You can find a list of supported codecs on the YouTube wiki.

As of today (2022–04) only the H.264 video encoding with AAC or MP3 audio compression are supported.

In the example below we chose a H.264, level 4.2 for 1080p 60 FPS video with AAC sound.

The FFmpeg [output options] look like that:

    -c:v libx264 -framerate 60 -crf 18 -maxrate 50M -pix_fmt yuv420p -c:a aac -b:a 128k

Above we tell FFmpeg to use:

H.264 for the video (-c:v libx264)
60 pictures per second (-framerate 60)
video quality of 18 (-crf 18) but reduce the image quality to not go over 50 megabytes per second (-maxrate 50M | 50 MBps corresponds to level 4.2)
use image format yuv420p (-pix_fmt yuv420p)
Use AAC for audio (-c:a aac) with 128 kilobits per second (-b:a 128k)

If you want to go deeper in the different options, you can dig into the FFmpeg page for streaming or the page for parameters and protocols.

4. Encode with FFmpeg: the [output]

Now that we have the [input options] [input] and [output options] we just need to define the [output], it is simple it looks like that ([YouTube_url] and [YouTube_key] being the ones you copied from YouTube):

    -f flv [YouTube_url]/[YouTube_key]

What is flv?

flv is the keyword for FFmpeg for Real-Time Messaging Protocol (RTMP). It is a communication protocol for streaming audio, video, and data over the Internet, it was developped by Macromedia (later Adobe) to stream between Flash Player (this is where flv comes from) to a server. It is still one of the most used broadcast protocols on the internet today. These links usually start by rtmp://

Abracadabra!

Now that we have set up the whole chain, we just need to put everything together, remember FFmpeg commands look like this:

    ffmpeg [input options] -i [input] [output options] [output]

So your command would look something like that:

    ffmpeg -f avfoundation -i "0" -c:v libx264 -framerate 60 -crf 18 -maxrate 50M -pix_fmt yuv420p -c:a aac -b:a 128k -f flv rtmp://a.rtmp.youtube.com/live2/mykey

Run the command above and you should see your video preview in YouTube Studio !

Now your grandma can use the link you sent her to see your live… magic !

How bad is my video? A video quality artifacts walk-through

Kodsama — Wed, 27 Apr 2022 12:29:41 +0000

How bad is my video? A video quality artifacts walk-through

Video compression has been a hot topic for many years and now with the multiplication of content providers (read Netflix, Disney+ or others) being able to deliver high quality content at medium/low bandwidth is getting increasingly important. After all, we pay for a service and expect to get our 4Kmovies on the train right?

But as anything in life, everything comes at a cost. To be able to deliver content we need to be able to compress it, and to compress it, at some point, we usually need to drop data, and with the drop of data, the quality drops.

So how do the big guys decide how much should be dropped? A very good source of information (which I recommend to follow) has been the Netflix Tech blog.

In this (rather short) article I will go through some of the most common artifacts we could see in videos. I will complete later on with an overview of objective metrics (read without the need for human intervention) available out there to try to detect those (and actually used out there by the big guys).

What is a video artifact?

A video artifact is a noticeable distortion (for the viewer) in the quality of a video, usually due to image capture limitations or to lossy data compression (where we drop information).

In the part below I will list the most common artifacts and explain briefly some possible causes. Some are usually compression related (blocking, banding, ringing), capture related (noise, aliasing, rolling-shutter) or transmission related (ghosting, freezing, data-error)

Which artifacts do we usually see?

Noise

By far the most common artifact. Image noise is random variation of brightness (luminance) or color (chrominance) information in images. It is is usually due to electronic noise (from the image capture or transmission). Image noise can range from almost imperceptible (usually on photograph taken in good light), to radioastronomical images that are almost entirely noise, from which a small amount of information can be derived by sophisticated processing. If you want to know more, check out this Wikipedia article.
In the image below noise is the small colored dots present in the image.

Blocking

The second most common, it usually appears in videos with higher compression settings and is a consequence of block-oriented compression where we drop a little too much data. We can clearly see the blocks used during the compression process in the image below. These artifacts happen because during the compression process, the image is split into squares which are compressed (relatively) independently, the big drop in information generates big differences between these blocks which are then visible in the end result.
Some de-blocking filtering can be applied to reduce this effect but usually adds blurriness to the end result.

Banding / posterization

Banding is a subtle result of posterization, where the conversion of a continuous gradation of tone to several regions of fewer tones, causes abrupt changes from one tone to another. It may be a deliberate or an unintended artifact of color quantization (where we do not have enough possible values to represent the fine differences in colors and end up with “jumps” in color becuase of that). We usually see those during nice sunset or dark scenes, most often because the video is encoded with 8 bits and a slight change in color could not be represented correctly. Banding can be seen in the image below in the sky around the sun.

Ringing

Ringing is somewhat a blanket term for edge artifacts. It is usually induced by some image enhancement filters. Ringing includes mosquito noise (shimmering blur of dots around edges), edge enhancement artifacts (strong contrast between edge colors), overshoot, or ring-like ringing caused by the Gibbs phenomenon. In the picture below we can see the mosquito effecy shown with the white arrows as well as some edge enhancement artifacts around the black curls. The image after shows ringing effect from edge enhancement.

Aliasing

It is an effect that causes different signals to become indistinguishable (or aliases of one another) when sampled. It also often refers to the distortion or artifact that results when a signal reconstructed from samples is different from the original continuous signal. With videos it often manifests as Moiré patterns. The Moiré pattern often arises because the image contains repetitive detail which exceeds the resolution of the camera (see the door below).

Another type of aliasing could be seen when the encoding process adds new textures which are not present in the original image (as the grass below)

or even the stroboscopic effect (fun aliasing occurring when continuous cyclic motion is represented by a series of short or instantaneous samples) which can lead to fun effects like non-propelled flying helicopters:

Ghosting

This appends when a replica of a transmitted image, offset in position, is visible in an other image. It is often caused when a TV signal travels by two different paths to a receiving antenna, with a slight difference in timing or when temporal filters (often for noise reduction) keep multiple replicas of an object in the image. You can see it below with the duplicated player.

Interlace artifacts

These are artifacts created by video interlacing. When watching interlaced video on a progressive monitor with without deinterlacing, they can see sort of combing in movement between two fields of one frame.

Block loss artifacts

Data errors in the compressed bit-stream, possibly due to transmission errors appear sometime due to missing or corrupted data preventing the decoder to properly reconstruct the image. The way it shows is different between decoders are most have some kind of error-correction (where the error is compensated for using surrounding or extra data) or error-concealment mechanism (where the error is hidden by replacing the corrupted data).
It often shows with blocks on the image (as below) or replacing the coirrupted blocks with colored blocks.

Freezing

This happens when some images in a video are missing/corrupted/delayed and creates a gap in the sequence of images. Again here the way this is experienced depends on the decoder as this could be compensated using interpolation or displaying a previous frame longer than needed on the screen. Some decoders display a black or gray image instead of the missing frame. The video below contains both freezing and ghosting.

Other (not-really) artifacts

Below are some other indicator which are not really artifacts but give a good indication of a video quality and detecting these could help:

Rolling shutter

These are pretty rare and are due to limitation of the readout-rate of image sensors which end up in a distorted image. This is the because the pixels in active-pixel sensors are read line by line and by the time a line is completed the object in the next one has moved, resulting in a distorted view of the object. It can be seen in the helicopter image or the distorted train on the left of the next image.

Flickering

Flicker is a visible change in luminous flux of an illuminated light source which can be due to fluctuations of the light source itself, or due to external causes. It can also happen due to correction algorithms oscillations. You can see a light source flicker in the animation below (this specific one likely due to the frequency of electricity being lower than the capture frequency of the video)

Blur

Even if not really an artifact, it is worth to mention as it often needs to be detected and corrected. Blur is when something is unclear or less distinct, an example can be seen below.

Brightness / contrast

When brightness or contrast are off for the image then some details are lost or emphasized and the image can look strange.

Closing words

Now, I must apologize, because next time you watch a video, you will start to pinpoint those errors you did not pay attention to earlier. But now you can also show-off and be a video content professional… Happy hunting!

DEV Community: Kodsama

AWS cost last resort, the killer lambda

Setting up Billing Alarms

Using a killer lambda

General setup

Step 1: Set Up AWS Budgets and send alert to SNS

Step 2: Create the DynamoDB database

Step 3: (after creating lambdas) Give proper permissions in IAM

Recovery lambda

Step 1: Create the Lambda Function

Environment Variables

Step 2: Give the Lambda function full access in IAM

Step 3: Create a CloudWatch Event Rule

Killer lambda

Step 1: Create a Lambda Function

Environment Variables

Step 2: Give the Lambda function full access in IAM

Step 3: Blacklist the restore Lambda

Step 4: Set Up SNS to Trigger the Lambda Function

Closing Thoughts

Controlling AWS Lambda Costs

Steps to Control Costs

Setting Up Billing Alarms

Limiting Lambda Concurrency

Reduce Global AWS Service Quotas for Lambda

Set Concurrency Limits for Individual Lambda Functions

Set Rate Limits on API Gateway

Use AWS WAF to Avoid HTTP Flooding

Using a Killer Lambda

Closing Thoughts

Let ChatGPT write your best LinkedIn summary & posts

How?

A concrete example

Job application on steroids with ChatGPT

The simple steps

1. Export your LinkedIn information

2. Compile a ChatGPT request

3. Explore

A concrete example

How good is my video? A (traditional) video quality metrics survey

Executive summary

Two metrics groups

Subjective tests

Objective tests

Full reference methods

PSNR: Peak signal-to-noise ratio 5

SSIM: Structural similarity index measure 6

MS-SSIM: Multi-scale Structural Similarity for Image Quality Assessment 7

FSIM: Feature Similarity Index for Image Quality Assessment 8

IFC: Information Fidelity Criterion for Image Quality Assessment Using Natural Scene Statistics 9

NQM: Noise Quality Measure [^16]

UQI: Universal Quality Index 10

VSNR: Visual signal-to-noise ratio 11

MPQM: Moving Pictures Quality Metric 12

VIF: Visual Information Fidelity 13

DLM: Detail Loss Metric 14

MOVIE: Motion tuned spatio-temporal quality assessment of natural videos 15

ST-MAD: A spatiotemporal most apparent distortion model for video quality assessment 16

VQM: Video quality metric 18

VMAF: Video multi-method assessment fusion 19

Reduced reference methods

FSI: Free-energy principle and Sparse Representation 20

SRR: SSIM Reduced-Reference 21

ST-RRED: ST-Reduced Reference Entropic Differencing 22

No reference methods

BRISQUE: No-reference Image Quality Assessment in the Spatial Domain 24

NIQE: A Feature-Enriched Completely Blind Image Quality Evaluator 25

NBIQA: A Novel Blind Image Quality Assessment Method Based on Refined Natural Scene Statistics 26

PSS: Blind Quality Assessment of Compressed Images via Pseudo Structural Similarity 27

BMPRI: Blind Image Quality Estimation via Distortion Aggravation 28

DIIVINE: Distortion Identification-based Image Verity and Integrity Evaluation 29

BLIINDS: No-reference image quality assessment based on local

Sources

How to live broadcast from my sofa to my grandma like a pro? (1) From FFmpeg to YouTube.

How to live broadcast from my sofa to my grandma like a pro? (1) From FFmpeg to YouTube.

Set up YouTube for the live broadcast

1. Get the link for your grandma

2. Set up the live in YouTube

How to Encode Videos for Video Sharing Sites

1. Install FFmpeg

PSNR: Peak signal-to-noise ratio ⁵

SSIM: Structural similarity index measure ⁶

MS-SSIM: Multi-scale Structural Similarity for Image Quality Assessment ⁷

FSIM: Feature Similarity Index for Image Quality Assessment ⁸

IFC: Information Fidelity Criterion for Image Quality Assessment Using Natural Scene Statistics ⁹

UQI: Universal Quality Index ¹⁰

VSNR: Visual signal-to-noise ratio ¹¹

MPQM: Moving Pictures Quality Metric ¹²

VIF: Visual Information Fidelity ¹³

DLM: Detail Loss Metric ¹⁴

MOVIE: Motion tuned spatio-temporal quality assessment of natural videos ¹⁵

ST-MAD: A spatiotemporal most apparent distortion model for video quality assessment ¹⁶

VQM: Video quality metric ¹⁸

VMAF: Video multi-method assessment fusion ¹⁹

FSI: Free-energy principle and Sparse Representation ²⁰

SRR: SSIM Reduced-Reference ²¹

ST-RRED: ST-Reduced Reference Entropic Differencing ²²

BRISQUE: No-reference Image Quality Assessment in the Spatial Domain ²⁴

NIQE: A Feature-Enriched Completely Blind Image Quality Evaluator ²⁵

NBIQA: A Novel Blind Image Quality Assessment Method Based on Refined Natural Scene Statistics ²⁶

PSS: Blind Quality Assessment of Compressed Images via Pseudo Structural Similarity ²⁷

BMPRI: Blind Image Quality Estimation via Distortion Aggravation ²⁸

DIIVINE: Distortion Identification-based Image Verity and Integrity Evaluation ²⁹