DEV Community: Igvir

AWS Lambda Functions Failure Management

Igvir — Fri, 10 May 2024 23:25:21 +0000

In AWS Lambda development, failure management is a crucial aspect to ensure the reliability and robustness of serverless applications. Here's how invocation failures are captured and handled, as well as how functions within distributed transactions handle failures, and how the system ensures duplicate events do not change the outcome:

Invocation Failure Capture and Handling:

Logging and Monitoring: Lambda functions typically integrate with CloudWatch for logging and monitoring. Invocation failures are captured in CloudWatch Logs, providing detailed insights into the nature of the failure, including any error messages or stack traces.

Dead Letter Queues (DLQ): Lambda allows configuring Dead Letter Queues for asynchronous invocations. When a Lambda function fails to process an event, the event payload can be automatically redirected to a designated DLQ. This enables further analysis or processing of failed events without losing them.

Retry Mechanisms: Lambda supports automatic retries for asynchronous invocations in case of transient failures. Developers can configure the number of retries and the backoff interval between retries to accommodate different failure scenarios. This helps in handling temporary issues like network timeouts or resource constraints.

Handling Failures Within Distributed Transactions

Transactional Processing: Lambda functions within a distributed transaction are designed to participate in ACID-compliant transactions using AWS Step Functions or similar orchestration services. In case of failure during transactional processing, the transaction coordinator ensures proper rollback of any changes made by the participating functions.

Compensation Logic: Each function within a distributed transaction implements compensation logic to revert any partial changes in case of failure. This ensures that the system maintains consistency even if one or more functions encounter errors during execution.

Idempotent Operations: Functions are designed to perform idempotent operations, meaning the same function invocation with the same input results in the same outcome regardless of how many times it's invoked. This ensures that even if a function is retried due to failure or duplicate events, it doesn't lead to unintended side effects.

Ensuring Outcome Consistency with Duplicate Event Handling:

Idempotent Processing: Lambda functions are designed to handle duplicate events gracefully by implementing idempotent processing logic. Before processing an event, functions can check whether the event has already been processed based on a unique identifier or by maintaining state information.

Event Deduplication: Event sources like S3, SNS, or Kinesis often provide mechanisms for event deduplication. Lambda functions can leverage these features to ensure that duplicate events are not processed multiple times, maintaining the desired outcome consistency.

Idempotent Data Updates: In scenarios where Lambda functions update data in external systems, they use idempotent data update techniques such as conditional writes or versioning. This ensures that even if the same update operation is performed multiple times due to duplicate events, it doesn't lead to inconsistent data states.

Here is another simplified Java sample code demonstrating how you might handle Lambda invocation failures, distributed transactions, and duplicate event handling using AWS Lambda:

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;

public class MyLambdaFunction implements RequestHandler<S3Event, String> {

    @Override
    public String handleRequest(S3Event event, Context context) {
        // 1. Invocation Failure Capture and Handling
        try {
            // Process S3 event
            processS3Event(event);
            return "Success";
        } catch (Exception e) {
            // Log failure to CloudWatch
            context.getLogger().log("Error processing S3 event: " + e.getMessage());
            // Redirect failed event to Dead Letter Queue
            // (DLQ configuration not shown here)
            return "Failure";
        }
    }

    // Sample method to process S3 event
    private void processS3Event(S3Event event) {
        // Your processing logic here
    }

    // Sample method to handle distributed transactions
    private void handleDistributedTransaction() {
        try {
            // Begin distributed transaction

            // Perform operations with ACID compliance
            // Rollback transaction on failure
        } catch (Exception e) {
            // Handle transaction failure
        }
    }

    // Sample method to handle duplicate event handling
    private boolean isEventProcessedBefore(String eventId) {
        // Check if the event ID has been processed before
        // Return true if already processed, false otherwise
        return false;
    }
}

In this sample:

The handleRequest method handles the incoming S3 events. Any exceptions thrown during event processing are caught and logged, and the failed events can be redirected to a Dead Letter Queue for further analysis.
The handleDistributedTransaction method demonstrates how to handle distributed transactions within the Lambda function. Transactional processing logic can be implemented here, ensuring rollback in case of failure.
The isEventProcessedBefore method checks if an event with a specific ID has been processed before, implementing duplicate event handling logic. This method can be used to ensure idempotent processing of events, preventing duplicate events from causing unintended side effects.

Please note that this code is a simplified example for demonstration purposes and may need to be adapted to fit your specific use case and AWS Lambda configuration. Additionally, you'll need to add AWS SDK dependencies to interact with AWS services like S3, Lambda, and CloudWatch. Remember to configure appropriate IAM roles and permissions for your Lambda function to interact with other AWS services like CloudWatch, Dead Letter Queue, etc.

Below is a more detailed example demonstrating how you might handle distributed transactions within an AWS Lambda function using Java with the AWS SDK for Java. The code uses AWS DynamoDB as the data store for the transaction. The Lambda function performs a simple transactional operation of transferring funds between two accounts.

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.amazonaws.services.dynamodbv2.document.DynamoDB;
import com.amazonaws.services.dynamodbv2.document.TableWriteItems;
import com.amazonaws.services.dynamodbv2.document.spec.TransactWriteItemsSpec;
import com.amazonaws.services.dynamodbv2.document.utils.ValueMap;

public class MyLambdaFunction implements RequestHandler<S3Event, String> {

    private final AmazonDynamoDB dynamoDBClient = AmazonDynamoDBClientBuilder.defaultClient();
    private final DynamoDB dynamoDB = new DynamoDB(dynamoDBClient);
    private final String sourceAccountId = "sourceAccountId";
    private final String destinationAccountId = "destinationAccountId";

    @Override
    public String handleRequest(S3Event event, Context context) {
        try {
            // Begin distributed transaction
            handleDistributedTransaction();
            return "Success";
        } catch (Exception e) {
            // Log failure to CloudWatch
            context.getLogger().log("Error processing transaction: " + e.getMessage());
            // Rollback transaction on failure
            return "Failure";
        }
    }

    // Sample method to handle distributed transactions
    private void handleDistributedTransaction() {
        try {
            // Create TransactWriteItemsSpec for the transaction
            TransactWriteItemsSpec writeItemsSpec = new TransactWriteItemsSpec();

            // Update operation for deducting amount from source account
            writeItemsSpec.addUpdateItem(new TableWriteItems("Account")
                    .withUpdateExpression("SET balance = balance - :amount")
                    .withConditionExpression("balance >= :amount")
                    .withValueMap(new ValueMap().withNumber(":amount", 100))
                    .withPrimaryKey("accountId", sourceAccountId));

            // Update operation for adding amount to destination account
            writeItemsSpec.addUpdateItem(new TableWriteItems("Account")
                    .withUpdateExpression("SET balance = balance + :amount")
                    .withValueMap(new ValueMap().withNumber(":amount", 100))
                    .withPrimaryKey("accountId", destinationAccountId));

            // Execute the transaction
            dynamoDB.transactWriteItems(writeItemsSpec);
            System.out.println("Transaction successful");
        } catch (Exception e) {
            // Handle transaction failure
            System.err.println("Transaction failed: " + e.getMessage());
            throw e; // Rethrow exception for Lambda to handle
        }
    }
}

Please notice that:

The handleDistributedTransaction method performs a distributed transaction using DynamoDB's transactWriteItems operation. It consists of two update operations: one deducting the amount from the source account and another adding the same amount to the destination account.
Conditional expressions are used to ensure that the source account has sufficient funds to perform the transfer.
In case of a transaction failure (e.g., insufficient funds, network issues), the method catches the exception, logs it, and rethrows it for AWS Lambda to handle. This allows Lambda to automatically retry the function according to its error handling behavior.

Make sure to replace "sourceAccountId" and "destinationAccountId" with actual account identifiers in your DynamoDB table.

By implementing these practices, AWS Lambda development ensures robust failure management, reliable distributed transaction processing, and consistent outcomes even in the presence of duplicate events.

Using Hash Codes as Unique Identifiers

Igvir — Fri, 24 Feb 2023 10:31:00 +0000

Introduction

In this article, we will cover a common use case for software developers: Using and generating Unique Identifiers.

Let´s start with a definition:
What is an identifier? The simplest explanation usually comes from Wikipedia:

An identifier is a name that identifies (that is, labels the identity of) either a unique object or a unique class of objects, where the "object" or class may be an idea, physical countable object (or class thereof), or physical noncountable substance (or class thereof). The abbreviation ID often refers to identity, identification (the process of identifying), or an identifier (that is, an instance of identification). An identifier may be a word, number, letter, symbol, or any combination of those.

Unique identifiers (UIDs), or identifiers, can be a string value or an integer that we use to address unique resources in a domain. Consumers (Systems or Users that need to refer to that resource) then use these identifiers to fetch the resource from a collection of resources. Without a unique identifier, the simple task of separating the resources and invoking those resources is almost impossible.

We use Identifiers to refer to database structural elements, like the names of tables, files in a collection, or elements in a set.
Another important concept here is Metadata. And again from Wikipedia:

Metadata is "data that provides information about other data", but not the content of the data, such as the text of a message or the image itself.
As we will see in this article, metadata and identifiers can be related for different purposes, including generating a Unique Identifier (UID).

Use Metadata to create Unique Identifiers

The metadata can be generated using algorithms, an example of this is a Hashing algorithm. Hashing consists of converting a general string of information into metadata that represent it. This is done to scramble the data so that it completely transforms the original value, making the hashed value utterly different from the original while also describing it in a certain way, to the point of identifying it.
Hans Peter Luhn invented Hashing. He was a German researcher and scientist who worked at IBM. During his professional life, he created over 80 patents in the field of Computer Science and Information Science.

Hashing is used for various functions, but here we are especially interested in its ability to verify the integrity of an object without having to recognize its content. Two files can be compared for equality easily through hashing. There is no need to open the two documents individually. Hashing compares them byte-by-byte and the computed hash value instantly tells if they are distinct.

Are Hash Codes Unique Identifiers?

A hash code is of a fixed length, so from a mathematical point of view it cannot be unique for all possible inputs. But all such hash functions are carefully designed to minimize the probability of a collision (two distinct files with the same hash value).

Scott Chacon and Ben Straub in "Pro Git" book talk about Hashing collision options using SHA-1 as UID:

If all 6.5 billion humans on Earth were programming, and every second, each one was producing code that was the equivalent of the entire Linux kernel history (6.5 million Git objects) and pushing it into one enormous Git repository, it would take roughly 2 years until that repository contained enough objects to have a 50% probability of a single SHA-1 object collision.

Therefore we can say that in most practical cases it is possible to use Hashing algorithms to generate Unique Identifiers.

Using Hashing to Create a Unique File Names

A simple use case where a Unique Identifier is needed is the naming process for an Object Storage. For this example, we assume that objects are immutable so if the object changes a new hashing Identifier will be associated with the new version. The idea is simple we will create a Hash code for every object and use it as a Unique Identifier of that object version.

Prerequisites

We use Java for our sample code but it could be easily ported to other programming languages. To run the file you will need to install a Java SDK. If you need it, you can see How to download and install JDK.

Hands-on with Code

The code could be separated into three steps:

Select the algorithm
Parse file(s)
Generate the UID

Step 1: The Algorithm

There are three main options when you think about the Hashing algorithm you want to use: MD5, SHA-1, or SHA-256. A Hash code is fixed length, MD5 generates a 128 bits code, SHA-1 is 160 bits and SHA-256 is 256 bits. Longer numbers minimize the likelihood of a collision of generated codes.

Federal Information Processing Standards define Secure Hash Standard (SHS) FIPS PUB 180-4 names seven algorithms: SHA-1, SHA-224, SHA-256,
SHA-384, SHA-512, SHA-512/224, and SHA-512/256.

In 2017, researchers were able to produce two different documents with the same SHA-1 hash. It took them 6610 years of processor time to do it.

We use java.security.MessageDigest class and SHA-1 Algorithm in this example.

Step 2: Parse File(s)

As mentioned earlier, we use java.security.MessageDigest class and SHA-1 Algorithm in this example. Message digests are secure one-way hash functions that take arbitrary-sized data and output a fixed-length hash value.

The code creates a new file input stream for reading the file content. The data is processed using the update method call. Once all the data has been updated, the digest methods should be called to complete the hash computation.

See the code:

Step 3: Generate the UID

The digest method returns an array of bytes in decimal format. We need to convert it to hexadecimal format before call toString() method to finally to get an UID based on file data.

If the file changes in any way the resultant hash code will be different. You can handle those differences according to the context as a new version or as an integrity check failure.

Complete Sample Code

The complete code can be found bellow:

Conclusions

We have made a tour of the concepts associated with the generation of Hash codes and their applications for the generation of Unique identifiers for objects in our application domain.
The main strength of Hashing algorithms is that they offer, at the same time, the possibility of generating an integrity verification code and an identifier reliable enough to be the unique identifier. Hashing libraries are widely available as standard in practically all programming languages, so their use is very feasible and quick to implement.

Although in the example we have used the case of generating filenames, many other uses of Hashing algorithms are well known. Just to name a few we can mention:

Database indexing
Password storage
Data compression
Search algorithms
Cryptography

Credits

Cover photo by George Prentzas on Unsplash
Gupta, L. (2022, January 25). Java File Checksum. Howtodoinjava. Retrieved February 18, 2033, from https://howtodoinjava.com/java/java-security/sha-md5-file-checksum-hash/

AWS Certified Professional Challenge 2023

Igvir — Sat, 04 Feb 2023 18:16:59 +0000

Are you ready?

A new challenge for those interested in AWS certification has already been announced. This time it's the Professional Challenge. A certification exam preparation program with advanced training including live and on-demand Twitch sessions with AWS experts.

If you join the Get AWS Certified: Professional Challenge before April 28, they will send you a 50% discount coupon for the professional level certification exam. The options are:

AWS Certified Solutions Architect - Professional
AWS Certified DevOps Engineer - Professional

For this challenge, the exam must be taken no later than May 31, 2023.
See program conditions

I want to be a community builder

Igvir — Sun, 06 Feb 2022 13:46:07 +0000

At the end of last year, 2021, I received an email with an invitation: Would you like to be an AWS community builder?

But, What is AWS Community Builder program?

The AWS Community Builders program offers technical resources, mentorship, and networking opportunities to AWS technical enthusiasts and emerging thought leaders who are passionate about sharing knowledge and connecting with the technical community.
source: https://aws.amazon.com/developer/community/community-builders/

It was an excellent and maybe logical question after a year of work with AWS services, not only to get my AWS Certification but to improve my career with products and services that are completely aligned with my path. Last year, I enjoyed working and learning about AWS, and after all that joy and work, it could be natural to start thinking about something like the AWS Community Builder Program, even when it represents some challenges.

So, back to the question, "Would you like to be an AWS Community Builder?", I think I should say:
Yes, I want to be an AWS Community builder.

Do Community builders get paid?
No, they do not get paid but they are rewarded with some AWS special gifts, most of them a surprise.

To start the process I need to set a personal goal of how much knowledge I want to share through blogs, videos, etc. and I have to complete an application. Unfortunately, the process is closed now until mid-2022. So during this time I will be working on my goal and sharing my work here.