Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 2 Durable synchronous and asynchronous steps

Introduction

In part 1, we introduced what we'll cover in this series and our sample application. We also gave a short introduction to the services that we'll use throughout this series: AWS Lambda durable functions and Amazon S3 Files. In this part of the series, we'll explore how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously.

First implementation attempt of our sample application

Throughout this article series, we'll develop a simple application to show how to build multi-step applications and orchestrate workflows using AWS Lambda durable functions. This simple application provides the functionality to extract the content for the provided user. To this content belongs searching for YouTube videos of the authors and their upcoming talks. We'll store the result on Amazon S3 Files in JSON format. As this operation may potentially take time, we'll additionally provide access to the author content through the Amazon API Gateway:

We'll start with the search for YouTube videos and upcoming talks of the author synchronously. For the sake of simplicity, we won't implement any persistent layer and store the static content in memory. We'll store the result on Amazon S3 Files in JSON format.

You can find the code examples in my aws-s3-files-lambda-durable-functions-java-sdk repository. We'll go through all the examples in more detail in the subsequent articles.

First of all, let's declare some important dependencies in pom.xml. The most important one is the AWS Lambda Durable Execution SDK for Java:

<dependency>
    <groupId>software.amazon.lambda.durable</groupId>
    <artifactId>aws-durable-execution-sdk-java</artifactId>
    <version>2.0.0</version>
</dependency>

Why do we need such an SDK? Under the hood, durable functions are regular Lambda functions using a checkpoint/replay mechanism to track progress and support long-running operations through user-defined suspension points, commonly referred to as durable execution. After your function resumes from a pause or interruption, the system performs replay. During replay, your code runs from the beginning but skips over completed checkpoints, using stored results instead of re-executing completed operations. This replay mechanism ensures consistency while enabling long-running executions.

To harness this checkpoint-and-replay mechanism in your applications, Lambda provides a durable execution SDK. The SDK abstracts away the complexity of managing checkpoints and replay, exposing simple primitives called durable operations that you use in your code. The SDK integrates seamlessly with your existing Lambda development workflow.

To provide the implementation of our Lambda durable function, we need the class to extend DurableHandler and implement only the method handleRequest. Let's do it for our AuthorContentExtractor Lambda durable function:

public class AuthorContentExtractor extends DurableHandler<Author, AuthorContent> implements AbstractAuthorContentExtractor {

@Override
public AuthorContent handleRequest(Author author, DurableContext ctx) {
   var config = this.getStepConfig();       
   var upcomingTalks= ctx.step("searchForUpcomingTalks-step", UpcomingTalks.class, stepCtx -> this.searchForUpcomingTalks(), config);
   var youtubeVideos= ctx.step("searchForYouTubeVideos-step", YouTubeVideos.class, stepCtx -> this.searchForYouTubeVideos(), config);

   var authorContent = new AuthorContent(author, upcomingTalks, youtubeVideos);

   ctx.step("writeAuthorContentToFile-step", Void.class, stepCtx -> this.writeAuthorContentToFile(authorContent), config);      

    return authorContent;
   }
}

Let's go step by step through what's happening here. First of all, we also implement the AbstractAuthorContentExtractor interface. We place some common methods there that we'll use when improving our application.

Now let's introduce the concept of durable steps. Steps run business logic with built-in retries and automatic checkpointing. Each step saves its result, so your function resumes from the last completed step after an interruption. With the SDK, you wrap your Lambda event handler, which then provides a DurableContext alongside your event. This context gives you access to durable operations like steps. You write your function logic as normal sequential code, but instead of calling services directly, you wrap those calls in steps for automatic checkpointing and retries. When you need to pause execution, you add waits that suspend your function without incurring charges. We'll show it in the next part of the series. The SDK handles all the complexity of state management and replay behind the scenes, so your code remains clean and readable.

Here is the picture, which describes the concept that I took from this source:

We wrapped several operations into separate durable steps:

• searchForUpcomingTalks-step
• searchForYouTubeVideos-step
• writeAuthorContentToFile-step

Let's look into the searchForUpcomingTalks-step first:

var upcomingTalks= ctx.step("searchForUpcomingTalks-step", UpcomingTalks.class, 
stepCtx -> this.searchForUpcomingTalks(), config);

To create a step, we need to give it a name and the result type. We also have to give it the function to execute, receiving a StepContext. The return type of the function should be the same as the result type. In our case, we execute the searchForUpcomingTalks function, which returns the static list of the upcoming talks. Optionally, we can also pass the StepConfig. In our case, we create it like this:

var stepConfig = StepConfig.builder()
    .semanticsPerRetry(StepSemantics.AT_LEAST_ONCE_PER_RETRY)                 
    .retryStrategy(RetryStrategies.exponentialBackoff(                                
      3, // max attempts
      Duration.ofSeconds(2),   // initial delay  
      Duration.ofSeconds(30),   // max delay
      2.0,                      // backoff multiplier
      JitterStrategy.FULL))
   .build();

We create the StepConfig by passing the following:

• retry strategy: fixed delay, linear backoff, or exponential backoff. We pass the latter here.
• semantics per retry: at-least-once delivery (default, which we also pass). The step may be re-executed if interrupted. START checkpoint is fire-and-forget. At-most-once delivery per retry attempt. START checkpoint is awaited before user code runs.
• custom serializer for the step with serDes method. We don't provide it here, as we're happy with the default one.

searchForYouTubeVideos-step step works the same. After we have collected the result of the search for the upcoming talks and YouTube videos, we use the writeAuthorContentToFile-step step to serialize the result to JSON and write it to the file:

public default Void writeAuthorContentToFile(AuthorContent authorContent) {
  var authorContentAsJson = OBJECT_MAPPER.writeValueAsString(authorContent);
  var fileName= authorContent.author().firstName()+"-"+authorContent.author().lastName()+".json";
  Path path = Paths.get(WORKSPACE_MOUNT, fileName);
  byte[] strToBytes = authorContentAsJson.getBytes();
  Files.write(path, strToBytes);
  return null;
}

We use the standard Java Path API to do it. This is because we use Amazon S3 Files, and it supports POSIX. Strictly speaking, we could also use S3 here, but I wanted to show the functionality of S3 Files.

The directory where we store the file is passed as an environment variable WORKSPACE_MOUNT. In our case, it is /mnt/workspace. See the Infrastructure as Code explanation below, especially the LocalMountPath setting of the file system configuration for the Lambda function.

For infrastructure as code (IaC), I use AWS SAM, and you can find it here. To indicate that our Lambda function is durable, we need to provide some additional durable config properties:

  AuthorContentExtractorFunction:
    Type: AWS::Serverless::Function
    ...
    Properties:
      FunctionName: AuthorContentExtractor
      Handler: dev.vkazulkin.handler.AuthorContentExtractor::handleRequest
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
        ...

• Execution timeout. The execution timeout controls how long a durable execution can run from start to completion. This is different from the Lambda function timeout, which controls how long a single function invocation can run. A durable execution can span multiple Lambda function invocations as it progresses through checkpoints, waits, and replays. The execution timeout applies to the total elapsed time of the durable execution, not to individual function invocations. Set the Execution timeout value in seconds (default: 86400 seconds / 24 hours, minimum: 60 seconds, maximum: 31536000 seconds / 1 year).
• Retention period. The retention period controls how long Lambda retains execution history and checkpoint data after a durable execution completes. This data includes step results, execution state, and the complete checkpoint log. After the retention period expires, Lambda deletes the execution history and checkpoint data. You can no longer retrieve execution details or replay the execution. Set the Retention period value in days (default: 14 days, minimum: 1 day, maximum: 90 days).

For a very detailed explanation of how to create S3 Files and mount them to the Lambda function, please read the brilliant article Lambda Just Got a File System. I Put AI Agents on It by Eric Johnson. I mostly copied the IaC part from it and adjusted it to my needs. I only describe the main steps here:

First, we reference the networking stack, which we place in a separate file. Then we create:

• VPC networking for Lambda functions using S3 Files.
• VPC with private subnets (for Lambda + mount targets).
• Security groups for NFS traffic.
• Public subnet with a NAT gateway. We'll need them for internet access to Bedrock and Bedrock AgentCore Web Search Tool. In later parts, we'll use both services.

  NetworkingStack:
    Type: AWS::Serverless::Application
    Properties:
      Location: src/main/resources/stacks/network.yaml

Next, we need to create an S3 Bucket:

 WorkspaceBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName : vadym-s3-files-workspace
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled

Please don't forget to rename the S3 Bucket. After that, we need to create S3 Files:

  S3FileSystem:
    Type: AWS::S3Files::FileSystem
    Properties:
      Bucket: !GetAtt WorkspaceBucket.Arn
      RoleArn: !GetAtt S3FilesRole.Arn
      AcceptBucketWarning: true

Here we reference an already created S3 Bucket. We also need to profile the appropriate IAM Role (see the SAM template for the IaC). Next, we need to create 2 different mount targets for the S3 File. They should use private subnets in different availability zones. Here is an example of one such mount target:

  MountTargetA:
    Type: AWS::S3Files::MountTarget
    Properties:
      FileSystemId: !GetAtt S3FileSystem.FileSystemId
      SubnetId: !GetAtt NetworkingStack.Outputs.PrivateSubnetAId
      SecurityGroups:
        - !GetAtt NetworkingStack.Outputs.MountTargetSGId

Next, we need to create an S3 Files access point:

  S3FilesAccessPoint:
    Type: AWS::S3Files::AccessPoint
    Properties:
      FileSystemId: !GetAtt S3FileSystem.FileSystemId
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        Path: /lambda
        CreationPermissions:
          OwnerUid: '1000'
          OwnerGid: '1000'
          Permissions: '755'

The CreationPermissions property is crucial. It auto-creates the /lambda directory within our S3 bucket with the right ownership when a client first connects. Without it, the root directory is owned by root (UID 0), and Lambda (running as UID 1000 through the access point) can’t create subdirectories.

Lastly, we need to provide configuration and give permissions to our Lambda function to use S3 Files:

  AuthorContentExtractorFunction:
    Type: AWS::Serverless::Function
    DependsOn:
      - MountTargetA
      - MountTargetB
    Properties:
      FunctionName: AuthorContentExtractor
      Handler: dev.vkazulkin.handler.AuthorContentExtractor::handleRequest
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
      VpcConfig:
        SecurityGroupIds:
          - !GetAtt NetworkingStack.Outputs.LambdaSGId
        SubnetIds:
          - !GetAtt NetworkingStack.Outputs.PrivateSubnetAId
          - !GetAtt NetworkingStack.Outputs.PrivateSubnetBId
      FileSystemConfigs:
        - Arn: !GetAtt S3FilesAccessPoint.AccessPointArn
          LocalMountPath: /mnt/workspace
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Sid: MountS3Files
              Effect: Allow
              Action:
                - s3files:ClientMount
                - s3files:ClientWrite
                - s3files:ClientRootAccess
              Resource: !GetAtt S3FileSystem.FileSystemArn

Here we provide the VPC and file system configuration (including local mount path) and the policy for our Lambda function to use the S3 Files.

Now we can build and package our application with mvn clean package and deploy it with sam deploy. The deployment process can take up to 10 minutes because of the creation and mounting of S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractor function, and go to the "Test" tab:

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

• Input and output as JSON
• Logger output
• Details like durable operations (with individual inputs and outputs of each step) and event history. If something goes wrong, the step status will be set to failed.

In our case, we invoked the durable function directly from the Lambda console. As an alternative, we can also use the Lambda SDK for it. We can also put an API Gateway in front of it if you wish and even invoke the Lambda function asynchronously.

We can also see the JSON file with author content in the S3 Bucket that we created. Alternatively, I implemented a GetAuthorContentResult Lambda function with the name GetAuthorContentResult. This Lambda function takes the author's first and last name and streams the content of the JSON file from the S3 bucket. This JSON file is stored in the subdirectory /lambda of our created S3 bucket. I also created and put an API Gateway in front of this Lambda function. Please use the /result/{firstname}/{lastname} HTTP GET endpoint for it, for example /result/Vadym/Kazulkin. Be aware that it takes several minutes for the file to appear in the S3 Bucket after being written to S3 Files. If you need the result immediately, you can rewrite this Lambda to retrieve the result from S3 Files instead.

This was a very simple application to explore the durable steps with the AWS Lambda Durable Execution SDK for Java. Let's improve our application a bit. Until now, we searched for the upcoming talks and YouTube videos sequentially. But we can do it in parallel. For that, we can use the stepAsync operation instead of step. I provided a separate implementation, AsyncAuthorContentExtractor, and the Lambda durable function with the name AsyncAuthorContentExtractor for it:

var upcomingTalksFuture= ctx.stepAsync("searchForUpcomingTalks-async-step", UpcomingTalks.class, 
stepCtx -> this.searchForUpcomingTalks(), config);
var youtubeVideosFuture= ctx.stepAsync("searchForYouTubeVideos-async-step", YouTubeVideos.class, 
stepCtx -> this.searchForYouTubeVideos(), config);

var upcomingTalks=upcomingTalksFuture.get();
var youtubeVideos= youtubeVideosFuture.get();

Now, stepAsync invocation returns a DurableFuture of the result type - a future representing the step result. This operation is non-blocking; we only block when invoking the get method on the result. Everything else remains the same in this asynchronous implementation as it was in the synchronous one.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. In the next part, we'll extend our application by adding and implementing wait and callback operations to it. Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why in the later parts, we'll move the logic associated with each step into a separate Lambda function. We'll explore how to invoke another Lambda function within the durable step. We'll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.