Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 6 Invoke a Lambda function and parallel execution

Introduction

In part 2 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. After that, in parts 3, 4, and 5 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement wait for callback, wait for condition, and map operations.

Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why, in this part, we'll move the logic associated with each step into a separate Lambda function. An additional advantage of this approach is that it allows us to apply individual settings to those functions, like setting the right memory and granting the exact permissions they need. We'll explore how to invoke another Lambda function within the durable step. Also, we'll show how to invoke multiple Lambda functions in parallel.

Sample application with invoking another Lambda function and performing parallel execution

Let's explore invoking another Lambda function and parallel execution. We get access to them through DurableContext.

• invoke - the invoke operation calls another Lambda function and waits for its result. It supports both durable functions and standard on-demand Lambda functions. When you call ctx.invoke() or ctx.invokeAsync(), the SDK checkpoints the start of the operation, and the durable functions backend invokes the target Lambda function. On replay, the SDK returns the checkpointed result without invoking the target again.
• parallel - parallel runs multiple independent branches concurrently, each in its own child context. Branches are registered via branch() and execute immediately (respecting maxConcurrency). The operation completes when all branches finish or completion criteria are met.

I first copied the application that we created in part 2 into the aws-s3-files-lambda-durable-functions-as-orchestrator-java-25 repository. We explain step by step how to adjust our application. Please go through the content of part 2 to understand the sample application and the basic concepts of Lambda durable functions. There will be some changes in the Infrastructure as Code (IaC) part as well. I'll leave out the wait for callbacks that we implemented in part 2. You can copy this logic into the Lambda durable function orchestrator, if you will.

We'll move the business logic for 3 individual steps out of our Lambda durable function AuthorContentExtractor to 3 different Lambda functions:

1. Search for upcoming talks
2. Search for YouTube videos
3. Write author content to file

The implementation of all 3 Lambda functions is trivial. We'll focus on the change in the Lambda durable function AuthorContentExtractor and in the IaC located in the AWS SAM template.

We already showed in part 2 how to execute the first 2 Lambda functions asynchronously. This time, we'll show how to execute them in parallel. Here is how the complete code of the AuthorContentExtractor Lambda durable function looks:

public class AuthorContentExtractor extends DurableHandler<Author, AuthorContent> {

private static final String UPCOMING_TALKS_EXTRACTOR_FUNCTION_ARN  = 
System.getenv("UpcomingTalksExtractorFunctionArn");
private static final String YOUTUBE_VIDEOS_EXTRACTOR_FUNCTION_ARN  =
System.getenv("YouTubeVideosExtractorFunctionArn");
private static final String WRITE_AUTHOR_CONTENT_TO_FILE_FUNCTION_ARN  =
System.getenv("WriteAuthorContentToFileFunctionArn");


@Override
public AuthorContent handleRequest(Author author, DurableContext ctx) {
  var config = ParallelConfig.builder()
      .maxConcurrency(5)
      .nestingType(NestingType.NESTED)
      .completionConfig(CompletionConfig.allCompleted())
      .build();

  var invokeConfig=InvokeConfig.builder().build();

  var parallel = ctx.parallel("parallel-search", config);

  var upcomingTalksFuture = parallel.branch("searchForUpcomingTalks-parallel-step",
      UpcomingTalks.class, branchCtx -> {
           return branchCtx.invoke("searchForUpcomingTalks-step",           
             UPCOMING_TALKS_EXTRACTOR_FUNCTION_ARN, author,
             UpcomingTalks.class, invokeConfig);
     });

  var youtubeVideosFuture = parallel.branch("searchForYouTubeVideos-parallel-step",
      YouTubeVideos.class, branchCtx -> {
      return branchCtx.invoke("searchForYouTubeVideos-step",                
             YOUTUBE_VIDEOS_EXTRACTOR_FUNCTION_ARN, author, 
             YouTubeVideos.class, invokeConfig);
    });

  var result = parallel.get();
  var upcomingTalks=upcomingTalksFuture.get();
  var youtubeVideos= youtubeVideosFuture.get();

  var authorContent = new AuthorContent(author, upcomingTalks, youtubeVideos);

  ctx.invoke("writeAuthorContentToFile-step", WRITE_AUTHOR_CONTENT_TO_FILE_FUNCTION_ARN, 
    authorContent, Void.class, invokeConfig);

  return authorContent;
  }
}

Let's go step by step through the code. First, we pass the ARNs of all 3 Lambda functions via environment variables. We set those for the Lambda durable function AuthorContentExtractorFunction in the AWS SAM template. Also, we give our Lambda durable function the required permissions to invoke the other 3 Lambda functions:

  AuthorContentExtractorFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: AuthorContentExtractorOrchestrator
      Handler: dev.vkazulkin.handler.AuthorContentExtractor::handleRequest
      Environment:
        Variables:
          UpcomingTalksExtractorFunctionArn: !GetAtt UpcomingTalksExtractorFunction.Arn
          YouTubeVideosExtractorFunctionArn: !GetAtt YouTubeVideosExtractorFunction.Arn
          WriteAuthorContentToFileFunctionArn: !GetAtt WriteAuthorContentToFileFunction.Arn
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Sid: InvokeLambda
              Effect: Allow
              Action: lambda:InvokeFunction
              Resource:
                  - !GetAtt YouTubeVideosExtractorFunction.Arn
                  - !Sub '${YouTubeVideosExtractorFunction.Arn}:*'
                  - !GetAtt UpcomingTalksExtractorFunction.Arn
                  - !Sub '${UpcomingTalksExtractorFunction.Arn}:*'
                  - !GetAtt WriteAuthorContentToFileFunction.Arn
                  - !Sub '${WriteAuthorContentToFileFunction.Arn}:*'
....

As we invoke UpcomingTalksExtractor and YouTubeVideosExtractor Lambda functions in parallel, we create ParallelConfig. Here we set the maximum concurrency value and nesting type. There are 2 nesting types:

• Nested (the default one). Create CONTEXT operations for each branch/iteration with full checkpointing. Operations within each branch/iteration are wrapped in their own context. Observability: High - each branch/iteration appears as a separate operation in execution history. Cost: Higher - consumes more operations due to CONTEXT creation overhead. Scale: Lower maximum iterations due to operation limits
• Flat. Skip CONTEXT operations for branches/iterations using virtual contexts. Operations execute directly without individual context wrapping. Observability: Lower - branches/iterations don't appear as separate operations. Cost: ~30% lower - reduces operation consumption by skipping CONTEXT overhead. Scale: Higher maximum iterations possible within operation limits

We also set the CompletionConfig. Here, there are many possibilities from which we can choose:

• allCompleted (we use it). All items run regardless of failures. Failures are captured per-item.
• allSuccessful. All items must succeed. Zero failures tolerated.
• firstSuccessful. Complete as soon as the first item succeeds.
• minSuccessful. Complete when the specified number of items have succeeded.
• toleratedFailureCount. Complete when more than the specified number of failures have occurred.
• toleratedFailurePercentage. Complete when the failure percentage exceeds the specified threshold (0.0 to 1.0).

Next, we create an InvocationConfig for each individual Lambda function invocation that we can optionally set. Here we go with the default configuration. But we additionally set the tenant ID, which is used to isolate execution state for different tenants. It's required when invoking multi-tenant Lambda functions. So far, we covered the following parts:

 var config = ParallelConfig.builder()
      .maxConcurrency(5)
      .nestingType(NestingType.NESTED)
      .completionConfig(CompletionConfig.allCompleted())
      .build();

  var invokeConfig=InvokeConfig.builder().build();

Next, let's implement the Lambda function invocation in parallel:

  var parallel = ctx.parallel("parallel-search", config);

  var upcomingTalksFuture = parallel.branch("searchForUpcomingTalks-parallel-step",
      UpcomingTalks.class, branchCtx -> {
           return branchCtx.invoke("searchForUpcomingTalks-step",           
             UPCOMING_TALKS_EXTRACTOR_FUNCTION_ARN, author,
             UpcomingTalks.class, invokeConfig);
     });

  var youtubeVideosFuture = parallel.branch("searchForYouTubeVideos-parallel-step",
      YouTubeVideos.class, branchCtx -> {
      return branchCtx.invoke("searchForYouTubeVideos-step",                
             YOUTUBE_VIDEOS_EXTRACTOR_FUNCTION_ARN, author, 
             YouTubeVideos.class, invokeConfig);
    });

  var result = parallel.get();
  var upcomingTalks=upcomingTalksFuture.get();
  var youtubeVideos= youtubeVideosFuture.get();

  var authorContent = new AuthorContent(author, upcomingTalks, youtubeVideos);

First, we invoke the parallel method on the DurableContext and pass the ParallelConfig. We get back the ParallelDurableFuture, with which we can start a branch. To do this, we invoke the branch method for each individual Lambda invocation and specify the result type. In each branch, we call the invoke method on the BranchContext. Here we pass the following parameters:

• name - the unique operation name within this context.
• functionName - the ARN or name of the Lambda function to invoke.
• payload - the input payload to send to the target function.
• resultType - the result class for deserialization.

We get back the DurableFuture of the resultType (in our case UpcomingTalks or YouTubeVideos).

When we invoke the blocking get method on the ParallelDurableFuture to get the results with respect to the ParallelConfig. Then, we invoke the get method on DurableFuture to get the individual results of both Lambda invocations. Finally, we create the _ AuthorContent _ with these results.

The last step is to invoke the Lambda function, which is responsible for writing the author content into the file:

 ctx.invoke("writeAuthorContentToFile-step", WRITE_AUTHOR_CONTENT_TO_FILE_FUNCTION_ARN, 
    authorContent, Void.class, invokeConfig);

As we explained in part 2, we use S3 Files here. The whole part on how to configure S3 Files and give the Lambda function access to it has now moved from the AuthorContentExtractorOrchestrator durable Lambda function to the WriteAuthorContentToFileFunction Lambda function. See the SAM template for more detail.

Now we can build and package our application with mvn clean package and deploy it with sam deploy. The deployment process can take up to 10 minutes because of the creation and mounting of S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractorOrchestrator function, and go to the "Test" tab.

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

Let's also look at event history:

We see that this matches what we implemented. Our Lambda durable function invokes in parallel 2 Lambda functions to search for YouTube videos and upcoming talks (each in an individual branch). Then, it awaits the results of both invocations because we defined in the parallel configuration that we wait until all invocations are completed. Then the last Lambda function to write the author content into the file gets invoked.

One general question at the end: is the Lambda function invocation from another Lambda function considered an anti-pattern? Or should we put an API Gateway in between instead? My answer is that with built-in fault-tolerance for AWS Lambda durable functions, it's now safe to invoke a Lambda function directly.

Conclusion

In this part of the series, we moved the logic associated with each step into a separate Lambda function. We explored how to invoke another Lambda function within the durable step. Also, we showed how to invoke multiple Lambda functions in parallel.

In the next part of the series, we'll transform our application into an agentic one. Currently, the search for YouTube and upcoming talks returns static content. We'll replace it with the Amazon Bedrock AgentCore Gateway Web Search Tool. As it's exposed via MCP, we need an MCP client to connect to it. For that, we'll use the Spring AI framework. First, we still use AWS Lambda to host the AI agents. In later parts, we'll show how to host those AI agents on the Amazon Bedrock AgentCore Runtime instead.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.