DEV Community: AWS Heroes

You're still the bottleneck

Allen Helton — Tue, 22 Sep 2026 14:33:41 +0000

I went to a conference a couple of weeks ago. During the keynote, the speaker asked the audience "how many of you have 40 or more agents running at any given time?" About a dozen hands went up. He then asked "How many of you have 30 or more agents running?" Another dozen. This continued down to 20, 15, 10, 5, and eventually 1. Each time he asked, another 15-25 hands went up in the air.

It wasn't until he got to 1 that I sheepishly put my hand in the air. Luckily I wasn't alone, but it immediately got me thinking. I think we substituted one bad practice for another.

Remember a few months ago when tokenmaxxing was a thing? If you don't, good. You missed a terrible practice. People were taking pride in the amount of tokens they were spending regardless of the value they were getting. I still don't really understand why that was a desirable thing.

Tokenmaxxing has settled down significantly, but I fear that it's been replaced by something equally as bad: agentmaxxing.

A quick Google search will show you there's no consensus on what agentmaxxing means (honestly I thought I made it up until I Googled it). But what I mean when I say it is that developers are taking pride in the number of agents they have running at any given time. Jumping back to that question in the conference keynote, people are running upwards of 40 agents at any given time. 40?

So I asked the person who delivered the keynote how many work streams 40 agents represent. And he told me about 4.

Ok that makes so much more sense. He has four projects he's working in parallel, each one with a single orchestrator and about 9 subagents each actually doing the coding. That's better, but still a problem.

You can't multitask like you think you can

I have bad news. You can't multitask. When you think you're multitasking, you're actually context switching really fast.

The more inputs that demand your attention, the worse it gets. I've had times where I'm talking to my wife, one of my kids starts pulling on my shirt to show me something, the other kid starts crying in the other room, while the dog starts scratching at the door to go outside. At best, I'm going to somehow defer three of them to later and work through them sequentially. At worst, I'm going to walk into the other room and yell into a pillow because I'm overstimulated. 😅

Now, imagine that was four agents reporting back at the same time asking for approval or a design decision or reporting a CI failure on four different projects. With each task that lands back on your plate, you have to reconstruct the mental model of what that app is, what you're trying to do, where the agent left off, then you can act.

Would you be able to handle each one appropriately? Probably not.

Realistically, you'll probably hit the accept button without reading the context of the ask and tell your agents to just "fix it". That's what anybody would do when they make themselves the approval bottleneck across multiple unrelated contexts.

When you start blindly approving tasks, you're inadvertently delegating the decision-making to the model. At that point, what are you there for?

You are the decision maker

Code is cheaper than it ever has been. As a developer, the amount of your value that comes from typing code is dwindling. And if that was the only thing you had to do as a dev, the answer to "will AI take our jobs" would be a whole heck of a lot scarier.

Our value (among other things) comes from decision making. Weighing the risks, evaluating tradeoffs, and a nose for what looks right, to name a few. Some decisions are bigger than others, and the important ones need your undivided attention.

I love AI, but it makes it way too easy to make bad decisions. And you often won't know about it until you're fighting a production issue when it's significantly more expensive to change. If an agent spits out a 1,000 word response after a long-running task, your task is to make an informed decision using the provided context, ask questions, and think two steps ahead.

If you glance, hit accept, tab, glance again, and hit accept again, then you aren't a decision maker, you're a rubber stamp with a keyboard.

Attention is your greatest resource

I don't want you reading this and your takeaway being "Allen says 40 agents is too many." That's not it at all. I'm not advocating for anything like that.

What I am advocating for is not spreading yourself too thin. People have been saying for years that attention is one of the scarcest and most important resources we have.

In 2026, everything wants your attention. Humans have gotten really good at creating things that steal your attention away from something else. With coding agents, it takes almost 0 effort to bring in another work stream so you can "force multiply yourself." Just open another terminal, type in claude and you're (basically) done.

Protect it. Treat your attention like the scarce resource it is. Keep the decision-heavy workstreams to a minimum, just 1 or 2. Then run as many low-judgment streams as you want next to them. But keep your attention on the tasks that need it. Don't split your attention across decisions, because when everything is a priority, nothing is.

Next time somebody asks you how many agents you have running, tell them that's the wrong thing to be counting. The better question is how many decisions you're actually making.

Happy coding!

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 6 Invoke a Lambda function and parallel execution

Vadym Kazulkin — Mon, 21 Sep 2026 14:00:00 +0000

Introduction

In part 2 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. After that, in parts 3, 4, and 5 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement wait for callback, wait for condition, and map operations.

Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why, in this part, we'll move the logic associated with each step into a separate Lambda function. An additional advantage of this approach is that it allows us to apply individual settings to those functions, like setting the right memory and granting the exact permissions they need. We'll explore how to invoke another Lambda function within the durable step. Also, we'll show how to invoke multiple Lambda functions in parallel.

Sample application with invoking another Lambda function and performing parallel execution

Let's explore invoking another Lambda function and parallel execution. We get access to them through DurableContext.

invoke - the invoke operation calls another Lambda function and waits for its result. It supports both durable functions and standard on-demand Lambda functions. When you call ctx.invoke() or ctx.invokeAsync(), the SDK checkpoints the start of the operation, and the durable functions backend invokes the target Lambda function. On replay, the SDK returns the checkpointed result without invoking the target again.
parallel - parallel runs multiple independent branches concurrently, each in its own child context. Branches are registered via branch() and execute immediately (respecting maxConcurrency). The operation completes when all branches finish or completion criteria are met.

I first copied the application that we created in part 2 into the aws-s3-files-lambda-durable-functions-as-orchestrator-java-25 repository. We explain step by step how to adjust our application. Please go through the content of part 2 to understand the sample application and the basic concepts of Lambda durable functions. There will be some changes in the Infrastructure as Code (IaC) part as well. I'll leave out the wait for callbacks that we implemented in part 2. You can copy this logic into the Lambda durable function orchestrator, if you will.

We'll move the business logic for 3 individual steps out of our Lambda durable function AuthorContentExtractor to 3 different Lambda functions:

The implementation of all 3 Lambda functions is trivial. We'll focus on the change in the Lambda durable function AuthorContentExtractor and in the IaC located in the AWS SAM template.

We already showed in part 2 how to execute the first 2 Lambda functions asynchronously. This time, we'll show how to execute them in parallel. Here is how the complete code of the AuthorContentExtractor Lambda durable function looks:

public class AuthorContentExtractor extends DurableHandler<Author, AuthorContent> {

private static final String UPCOMING_TALKS_EXTRACTOR_FUNCTION_ARN  = 
System.getenv("UpcomingTalksExtractorFunctionArn");
private static final String YOUTUBE_VIDEOS_EXTRACTOR_FUNCTION_ARN  =
System.getenv("YouTubeVideosExtractorFunctionArn");
private static final String WRITE_AUTHOR_CONTENT_TO_FILE_FUNCTION_ARN  =
System.getenv("WriteAuthorContentToFileFunctionArn");


@Override
public AuthorContent handleRequest(Author author, DurableContext ctx) {
  var config = ParallelConfig.builder()
      .maxConcurrency(5)
      .nestingType(NestingType.NESTED)
      .completionConfig(CompletionConfig.allCompleted())
      .build();

  var invokeConfig=InvokeConfig.builder().build();

  var parallel = ctx.parallel("parallel-search", config);

  var upcomingTalksFuture = parallel.branch("searchForUpcomingTalks-parallel-step",
      UpcomingTalks.class, branchCtx -> {
           return branchCtx.invoke("searchForUpcomingTalks-step",           
             UPCOMING_TALKS_EXTRACTOR_FUNCTION_ARN, author,
             UpcomingTalks.class, invokeConfig);
     });

  var youtubeVideosFuture = parallel.branch("searchForYouTubeVideos-parallel-step",
      YouTubeVideos.class, branchCtx -> {
      return branchCtx.invoke("searchForYouTubeVideos-step",                
             YOUTUBE_VIDEOS_EXTRACTOR_FUNCTION_ARN, author, 
             YouTubeVideos.class, invokeConfig);
    });

  var result = parallel.get();
  var upcomingTalks=upcomingTalksFuture.get();
  var youtubeVideos= youtubeVideosFuture.get();

  var authorContent = new AuthorContent(author, upcomingTalks, youtubeVideos);

  ctx.invoke("writeAuthorContentToFile-step", WRITE_AUTHOR_CONTENT_TO_FILE_FUNCTION_ARN, 
    authorContent, Void.class, invokeConfig);

  return authorContent;
  }
}

Let's go step by step through the code. First, we pass the ARNs of all 3 Lambda functions via environment variables. We set those for the Lambda durable function AuthorContentExtractorFunction in the AWS SAM template. Also, we give our Lambda durable function the required permissions to invoke the other 3 Lambda functions:

  AuthorContentExtractorFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: AuthorContentExtractorOrchestrator
      Handler: dev.vkazulkin.handler.AuthorContentExtractor::handleRequest
      Environment:
        Variables:
          UpcomingTalksExtractorFunctionArn: !GetAtt UpcomingTalksExtractorFunction.Arn
          YouTubeVideosExtractorFunctionArn: !GetAtt YouTubeVideosExtractorFunction.Arn
          WriteAuthorContentToFileFunctionArn: !GetAtt WriteAuthorContentToFileFunction.Arn
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Sid: InvokeLambda
              Effect: Allow
              Action: lambda:InvokeFunction
              Resource:
                  - !GetAtt YouTubeVideosExtractorFunction.Arn
                  - !Sub '${YouTubeVideosExtractorFunction.Arn}:*'
                  - !GetAtt UpcomingTalksExtractorFunction.Arn
                  - !Sub '${UpcomingTalksExtractorFunction.Arn}:*'
                  - !GetAtt WriteAuthorContentToFileFunction.Arn
                  - !Sub '${WriteAuthorContentToFileFunction.Arn}:*'
....

As we invoke UpcomingTalksExtractor and YouTubeVideosExtractor Lambda functions in parallel, we create ParallelConfig. Here we set the maximum concurrency value and nesting type. There are 2 nesting types:

Nested (the default one). Create CONTEXT operations for each branch/iteration with full checkpointing. Operations within each branch/iteration are wrapped in their own context. Observability: High - each branch/iteration appears as a separate operation in execution history. Cost: Higher - consumes more operations due to CONTEXT creation overhead. Scale: Lower maximum iterations due to operation limits
Flat. Skip CONTEXT operations for branches/iterations using virtual contexts. Operations execute directly without individual context wrapping. Observability: Lower - branches/iterations don't appear as separate operations. Cost: ~30% lower - reduces operation consumption by skipping CONTEXT overhead. Scale: Higher maximum iterations possible within operation limits

We also set the CompletionConfig. Here, there are many possibilities from which we can choose:

allCompleted (we use it). All items run regardless of failures. Failures are captured per-item.
allSuccessful. All items must succeed. Zero failures tolerated.
firstSuccessful. Complete as soon as the first item succeeds.
minSuccessful. Complete when the specified number of items have succeeded.
toleratedFailureCount. Complete when more than the specified number of failures have occurred.
toleratedFailurePercentage. Complete when the failure percentage exceeds the specified threshold (0.0 to 1.0).

Next, we create an InvocationConfig for each individual Lambda function invocation that we can optionally set. Here we go with the default configuration. But we additionally set the tenant ID, which is used to isolate execution state for different tenants. It's required when invoking multi-tenant Lambda functions. So far, we covered the following parts:

 var config = ParallelConfig.builder()
      .maxConcurrency(5)
      .nestingType(NestingType.NESTED)
      .completionConfig(CompletionConfig.allCompleted())
      .build();

  var invokeConfig=InvokeConfig.builder().build();

Next, let's implement the Lambda function invocation in parallel:

  var parallel = ctx.parallel("parallel-search", config);

  var upcomingTalksFuture = parallel.branch("searchForUpcomingTalks-parallel-step",
      UpcomingTalks.class, branchCtx -> {
           return branchCtx.invoke("searchForUpcomingTalks-step",           
             UPCOMING_TALKS_EXTRACTOR_FUNCTION_ARN, author,
             UpcomingTalks.class, invokeConfig);
     });

  var youtubeVideosFuture = parallel.branch("searchForYouTubeVideos-parallel-step",
      YouTubeVideos.class, branchCtx -> {
      return branchCtx.invoke("searchForYouTubeVideos-step",                
             YOUTUBE_VIDEOS_EXTRACTOR_FUNCTION_ARN, author, 
             YouTubeVideos.class, invokeConfig);
    });

  var result = parallel.get();
  var upcomingTalks=upcomingTalksFuture.get();
  var youtubeVideos= youtubeVideosFuture.get();

  var authorContent = new AuthorContent(author, upcomingTalks, youtubeVideos);

First, we invoke the parallel method on the DurableContext and pass the ParallelConfig. We get back the ParallelDurableFuture, with which we can start a branch. To do this, we invoke the branch method for each individual Lambda invocation and specify the result type. In each branch, we call the invoke method on the BranchContext. Here we pass the following parameters:

name - the unique operation name within this context.
functionName - the ARN or name of the Lambda function to invoke.
payload - the input payload to send to the target function.
resultType - the result class for deserialization.

We get back the DurableFuture of the resultType (in our case UpcomingTalks or YouTubeVideos).

When we invoke the blocking get method on the ParallelDurableFuture to get the results with respect to the ParallelConfig. Then, we invoke the get method on DurableFuture to get the individual results of both Lambda invocations. Finally, we create the _ AuthorContent _ with these results.

The last step is to invoke the Lambda function, which is responsible for writing the author content into the file:

 ctx.invoke("writeAuthorContentToFile-step", WRITE_AUTHOR_CONTENT_TO_FILE_FUNCTION_ARN, 
    authorContent, Void.class, invokeConfig);

As we explained in part 2, we use S3 Files here. The whole part on how to configure S3 Files and give the Lambda function access to it has now moved from the AuthorContentExtractorOrchestrator durable Lambda function to the WriteAuthorContentToFileFunction Lambda function. See the SAM template for more detail.

Now we can build and package our application with mvn clean package and deploy it with sam deploy. The deployment process can take up to 10 minutes because of the creation and mounting of S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractorOrchestrator function, and go to the "Test" tab.

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

Let's also look at event history:

We see that this matches what we implemented. Our Lambda durable function invokes in parallel 2 Lambda functions to search for YouTube videos and upcoming talks (each in an individual branch). Then, it awaits the results of both invocations because we defined in the parallel configuration that we wait until all invocations are completed. Then the last Lambda function to write the author content into the file gets invoked.

One general question at the end: is the Lambda function invocation from another Lambda function considered an anti-pattern? Or should we put an API Gateway in between instead? My answer is that with built-in fault-tolerance for AWS Lambda durable functions, it's now safe to invoke a Lambda function directly.

Conclusion

In this part of the series, we moved the logic associated with each step into a separate Lambda function. We explored how to invoke another Lambda function within the durable step. Also, we showed how to invoke multiple Lambda functions in parallel.

In the next part of the series, we'll transform our application into an agentic one. Currently, the search for YouTube and upcoming talks returns static content. We'll replace it with the Amazon Bedrock AgentCore Gateway Web Search Tool. As it's exposed via MCP, we need an MCP client to connect to it. For that, we'll use the Spring AI framework. First, we still use AWS Lambda to host the AI agents. In later parts, we'll show how to host those AI agents on the Amazon Bedrock AgentCore Runtime instead.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 5 Map and child context

Vadym Kazulkin — Mon, 14 Sep 2026 14:30:00 +0000

Introduction

In part 2 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. Later, in parts 3 and 4 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement wait for callback and wait for condition.

In this part, we'll explore map and child context operations.

Sample application with map and child context

Let's explore the child context operation first. Child contexts run an isolated stream of work with their own operation counter and checkpoint log. They support the full range of durable operations — step, wait, invoke, createCallback, and nested child contexts. In our sample application, we'll add a map operation, which uses a child context underneath. We get access to it through DurableContext. A Map operation applies a function to each item in a collection concurrently, with each item running in its own child context.

Now let's add an artificial use case: for each YouTube video found, we add it to the author's YouTube playlist. I first copied the application that we created in part 2 into the aws-s3-files-lambda-durable-functions-with-map-and-child-context-java-25 repository.

The main business logic we have to add is in the saveYouTubeVideosToPlayList method of the AbstractAuthorContentExtractor:

public default void saveYouTubeVideosToPlayList (DurableContext ctx, Author author, YouTubeVideos youtubeVideos)  {
   var youtubePlayList = findOrCreateYouTubePlayListForTheAuthor(author);

   var mapConfig = MapConfig.builder()
       .nestingType(NestingType.NESTED)
       .maxConcurrency(5)                                   
       .completionConfig(CompletionConfig.allCompleted())    
       .build();

   var mapResult = ctx.map("saveYouTubeVideoToPlayList-step", youtubeVideos.youtubeVideos(), 
      Boolean.class, (youtubeVideo, index, childCtx) -> {       
        return childCtx.step("saveYouTubeVideoToPlayList-step-"+index, Boolean.class, 
       stepCtx -> saveYouTubeVideoToPlayList(youtubeVideo, youtubePlayList)
      , this.getStepConfig());
    }, mapConfig);

}


private static int findOrCreateYouTubePlayListForTheAuthor(Author author) {
     // Use YouTube API to log in to the YouTube Account and search for the author's playlist
     // and return it if it already exists or create a new one
     // move this logic into a separate Lambda function and invoke it as a durable step
     return 1;
}

private static Boolean saveYouTubeVideoToPlayList(YouTubeVideo youtubeVideo, int playListId) {
    //Use YouTube API to save the video URL to the playlist. 
    return true;
}

We invoke this saveYouTubeVideosToPlayList method in the AuthorContentExtractor or AsyncAuthorContentExtractor classes.

First of all, I won't use the YouTube API to create or retrieve the YouTube playlist for the given author and store each found video there. I use some static output, as the goal of this article is to demonstrate the functionality of the map operation.

Let's go step by step through this code. Map applies a function to each item in a collection concurrently, with each item running in its own child context. Results are collected into a MapResult that maintains input order. Each item's function receives its own DurableContext, so you can use any durable operation (step(), wait(), invoke(), etc.) inside the map function. The function passed to map() is a MapFunction:

@FunctionalInterface
public interface MapFunction<I, O> {
    O apply(I item, int index, DurableContext context);
}

The index parameter is the zero-based position of the item in the input collection, useful for naming operations or correlating results. For each found YouTube video, we invoke the saveYouTubeVideoToPlayList method, which returns the result of type Boolean. In our case, it is always true, indicating that the video was stored in the playlist.

The input collection must have deterministic iteration order. List, LinkedList, and TreeSet are accepted. HashSet and unordered map views are not, leading to the IllegalArgumentException. That's why I changed the implementation of the YouTubeVideos class to use List instead of Set there.

Next, let's explore MapConfig in more detail. There are several main settings there:

maxConcurrency controls how many items execute concurrently. When set, items beyond the limit are queued and started as earlier items complete. The default is null (unlimited).

nestingType, with Nested as the default.

Nested. Create CONTEXT operations for each branch/iteration with full checkpointing. Operations within each branch/iteration are wrapped in their own context. Observability: High - each branch/iteration appears as a separate operation in execution history. Cost: Higher - consumes more operations due to CONTEXT creation overhead. Scale: Lower maximum iterations due to operation limits
Flat. Skip CONTEXT operations for branches/iterations using virtual contexts. Operations execute directly without individual context wrapping. Observability: Lower - branches/iterations don't appear as separate operations. Cost: ~30% lower - reduces operation consumption by skipping CONTEXT overhead. Scale: Higher maximum iterations possible within operation limits

CompletionConfig controls when the map operation stops starting new items. Here, there are many possibilities from which we can choose:

allCompleted (we use it). All items run regardless of failures. Failures are captured per-item.
allSuccessful. All items must succeed. Zero failures tolerated.
firstSuccessful. Complete as soon as the first item succeeds.
minSuccessful. Complete when the specified number of items have succeeded.
toleratedFailureCount. Complete when more than the specified number of failures have occurred.
toleratedFailurePercentage. Complete when the failure percentage exceeds the specified threshold (0.0 to 1.0).

When early termination is triggered, items that were never started have SKIPPED status with null for both result and error in the MapResult.

Each MapResultItem contains the following methods:

status: SUCCEEDED, FAILED, or SKIPPED.
result: the result value, or null if failed/skipped.
error: the error details as MapError, or null if succeeded/skipped.

Failed items store error details as MapError, a serializable record that survives checkpoint-and-replay cycles with the following methods that can be invoked:

errorType: Fully qualified exception class name (e.g., java.lang.RuntimeException).
errorMessage: The exception message.
stackTrace: Stack trace frames as a list of strings, or null.

One item's failure does not prevent other items from completing. MapResult captures the failed items at their corresponding index.

Map operations are fully durable. On replay after interruption:

Completed items return cached results without re-execution.
Incomplete items resume from their last checkpoint.
Items that never started execute fresh.

Small results (< 256KB) are checkpointed directly, whereas large results are reconstructed from individual child context checkpoints on replay.

That's it. There are no changes in the Infrastructure as Code part that we need to make. Now we can build and package our application with mvn clean package and deploy it with sam deploy. The deployment process can take up to 10 minutes because of the creation and mounting of S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractor or AsyncAuthorContentExtractor function, and go to the "Test" tab.

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

Here we see that the operation with the name saveYouTubeVideoToPlayList-step and the subtype Map. We also see 2 iterations (for each of the 2 YouTube videos found) with the subtype MapIteration. Each iteration is executed as an individual durable step.

Let's also look at event history:

Here we see all events (durable operations) in the right order required for the execution of our Lambda durable function.

There is also the mapAsync operation. It starts the map operation without blocking, returning a DurableFuture.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement map and child context operations.

Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why, in the next part, we'll move the logic for each step into a separate Lambda function. With that, we'll explore how to invoke another Lambda function within the durable step. We'll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Payment is authorization

Allen Helton — Thu, 10 Sep 2026 18:52:30 +0000

I can't tell you how many times I've needed to do something simple like remove the background from an image or convert a file from one type to another but wasn't able to do it. While these feel like simple, common things your OS should just handle, they require a service to do them. Luckily, I'm usually able to Google the problem, find a website dedicated to solving that one problem and do it.

It's super nice because I land on a website like remove.bg, drag my image onto the page, and the background is removed. I don't have to log in, it's not tracking me (that I'm aware of), and it just works. The API version of this is something I'd call a utility API. It simply does a thing and gives you a result.

Compare a utility API with what we're growing more and more accustomed to. I click on a marketing site, sign up, pick a plan, confirm my email address, then start making requests. You know what I call that? Friction.

And in a world where AI agents are becoming increasingly important consumers of APIs, friction is holding us back.

There's a lot of attention right now on agent identity, and for good reason. Agents that read private data, modify infrastructure, operate business systems, or act on behalf of people absolutely need durable identity, delegated authority, auditability, and policy.

But I think we're going a little too far with it.

For a lot of utility APIs, identity doesn't need to be part of the authorization decision. Instead, it just becomes who we charge. Let's dig into that a little bit.

Bring your own access

To perform a basic utility, it's likely that an agent will discover your API at runtime, perform a single operation, and never call it again. So requiring it to create an account and establish a permanent relationship with your service before doing 2 cents of work feels wrong. What if instead, your API provided a pay-as-you-go on-demand model (something my readers should be familiar with). x402 is an open protocol for payments over HTTP, led by the x402 Foundation under the Linux Foundation.

The flow looks like this:

The caller doesn't necessarily need to create an account first. It requests the resource, gets back machine-readable payment details, pays, and retries with proof of payment.

Which, when you think about it, means that payment alone can be the authorization decision. While yes, you could still add a layer of AuthN/AuthZ on top of the API, the x402 protocol doesn't require it. This removes that friction we were talking about earlier and lets agents pay and continue (as long as they have access to a wallet).

This is a very different access model than we're used to and makes many developers feel a little uneasy because it goes against the paradigm we've been handcuffing ourselves to for years. But if you ask yourself five whys, what we're really after is payment. And this lets us cut out a lot of extra work to get to it.

What does this look like?

My favorite way to learn is by doing, so let's get into what implementing x402 in a serverless API would look like. Because I love the idea, but this sounds tricky to implement once you actually get down to it. If payment is part of authorization, do you put that in a Lambda authorizer?

You shouldn't. Lambda authorizers are built around the idea that an authorization decision can be reused. API Gateway evaluates an identity source, decides whether that caller may invoke the API (and specifically which endpoints inside of it), and can cache the result for later requests.

Lambda authorizers work well for things like bearer tokens because the claim being evaluated usually authorizes users for an entire session. This makes the generated policy cacheable and an easy deciding factor for where it lives.

Of course, you can disable the cache with a 0 second TTL, but the authorizer sits at the wrong point of the request lifecycle. It runs before a request hits your API. For payment authorization, we need to go a little further down.

A payment is generally going to be for a single call. If you want to call it 10 times, then you need to pay for it 10 times. Nothing here is cacheable. For this architecture, that leaves us with one option, really. Bringing payment logic into the app code 😬. Well, technically you can also use AWS WAF AI traffic monetization, but more on those limitations in a bit.

The API controller

For years, the serverless community had a debate on whether you should build your APIs as a "lambdalith" where a single Lambda function acts as your API controller and serves your entire API, vs a single Lambda function per endpoint. I was team "function per endpoint" for years, but have come around to the idea that I was wrong.

I won't jump back into the philosophical debate here, just know that we're designing a single Lambda function to serve the entire API using the Lambda Powertools HTTP event handler to define our routes and do input/output transformations for us.

It also sets us up for treating x402 as middleware. Lambda Powertools is great at a lot of things, and middleware is where it's golden. Since we don't really want payment processing logic littered through the app code, surfacing it as middleware is the right abstraction. Doing it this way gives us some handy conveniences as well.

Route pricing

If you've ever built an API with Express, Hono, or Lambda Powertools, you'll be familiar with the syntax:

app.post('/generate', async () => {
  // Generate something
  return { message: 'success!'}
});

This defines a POST /generate endpoint that does some work and returns a success message. But let's pretend like we wanted to charge money to call it. All we need to do is add some middleware to the route:

app.post(
  '/generate',
  [ x402.paid({ price: '$0.05', description: 'Create an image of something' })],
  async () => {
    // Generate something
    return { message: 'success!'}
  }
);

Isn't that slick? The handler has no payment logic, nor does it parse payment headers, talk to a facilitator, or know anything about wallets.

The middleware verifies that the request has acquired the right to execute, runs the handler, and handles settlement around the request lifecycle.

The coolest part of the whole thing is, declared this way, the price is essentially route metadata. It just kinda fits in with method, path, validation, and rate limits as properties of a route. Looking at what that implies for documentation, that could easily turn into something documented in your OpenAPI spec or tool definitions so agents don't need to receive that first 402 Payment Required status code.

If an agent discovers a capability and can see its price as part of that discovery, then the access model becomes much easier to reason about. Instead of discovering an API and immediately starting a human onboarding flow, the agent can evaluate the cost as part of deciding whether to use it at all.

NOTE - the middleware I used in the example above is an open-source project I built that's available via npm. It's specifically designed to work with Lambda Powertools.

Error handling

You know what would be bad? If a caller paid for an operation, it failed, and returned a receipt saying you paid for the failure. Luckily, the payment flow we're using is designed to prevent that from happening, but it's not bulletproof.

If we jump back to the sequence diagram from earlier, you can see an important order of operations that should prevent paying for failures.

verify payment
→ run handler
→ handler succeeded?
    → yes: settle
    → no: do not settle

This is how the AI monetization feature in AWS WAF approaches the problem, but it's not perfect. WAF runs at the edge, and pragmatically it is inspecting a status code. That's (generally) fine for selling a document, but not robust enough for selling an operation. An empty generation, a model refusal, or a search that found nothing could all still return a 200 OK, which means that payment settled and the caller got nothing.

Middleware runs in-process, so it can decide on the actual result:

x402.paid({
  price: '$0.05',
  billable: (result) => result.imageUrl != null
})

In the above example, if the response body doesn't contain an imageUrl property, they would still get a 200 OK, but they wouldn't be charged. This is a meaningful improvement over status code checking and gives you endless levers for fairly charging your callers.

But what happens if settlement succeeds and the response fails to return to the caller? The agent retries. The payment-identifier extension is your best friend here. The caller can reuse the same payment id, allowing the API to return the cached result without charging or executing the operation again. Of course, that means you have to persist the result somewhere durable and associate the id to the original request (sounds like idempotency to me). It's still your job to implement this correctly. 🤷‍♂️

This is the next frontier in AI

Our industry has done a phenomenal job the past several years rocketing us through uncharted territories. In what feels like no time at all, we've gone from pioneering chatbots to agents to memory to security to multi-agent collaboration and everything in between. But we're just starting to scratch the surface on agentic payments.

There are still many hard problems to solve, like transaction latency and reputation tracking (among other things). Not to mention the elephant in the room when it comes to humans handing over control: trust. Anecdotally, I've already heard stories about how people are implementing spiked prices via x402 specifically for bots (by reading User-Agent headers), like raising the price from $200 for a pair of sneakers to $1,200 in hopes of surpassing allowed price limits without a human in the loop.

Not to mention the discovery problem. Updating your APIs to accept x402 is step one. But with software, the adage "if you build it, they will come" is unfortunately not true. At all.

Coinbase has taken a stab at a registry with their Bazaar catalog, which gives us a great starting ground. But for long-term success, I think we'll need something a bit more built-out. More on this in a blog post to come. 😉

For now though, what I want you to take away is that API consumption has fundamentally changed. And how we look at it going forward needs to change as well. Maybe we rethink the multi-step authentication mechanism and opt for a more streamlined "auth by payment" protocol (when applicable).

It's already relatively easy to retrofit your APIs with x402 if you're using Lambda Powertools. Honestly, the core x402 package makes it easy no matter what you're using. But the more important part here is for you to start thinking about your APIs differently.

Do you have a utility API? Is your current auth flow absolutely necessary? Are you able to charge on an operation level for your API capabilities?

Try it out. Tell me what you think. Let's pioneer something new.

Happy coding!

Building AI Agents with Embabel, Spring AI and Amazon Bedrock AgentCore - Part 4 Upgrade sample application to Embabel 1.5

Vadym Kazulkin — Mon, 07 Sep 2026 14:00:00 +0000

Upgrade sample application to Embabel 1.5 to support Spring Boot 4.x and Spring AI 2.x

In parts 2 and 3, we developed our sample application and interacted with it via the shell and HTTP. We used Embabel version 1.0 there, which targets Spring Boot 3.x / Spring AI 1.x.

Recently, the Embabel team released version 1.5, which supports Spring Boot 4.x / Spring AI 2.x. In this article, we'll explain how to migrate our sample application to the Embabel 1.5.x version. You can find the final version of the application in my GitHub repository.

The migration for my use case is really simple. First, in pom.xml we need to declare this Embabel version:

<properties>
   <java.version>25</java.version>
   <embabel-agent.version>1.5.1</embabel-agent.version>
</properties>

Please note that we use version 1.5.1 here. With version 1.5.0, I ran into a specific problem while using the Amazon Bedrock Converse API. This was due to the fact that BedrockOptionsConverter created generic ToolCallingChatOptions. Spring AI 2.0’s BedrockProxyChatModel expects BedrockChatOptions and casts the request options accordingly, causing:

ClassCastException: DefaultToolCallingChatOptions cannot be cast to BedrockChatOptions.

With version 1.5.1, the Embabel team fixed the problem. Next, as Spring Boot 4.x / Spring AI 2.x uses Jackson version 3 by default, we need to declare a Jackson 3 mapper Spring bean. We do it in the Spring Boot main EmbabelConferenceApplication application class of our sample application:

@Bean
public JsonMapper objectMapper() {
    return new tools.jackson.databind.json.JsonMapper();
}

That's it. Now you can compile, start, and interact with our sample application the same way we described in parts 2 and 3.

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 4 Wait for condition

Vadym Kazulkin — Mon, 31 Aug 2026 14:00:00 +0000

Introduction

In part 2 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously.
Later, in part 3 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement wait and callback.

In this part, we'll explore the wait for condition operation.

Sample application with wait for condition

Let's explore the "wait for condition" operation. We get access to it through DurableContext. It repeatedly calls a check function until it signals done. Between polls, the Lambda suspends without consuming compute. State is checkpointed after each check, so progress survives interruptions.

I first copied the application that we created in part 3 into the aws-s3-files-lambda-durable-functions-with-wait-for-condition-java-25 repository. The only change we had to make was to reimplement the waitForUpcomingTalksApproval method of the AbstractAuthorContentExtractor class. Here we'll now use the "wait for condition" instead of the "wait for callback" operation.

This is how the code now looks:

public default UpcomingTalkApprovalStatus waitForUpcomingTalksApproval(DurableContext ctx, Author author, UpcomingTalks upcomingTalks) {

  var config = WaitForConditionConfig.<UpcomingTalkApprovalStatus>builder()                
     .waitStrategy(WaitStrategies.defaultStrategy())
     .initialState(new UpcomingTalkApprovalStatus(upcomingTalks, 
         UpcomingTalkApprovalStatus.Status.PENDING))
     .build();

  return ctx.waitForCondition(
     "wait-for-approval",
     UpcomingTalkApprovalStatus.class, 
         (currentStatus, stepCtx) -> {
         var latest = this.getUpcomingTalkApprovalStatus(author,upcomingTalks);
         return  
         UpcomingTalkApprovalStatus.Status.APPROVED.equals(latest.status())                     
         ? WaitForConditionResult.stopPolling(latest)
         : WaitForConditionResult.continuePolling(latest);
    },
    config);
}

private UpcomingTalkApprovalStatus getUpcomingTalkApprovalStatus(Author author, UpcomingTalks upcomingTalks) {
     // Generate random integers in range 0 to 3
     int r = RANDOM.nextInt(4);
     return r < 3 ? 
       new UpcomingTalkApprovalStatus(upcomingTalks, 
          UpcomingTalkApprovalStatus.Status.NOT_APPROVED) :
       new UpcomingTalkApprovalStatus(upcomingTalks, 
          UpcomingTalkApprovalStatus.Status.APPROVED);
}

Let's go step by step through this code. The ctx.waitForCondition invocation receives the current state and a StepContext, and returns a WaitForConditionResult. WaitForConditionResult contains the result type, in our case UpcomingTalkApprovalStatus :

WaitForConditionResult.stopPolling(value) - condition met (in our case, status "APPROVED"), return value as the final result.
WaitForConditionResult.continuePolling(value) - keep polling (bacause the status is "NOT_APPROVED"), pass value to the next check.

For the sake of simplicity, we haven't implemented the proper endpoint to poll the status. We provided a simple implementation in the getUpcomingTalkApprovalStatus method, which generates a random integer between 0 and 3. If 3 is generated, the "APPROVED" status will be returned, indicating that the condition is met and the polling should stop. If 0, 1, or 2 are generated, the operation will keep polling.

Next, let's look at how to configure WaitForConditionConfig. First, we pass the initialState with the status "PENDING," and this status is passed to the first check invocation. After that, we also configure the waitStrategy. The wait strategy controls the delay between polls. By default, waitForCondition uses an exponential backoff wait strategy (60 max attempts, 5s initial delay, 300s max delay, 1.5x backoff rate, FULL jitter). We can invoke the WaitStrategies.exponentialBackoff method to configure exponential backoff with different settings than the default ones. Alternatively, we can use the WaitStrategies.fixedDelay method. With this wait strategy, we need to configure:

fixedDelay - a constant delay between polls and
maxAttempts - a maximum number of attempts before throwing WaitForConditionFailedException. This exception is thrown when the maximum attempts are exceeded (thrown by the wait strategy).

We can also implement Custom Wait Strategies, which we won't cover in this article.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractor or AsyncAuthorContentExtractor function, and go to the "Test" tab.

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

Here we see that the operation with the name wait-for-approval and the subtype WaitForCondition succeeded after 3 attempts. The number of retries is 2. This means that it took 3 attempts to generate the integer 3. We defined the generation of this value as the condition to stop polling and return the result.

Let's also look at event history:

Here we see all events (durable operations) in the right order required for the execution of our Lambda durable function.

There is also the waitForConditionAsync operation for non-blocking polling that you can explore on your own. It starts polling but returns a DurableFuture immediately, allowing other operations to run concurrently.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement the wait for condition operation.

In the next part, we'll explore map and child context operations. Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why, in the subsequent parts, we'll move the logic for each step into a separate Lambda function. With that, we'll explore how to invoke another Lambda function within the durable step. We'll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

DynamoDB Streams is not an outbox

Allen Helton — Tue, 25 Aug 2026 16:39:57 +0000

Hopefully you didn't notice, but a couple of weeks ago I sent an issue of my newsletter that never actually got emailed. My dashboard looked fine and it was marked as published, but the email never hit my inbox (or any of yours 😬).

I didn't notice until two days later when I went to look at the weekly statistics and found nothing. Just a Sent status and a bunch of 0's on the screen.

A quick browse through the code didn't turn up anything obvious. Emails are sent via an async handler triggered by an EventBridge event while the status of the newsletter is marked separately in DynamoDB, a classic "db save and domain event publish" pattern. We've all done it, it looks something like this:

await ddb.send(new UpdateItemCommand({ ... }));
await eventBridge.send(new PutEventsCommand({ ... }));

Despite its simple appearance, this is a nasty anti-pattern. Formally known as the dual write problem, if those two actions aren't contained in a single transaction, you're asking for trouble. Why? I'm glad you asked.

Imagine the write succeeds, but the publish doesn't. Data is updated but the follow-on actions that react to the domain event never run (like your email never being sent 🙃). Things fall into a bad state and you (maybe) end up stumbling upon a problem days later.

Ok, so flip them. Put the event first and the data save second. But if the publish succeeds and the db save fails, then you risk acting on stale data, which depending on your domain, is much worse.

Before you ask, no, serverless doesn't make it better. In many cases, serverless makes us too laissez-faire to solve it the "write" way (get it?).

What you're probably doing

Let's say you already knew not to do the data save and the event publish at the same time. I bet I know what you're doing instead. It's a pattern AWS has recommended for years and is all over Serverless Land.

You do your save to DynamoDB, let the change data flow into DynamoDB Streams, process it in Lambda, then fire your domain events from the stream processor.

When it comes down to it, this is legitimately better, and it's the same thought process behind going storage-first. You persist it first, then process it asynchronously. That makes the database write the event trigger, so now your domain events are tightly coupled to the corresponding data change. Plus, DynamoDB Streams retries automatically (and even bisects batches when configured) if there's a processing failure.

Sounds like our dual write problem is solved, right? Wrong. Kind of. It's complicated. 😅

DynamoDB Streams records are change data capture, not domain events. The Lambda function you write to handle them receives a NewImage and OldImage record, and it's up to you to infer what happened upstream. Smells like leaky business logic to me. Before you know it, you have this thousand-line file that looks at 40 different data shapes to determine which event to publish and which transformation to perform. As someone who's found themselves in that hole before, believe me when I say that is a nightmare situation.

Let's take it a step further. A business operation could (and probably does) affect multiple DynamoDB records. When I publish my newsletter, I create a DynamoDB record with a short TTL for each email address that was successfully sent to. If I am processing entity changes in a stream handler, I'd be up to my ears in duplicate newsletter.sent events because I'm working on individual records and not an aggregate operation. That ultimately leads to workarounds with atomic counters and lastAction flags that make it overly complex with little value add.

And most importantly, you don't know what did or didn't publish. Streams retain data for 24 hours, and if your publisher was broken for a day because of an outage or a rogue IAM change, you're out of luck. Even if you opted for something like Kinesis, which has 365 days of data retention, that's not the point. These mechanisms are change data, not the intended effect on the system.

DynamoDB Streams are a wonderful piece of engineering that gives you retries, but not the durability of intent.

What you should do instead

Time for a bit of a mental shift. We need to prioritize domain events. If an event is important enough to publish, it's important enough to persist in a database.

This is where the outbox pattern comes into play. Instead of deriving your domain events from the data change in a downstream handler, you save the event in an outbox table in a transaction along with the data mutation. The transaction guarantees both the domain event and the data change will save together (or neither).

With DynamoDB, that could look like this:

const eventId = ulid();

await ddb.send(new TransactWriteItemsCommand({
  TransactItems: [
    {
      Update: {
        TableName: 'newsletter',
        Key: marshall({ pk: issueId, sk: 'issue' }),
        UpdateExpression: 'SET #status = :sent',
        ...
      }
    },
    {
      Put: {
        TableName: 'outbox',
        Item: marshall({
          pk: issueId,
          sk: `EVENT#${eventId}`,
          status: 'PENDING',
          pendingBucket: 'PENDING',              // GSI1 PK, removed on publish
          pendingAt: new Date().toISOString(),   // GSI1 SK
          source: 'newsletter.service',
          detailType: 'newsletter.sent',
          detail: { issueId, date, idempotencyKey: eventId }
        })
      }
    }
  ]
}));

Funnily enough, the architecture doesn't change with this pattern. You're still doing Lambda (or whatever compute) to DynamoDB to a stream to another Lambda function. But this time the downstream Lambda function is processing the event you saved, not the mutated data. It doesn't have to infer anything.

AWS does a fantastic job in their transactional outbox design pattern docs showing you how to setup and implement this pattern yourself. So I won't go into more of the code here. But I will address some questions I had the first time I built this.

What do I do with an event after it's published?

After you publish an event in your stream handler, it's up to you to either delete it from the outbox or mark it as published. Personally, I prefer to mark it as published and set a 30 day TTL. This means in the stream handler, we're hitting the dual write problem again because we are both publishing an event and updating the outbox with a published status.

This one is ok though 😉 and it brings up an important point we haven't covered yet. Idempotency is crucial for success with this pattern. The outbox pattern trades duplicate delivery in exchange for never losing durable content. If your publisher fails to send the event or update the database with a published status, that's fine! Do it again. The important part is that the intent wasn't lost.

Can I replace the stream handler function with EventBridge Pipes?

Fewer pieces, easier system to maintain, right? In this instance, the Lambda function is the better solution over Pipes. If you were only publishing an event to EventBridge, you could easily filter the stream to the relevant outbox records and forward it straight to a bus. But because you also need to write back to DynamoDB to update the status of the event, the function wins. A pipe can only perform a single operation (outside of enrichments and transformations), but a function can perform many.

Does this guarantee ordering?

Nope. The outbox pattern is designed for correctness only. Eventually consistent state across a distributed system. This does not tackle other hard distributed system problems like ordering or exactly-once delivery. Quite the opposite in fact. While Streams do process change data in the order they happen, it's per shard (not globally), and you can't guarantee ordering when it comes to retries. And as soon as you drop an event on EventBridge, you lose any sort of ordering.

How does this guarantee events aren't skipped?

If you implemented this exactly as described, it wouldn't. It gives you the normal retry path you're already used to, but still has that 24-hour clock because of the stream data retention. The important piece that's missing is a scheduled reconciler. It's usually another Lambda function that queries a sparse GSI for anything still pending from more than five minutes ago, which is long enough that the stream handler has had its shot, and republishes it. So your full flow would look something like this:

Remember this

There is no clever workaround to the dual write problem. You can't order the calls in a way that mitigates the risk, you can't retry around it, and you can't/shouldn't move it into a stream handler and hope.

But you can stop having two writes. Well, two writes, but one transaction. 😅

Remember, what we're solving with the outbox pattern is a correctness problem. If someone asks "how can you guarantee the system is eventually consistent, even under backpressure?" this is what you reach for. It took me years to arrive here naturally and I've been accidentally building holes in my systems longer than I'd care to admit.

This pattern does make your architecture more complicated, but the trade-offs are often worth it, especially at scale. In exchange for 2x the WCUs, you get a solid way to store and act upon intent in your system. Anyway, take a look at how you're publishing events and see if there are any unexplained missing events in your system. If there are, you know what to do. 😉 That's what I did and you should be reliably getting your newsletters from now on.

Happy coding!

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 3 Waits and callbacks

Vadym Kazulkin — Mon, 24 Aug 2026 14:30:00 +0000

Introduction

In part 2 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. In this part, we'll extend our application by adding and implementing wait and callback operations to it.

Sample application with wait for callback

Let's explore waits and callbacks. We get access to them through DurableContext. Waits are planned pauses where your function stops running and stops charging until it's time to continue. Use them for time periods, external callbacks, or polling for a condition. Callbacks suspend execution until an external system sends a result. Use this for human approvals, webhooks, or any event-driven workflow. In our sample application, we'll combine both and use waitForCallback. It simplifies callback handling by combining callback creation and submission in one operation.

I first copied the application that we created in part 2 into the aws-s3-files-lambda-durable-functions-with-wait-for-callback-java-25 repository. We'll work on it to extend our application. Please go through the content of part 2 to understand the sample application and the basic concepts of Lambda durable functions. We'll completely reuse the Infrastructure as Code part as well.

Let's add another, a bit artificial use case to our sample application. After searching for the author's YouTube videos and upcoming talks, the author should confirm those talks. For this, let's extend our AsyncAuthorContentExtractor Lambda function and add the following invocation:

var upcomingTalkApprovalStatus = this.waitForUpcomingTalksApproval(ctx, author, upcomingTalks);

Next, let's look at the implementation of the waitForUpcomingTalksApproval method in the AbstractAuthorContentExtractor class:

public default UpcomingTalkApprovalStatus waitForUpcomingTalksApproval(DurableContext ctx, Author author, UpcomingTalks upcomingTalks) {
    var waitCallbackConfig= WaitForCallbackConfig.builder()
           .callbackConfig(CallbackConfig.builder()
           .timeout(Duration.ofHours(1))
           .build())
       .build();

   return ctx.waitForCallback(
            "wait-for-approval",
             UpcomingTalkApprovalStatus.class,
             (callbackId, stepCtx) 
              -> this.sendApprovalRequest(callbackId, author, upcomingTalks),
             waitCallbackConfig);
}

private void sendApprovalRequest(String callbackId, Author author, UpcomingTalks upcomingTalks) {
       LOGGER.info("get approval for the talk of  "+ author+ " for the talks "+upcomingTalks + 
       " with the callback id "+callbackId);
}

Let's explain step by step what is happening here. We use DurableContext to create the waitForCallback operation. We give it a name and the return type. This return type corresponds to the type of the object that will be passed in the callback. We also provide the submitter function sendApprovalRequest, receiving the callback ID and a StepContext. In this function, we can perform the desired operation, like sending the talk approval request to the author. We won't implement this functionality but only log the callback ID, which we'll require later to send the callback. We can optionally pass the WaitForCallbackConfig. In our case, we only set the CallbackConfig with a timeout of 1 hour. This means that if no callback arrives within this period of time, the Lambda durable function will fail. We can also set the StepConfig. Please review part 2 for more information.

When we invoke the waitForCallback method, the invocation will block until the callback arrives. After its arrival, the payload of the callback is deserialized into the object of type UpcomingTalkApprovalStatus, and this object will be returned.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AsyncAuthorContentExtractor function, and go to the "Test" tab.

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin",
  "email": "bla@bla.de"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

What we see here is that the first 2 steps (searching for YouTube videos and upcoming talks) have already been executed asynchronously. Now, our Lambda function is in the "wait for the approval" state. The invocation of the Lambda durable function is now completed. Why? Because waits suspend our function without incurring charges.

Now let's send the reply (callback) to our Lambda durable function. We first need to grab the callback ID from the log. There are several ways to send the callback (success or failure). We can use the Lambda CLI send-durable-execution-callback-success or execution-callback-failure commands. Alternatively, we can use the AWS SDK to do the same. Let's use the AWS Java SDK for it. First, we need to add the following dependency to pom.xml:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>lambda</artifactId>
</dependency>

We implemented the functionality to send the callback in the SendDurableExecutionCallback class:

public class SendDurableExecutionCallback {

  private final static String CALLBACK_ID=
"Ab9hZXiWYXJuOmF3czpsYW1iZGE6dXMtZWFzdC0xOjI2NTYzNDI1NzYxMDpmdW5jdGlvbjpBdXRob3JDb250ZW50RXh0cmFjdG9yOiRMQVRFU1QvZHVyYWJsZS1leGVjdXRpb24vc2VhcmNoRm9yVmFkeW0xMzUvMmM2NzRiOWEtNTQ0OC0zZDVjLWI3MzUtNTQ1NWM5ODA5MWUwYWl4JDA2N2JkNmRkLTViN2MtNGZmYi05NmY4LTRkM2Y1Yjk3MWE0Zf8";

  private final static LambdaClient LAMBDA_CLIENT = 
  LambdaClient.builder().region(Region.US_EAST_1).build();

  private final static ObjectMapper MAPPER= new ObjectMapper();

  public static void main(String[] args) {

    var approvedStatus= MAPPER.writeValueAsString(
         new UpcomingTalkApprovalStatus(UpcomingTalks.getDefaultUpcomingTalks(), "approved"));

    var response= LAMBDA_CLIENT.sendDurableExecutionCallbackSuccess(
                  builder -> builder.callbackId(CALLBACK_ID)
                  .result(SdkBytes.fromUtf8String(approvedStatus))
                  .build());
    }
}

First, we need to configure the callback ID (please replace it with your individual one). Then, we create the Lambda client. Next, we invoke its sendDurableExecutionCallbackSuccess method to send the successful callback. We pass the callback ID and payload as JSON. This payload should be deserializable into the object of type UpcomingTalkApprovalStatus (see above). Similarly, we can send the failure callback by using the sendDurableExecutionCallbackFailure operation. Please move this logic with all required dependencies to the service that sends the callbacks. Now let's send the callback (approximately 23 minutes later) and see what happens:

We observe that the callback was successfully received and processed. After it, the last writeAuthorContentToFile-step step was executed as well. But we also observe that execution of our Lambda durable function started from the beginning. What happened here exactly? After our function resumes from a pause or interruption, the system performs replay. During replay, our code runs from the beginning but skips over completed checkpoints, using stored results instead of re-executing completed operations. This replay mechanism ensures consistency while enabling long-running executions.

Here is the picture, which describes the concept that I took from this source:

There are also other individual operations that AWS Lambda Durable Execution SDK for Java supports:

wait - wait suspends the function and resumes after the specified duration. There is no charge during suspension.
waitAsync - starts the wait timer but returns a DurableFuture immediately, allowing other operations to run concurrently. The execution only suspends when you call .get() on the future (if the wait hasn't completed yet).
createCallback - callbacks suspend execution until an external system sends a result. Use this for human approvals, webhooks, or any event-driven workflow.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement wait and callback.

In the next part, we'll explore the "wait for condition" operation to poll a condition until it's met. Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why, in the subsequent parts, we'll move the logic for each step into a separate Lambda function. With that, we'll explore how to invoke another Lambda function within the durable step. We'll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 2 Durable synchronous and asynchronous steps

Vadym Kazulkin — Mon, 17 Aug 2026 13:00:00 +0000

Introduction

In part 1, we introduced what we'll cover in this series and our sample application. We also gave a short introduction to the services that we'll use throughout this series: AWS Lambda durable functions and Amazon S3 Files. In this part of the series, we'll explore how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously.

First implementation attempt of our sample application

Throughout this article series, we'll develop a simple application to show how to build multi-step applications and orchestrate workflows using AWS Lambda durable functions. This simple application provides the functionality to extract the content for the provided user. To this content belongs searching for YouTube videos of the authors and their upcoming talks. We'll store the result on Amazon S3 Files in JSON format. As this operation may potentially take time, we'll additionally provide access to the author content through the Amazon API Gateway:

We'll start with the search for YouTube videos and upcoming talks of the author synchronously. For the sake of simplicity, we won't implement any persistent layer and store the static content in memory. We'll store the result on Amazon S3 Files in JSON format.

You can find the code examples in my aws-s3-files-lambda-durable-functions-java-sdk repository. We'll go through all the examples in more detail in the subsequent articles.

First of all, let's declare some important dependencies in pom.xml. The most important one is the AWS Lambda Durable Execution SDK for Java:

<dependency>
    <groupId>software.amazon.lambda.durable</groupId>
    <artifactId>aws-durable-execution-sdk-java</artifactId>
    <version>2.0.0</version>
</dependency>

Why do we need such an SDK? Under the hood, durable functions are regular Lambda functions using a checkpoint/replay mechanism to track progress and support long-running operations through user-defined suspension points, commonly referred to as durable execution. After your function resumes from a pause or interruption, the system performs replay. During replay, your code runs from the beginning but skips over completed checkpoints, using stored results instead of re-executing completed operations. This replay mechanism ensures consistency while enabling long-running executions.

To harness this checkpoint-and-replay mechanism in your applications, Lambda provides a durable execution SDK. The SDK abstracts away the complexity of managing checkpoints and replay, exposing simple primitives called durable operations that you use in your code. The SDK integrates seamlessly with your existing Lambda development workflow.

To provide the implementation of our Lambda durable function, we need the class to extend DurableHandler and implement only the method handleRequest. Let's do it for our AuthorContentExtractor Lambda durable function:

public class AuthorContentExtractor extends DurableHandler<Author, AuthorContent> implements AbstractAuthorContentExtractor {

@Override
public AuthorContent handleRequest(Author author, DurableContext ctx) {
   var config = this.getStepConfig();       
   var upcomingTalks= ctx.step("searchForUpcomingTalks-step", UpcomingTalks.class, stepCtx -> this.searchForUpcomingTalks(), config);
   var youtubeVideos= ctx.step("searchForYouTubeVideos-step", YouTubeVideos.class, stepCtx -> this.searchForYouTubeVideos(), config);

   var authorContent = new AuthorContent(author, upcomingTalks, youtubeVideos);

   ctx.step("writeAuthorContentToFile-step", Void.class, stepCtx -> this.writeAuthorContentToFile(authorContent), config);      

    return authorContent;
   }
}

Let's go step by step through what's happening here. First of all, we also implement the AbstractAuthorContentExtractor interface. We place some common methods there that we'll use when improving our application.

Now let's introduce the concept of durable steps. Steps run business logic with built-in retries and automatic checkpointing. Each step saves its result, so your function resumes from the last completed step after an interruption. With the SDK, you wrap your Lambda event handler, which then provides a DurableContext alongside your event. This context gives you access to durable operations like steps. You write your function logic as normal sequential code, but instead of calling services directly, you wrap those calls in steps for automatic checkpointing and retries. When you need to pause execution, you add waits that suspend your function without incurring charges. We'll show it in the next part of the series. The SDK handles all the complexity of state management and replay behind the scenes, so your code remains clean and readable.

Here is the picture, which describes the concept that I took from this source:

We wrapped several operations into separate durable steps:

searchForUpcomingTalks-step
searchForYouTubeVideos-step
writeAuthorContentToFile-step

Let's look into the searchForUpcomingTalks-step first:

var upcomingTalks= ctx.step("searchForUpcomingTalks-step", UpcomingTalks.class, 
stepCtx -> this.searchForUpcomingTalks(), config);

To create a step, we need to give it a name and the result type. We also have to give it the function to execute, receiving a StepContext. The return type of the function should be the same as the result type. In our case, we execute the searchForUpcomingTalks function, which returns the static list of the upcoming talks. Optionally, we can also pass the StepConfig. In our case, we create it like this:

var stepConfig = StepConfig.builder()
    .semanticsPerRetry(StepSemantics.AT_LEAST_ONCE_PER_RETRY)                 
    .retryStrategy(RetryStrategies.exponentialBackoff(                                
      3, // max attempts
      Duration.ofSeconds(2),   // initial delay  
      Duration.ofSeconds(30),   // max delay
      2.0,                      // backoff multiplier
      JitterStrategy.FULL))
   .build();

We create the StepConfig by passing the following:

retry strategy: fixed delay, linear backoff, or exponential backoff. We pass the latter here.
semantics per retry: at-least-once delivery (default, which we also pass). The step may be re-executed if interrupted. START checkpoint is fire-and-forget. At-most-once delivery per retry attempt. START checkpoint is awaited before user code runs.
custom serializer for the step with serDes method. We don't provide it here, as we're happy with the default one.

searchForYouTubeVideos-step step works the same. After we have collected the result of the search for the upcoming talks and YouTube videos, we use the writeAuthorContentToFile-step step to serialize the result to JSON and write it to the file:

public default Void writeAuthorContentToFile(AuthorContent authorContent) {
  var authorContentAsJson = OBJECT_MAPPER.writeValueAsString(authorContent);
  var fileName= authorContent.author().firstName()+"-"+authorContent.author().lastName()+".json";
  Path path = Paths.get(WORKSPACE_MOUNT, fileName);
  byte[] strToBytes = authorContentAsJson.getBytes();
  Files.write(path, strToBytes);
  return null;
}

We use the standard Java Path API to do it. This is because we use Amazon S3 Files, and it supports POSIX. Strictly speaking, we could also use S3 here, but I wanted to show the functionality of S3 Files.

The directory where we store the file is passed as an environment variable WORKSPACE_MOUNT. In our case, it is /mnt/workspace. See the Infrastructure as Code explanation below, especially the LocalMountPath setting of the file system configuration for the Lambda function.

For infrastructure as code (IaC), I use AWS SAM, and you can find it here. To indicate that our Lambda function is durable, we need to provide some additional durable config properties:

  AuthorContentExtractorFunction:
    Type: AWS::Serverless::Function
    ...
    Properties:
      FunctionName: AuthorContentExtractor
      Handler: dev.vkazulkin.handler.AuthorContentExtractor::handleRequest
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
        ...

Execution timeout. The execution timeout controls how long a durable execution can run from start to completion. This is different from the Lambda function timeout, which controls how long a single function invocation can run. A durable execution can span multiple Lambda function invocations as it progresses through checkpoints, waits, and replays. The execution timeout applies to the total elapsed time of the durable execution, not to individual function invocations. Set the Execution timeout value in seconds (default: 86400 seconds / 24 hours, minimum: 60 seconds, maximum: 31536000 seconds / 1 year).
Retention period. The retention period controls how long Lambda retains execution history and checkpoint data after a durable execution completes. This data includes step results, execution state, and the complete checkpoint log. After the retention period expires, Lambda deletes the execution history and checkpoint data. You can no longer retrieve execution details or replay the execution. Set the Retention period value in days (default: 14 days, minimum: 1 day, maximum: 90 days).

For a very detailed explanation of how to create S3 Files and mount them to the Lambda function, please read the brilliant article Lambda Just Got a File System. I Put AI Agents on It by Eric Johnson. I mostly copied the IaC part from it and adjusted it to my needs. I only describe the main steps here:

First, we reference the networking stack, which we place in a separate file. Then we create:

VPC networking for Lambda functions using S3 Files.
VPC with private subnets (for Lambda + mount targets).
Security groups for NFS traffic.
Public subnet with a NAT gateway. We'll need them for internet access to Bedrock and Bedrock AgentCore Web Search Tool. In later parts, we'll use both services.

  NetworkingStack:
    Type: AWS::Serverless::Application
    Properties:
      Location: src/main/resources/stacks/network.yaml

Next, we need to create an S3 Bucket:

 WorkspaceBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName : vadym-s3-files-workspace
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled

Please don't forget to rename the S3 Bucket. After that, we need to create S3 Files:

  S3FileSystem:
    Type: AWS::S3Files::FileSystem
    Properties:
      Bucket: !GetAtt WorkspaceBucket.Arn
      RoleArn: !GetAtt S3FilesRole.Arn
      AcceptBucketWarning: true

Here we reference an already created S3 Bucket. We also need to profile the appropriate IAM Role (see the SAM template for the IaC). Next, we need to create 2 different mount targets for the S3 File. They should use private subnets in different availability zones. Here is an example of one such mount target:

  MountTargetA:
    Type: AWS::S3Files::MountTarget
    Properties:
      FileSystemId: !GetAtt S3FileSystem.FileSystemId
      SubnetId: !GetAtt NetworkingStack.Outputs.PrivateSubnetAId
      SecurityGroups:
        - !GetAtt NetworkingStack.Outputs.MountTargetSGId

Next, we need to create an S3 Files access point:

  S3FilesAccessPoint:
    Type: AWS::S3Files::AccessPoint
    Properties:
      FileSystemId: !GetAtt S3FileSystem.FileSystemId
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        Path: /lambda
        CreationPermissions:
          OwnerUid: '1000'
          OwnerGid: '1000'
          Permissions: '755'

The CreationPermissions property is crucial. It auto-creates the /lambda directory within our S3 bucket with the right ownership when a client first connects. Without it, the root directory is owned by root (UID 0), and Lambda (running as UID 1000 through the access point) can’t create subdirectories.

Lastly, we need to provide configuration and give permissions to our Lambda function to use S3 Files:

  AuthorContentExtractorFunction:
    Type: AWS::Serverless::Function
    DependsOn:
      - MountTargetA
      - MountTargetB
    Properties:
      FunctionName: AuthorContentExtractor
      Handler: dev.vkazulkin.handler.AuthorContentExtractor::handleRequest
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
      VpcConfig:
        SecurityGroupIds:
          - !GetAtt NetworkingStack.Outputs.LambdaSGId
        SubnetIds:
          - !GetAtt NetworkingStack.Outputs.PrivateSubnetAId
          - !GetAtt NetworkingStack.Outputs.PrivateSubnetBId
      FileSystemConfigs:
        - Arn: !GetAtt S3FilesAccessPoint.AccessPointArn
          LocalMountPath: /mnt/workspace
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Sid: MountS3Files
              Effect: Allow
              Action:
                - s3files:ClientMount
                - s3files:ClientWrite
                - s3files:ClientRootAccess
              Resource: !GetAtt S3FileSystem.FileSystemArn

Here we provide the VPC and file system configuration (including local mount path) and the policy for our Lambda function to use the S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractor function, and go to the "Test" tab:

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

Input and output as JSON
Logger output
Details like durable operations (with individual inputs and outputs of each step) and event history. If something goes wrong, the step status will be set to failed.

In our case, we invoked the durable function directly from the Lambda console. As an alternative, we can also use the Lambda SDK for it. We can also put an API Gateway in front of it if you wish and even invoke the Lambda function asynchronously.

We can also see the JSON file with author content in the S3 Bucket that we created. Alternatively, I implemented a GetAuthorContentResult Lambda function with the name GetAuthorContentResult. This Lambda function takes the author's first and last name and streams the content of the JSON file from the S3 bucket. This JSON file is stored in the subdirectory /lambda of our created S3 bucket. I also created and put an API Gateway in front of this Lambda function. Please use the /result/{firstname}/{lastname} HTTP GET endpoint for it, for example /result/Vadym/Kazulkin. Be aware that it takes several minutes for the file to appear in the S3 Bucket after being written to S3 Files. If you need the result immediately, you can rewrite this Lambda to retrieve the result from S3 Files instead.

This was a very simple application to explore the durable steps with the AWS Lambda Durable Execution SDK for Java. Let's improve our application a bit. Until now, we searched for the upcoming talks and YouTube videos sequentially. But we can do it in parallel. For that, we can use the stepAsync operation instead of step. I provided a separate implementation, AsyncAuthorContentExtractor, and the Lambda durable function with the name AsyncAuthorContentExtractor for it:

var upcomingTalksFuture= ctx.stepAsync("searchForUpcomingTalks-async-step", UpcomingTalks.class, 
stepCtx -> this.searchForUpcomingTalks(), config);
var youtubeVideosFuture= ctx.stepAsync("searchForYouTubeVideos-async-step", YouTubeVideos.class, 
stepCtx -> this.searchForYouTubeVideos(), config);

var upcomingTalks=upcomingTalksFuture.get();
var youtubeVideos= youtubeVideosFuture.get();

Now, stepAsync invocation returns a DurableFuture of the result type - a future representing the step result. This operation is non-blocking; we only block when invoking the get method on the result. Everything else remains the same in this asynchronous implementation as it was in the synchronous one.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. In the next part, we'll extend our application by adding and implementing wait and callback operations to it. Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why in the later parts, we'll move the logic associated with each step into a separate Lambda function. We'll explore how to invoke another Lambda function within the durable step. We'll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 1 Introduction to the series and services

Vadym Kazulkin — Mon, 10 Aug 2026 13:00:00 +0000

Introduction to the series

Throughout this article series, we'll develop a simple application to show how to build multi-step applications and orchestrate workflows using AWS Lambda durable functions. For it, we'll explore the AWS Lambda Durable Execution SDK for Java. This simple application provides the functionality to extract the content for the provided user. This content involves searching for YouTube videos of the authors and their upcoming talks. We'll store the result on Amazon S3 Files in JSON format. As this operation may potentially take time, we'll additionally provide access to the author content through the Amazon API Gateway:

We'll start with the simple application to search for YouTube videos and upcoming talks of the author synchronously. First, we'll use durable steps and then later add waits and callbacks. We won't implement any persistent layer and store the static content in memory.
We'll then improve our application by adding asynchronous communication where it makes sense. For example, searching for YouTube videos and upcoming talks can be done in parallel.
We'll then further improve our application by converting the AWS durable function to a pure orchestrator. This means we'll move the business logic for searching for YouTube videos and upcoming talks into separate Lambda functions. With that, our Lambda durable function will invoke those separate Lambda functions in parallel and combine the results
Finally, we'll further extend our application to use Amazon Bedrock AgentCore Web Search Tool to search for the author content on the web. With that, our Lambda durable function becomes an orchestrator of the agentic AI workflows. As we'll use Java in this series, we'll explore how to run this business logic responsible for the Amazon Bedrock AgentCore Web Search Tool. As this tool is exposed as an MCP tool by the Amazon Bedrock AgentCore Gateway, we need a framework capable of providing AI functionality and supporting MCP. That's why we'll use Spring AI. As 2 operational environments for our agentic AI workflows, we'll explore an AWS Lambda function and Amazon Bedrock AgentCore Runtime.

You can find the code examples in my aws-s3-files-lambda-durable-functions-java-sdk repository. We'll go through all the examples in more detail in the subsequent articles.

Introduction to the services used throughout the series

Throughout this series, we'll use the following AWS services:

AWS Lambda durable functions. With Lambda durable functions, you can build resilient multi-step applications and AI workflows. These can execute for up to one year while maintaining reliable progress despite interruptions. When a durable function runs, this complete lifecycle is called a durable execution, which uses checkpoints to track progress and automatically recover from failures through replay, re-executing from the beginning while skipping completed work. Within each function, you use durable operations as fundamental building blocks. Steps execute business logic with built-in retries and progress tracking. Waits suspend execution without incurring compute charges, making them ideal for long-running processes. These processes include human-in-the-loop workflows or polling external dependencies.
Amazon S3 Files. S3 Files is a shared file system that connects any AWS compute resource directly with your data in Amazon S3. It provides fast, direct access to all of your S3 data as files with full file system semantics and low-latency performance, without your data ever leaving S3. Every file-based application, agent, and team can access and work with your S3 data as a file system using the tools they already depend on. Built using Amazon EFS, S3 Files gives you the performance and simplicity of a file system with the scalability, durability, and cost-effectiveness of S3. You can read, write, and organize data using file and directory operations. At the same time, S3 Files manages the synchronization of changes between your bucket and file system.
We'll also introduce in the later parts of this series the following Amazon Bedrock AgentCore services or features: Gateway, Web Search Tool, and Runtime.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Building AI Agents with Embabel, Spring AI and Amazon Bedrock AgentCore - Part 3 Develop Agents as web application with Embabel

Vadym Kazulkin — Mon, 03 Aug 2026 13:00:00 +0000

Introduction

In part 2, we developed our first AI agents with the Embabel shell. We learned many concepts of this framework, such as agents, actions, and goals. In this article, we'll slightly adjust our application to convert it into a web application.

Update from September 2026 Please read my article Building AI Agents with Embabel, Spring AI and Amazon Bedrock AgentCore - Part 4 Upgrade sample application to Embabel 1.5 to find out how to upgrade this sample application to Embabel version 1.5, which supports Spring Boot 4.x and Spring AI 2.x.

Implement AI agents as a web application with Embabel

You can look at my GitHub embabel-1.x-conference-app-agent-local repository for the example. We'll completely reuse the implementation of both agents, the CreateTalksAndApplyForConferencesAgent and the SearchForTalksAndApplyForConferencesAgent that we implemented in part 2.

Those agents are capable of responding to the following prompts:

"Please provide me with the list of conferences, including their IDs, with Java topics happening in 2027, with the call for papers open today. Also, provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference." SearchForTalksAndApplyForConferencesAgent is responsible for responding to this prompt.
"Please create a talk with a cool title (max 60 characters long) and description (max 300 characters long) about using Spring AI on the Amazon Bedrock AgentCore service. Then provide me with the list of conferences, including their IDs, with Java topics happening in 2026 and 2027, with the call for papers open today. Finally, for each conference, apply individually for it with the talk just created." CreateTalksAndApplyForConferencesAgent is responsible for responding to this prompt.

First, let's check that we declared the following dependencies in pom.xml:

 <dependency>
    <groupId>com.embabel.agent</groupId>
    <artifactId>embabel-agent-starter-bedrock</artifactId>
    <version>${embabel-agent.version}</version>
 </dependency>
 <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

Also, make sure not to declare the embabel-agent-starter-shell dependency, as we don't use Embabel shell in this article.

All we need to do now is to additionally implement the EmbabelAgentController controller.

Let's first implement a generic endpoint that will accept both prompts. Let's first autowire and create some objects coming from Embabel that we'll use for this:

@RestController
public class EmbabelAgentController {

private final AgentPlatform agentPlatform;
private final AgentInvocation<Domain.ConferenceApplications> invocation;

private final ProcessOptions processOptions = new ProcessOptions()
     .withVerbosity(new Verbosity()
     .withShowPrompts(true)
     .withShowLlmResponses(true)
     .withShowPlanning(true)
     .withDebug(true));


public EmbabelAgentController(AgentPlatform agentPlatform) {
      this.agentPlatform = agentPlatform;

      this.invocation = AgentInvocation
            .builder(agentPlatform)
        .options(processOptions)
        .build(Domain.ConferenceApplications.class);
  }
...

}

First, we autowire the Embabel's AgentPlatform object and use it to create the AgentInvocation object. We additionally pass the ProcessOption object that is configured to show planning, LLM responses, and prompts. We also pass the type of the response - the domain object returned after the agents achieve their goals; in our case, the Domain.ConferenceApplications class:

Now, let's implement the endpoint itself:

@GetMapping(value = "/generic", consumes = "text/plain")
public Domain.ConferenceApplications genericPrompt(@RequestParam String prompt) {
    var inputs = Map.of("request", new UserInput(prompt)); 
    return this.invocation.run(inputs)
         .last(Domain.ConferenceApplications.class);    
}

Here we wrap the prompt into Embabel's UserInput object that we put into the map with all inputs required to run the agent. Then we use the AgentInvocation object to run the agent with the specified input synchronously. We can also invoke runAsync to run the agent asynchronously. Finally, we retrieve the result of the agent invocation with the specified type.

Let's build the application with mvn clean package and run it locally with mvn spring-boot:run. We'll send the prompt number 1 to this endpoint. What we'll see is that Embabel picked the wrong agent, CreateTalksAndApplyForConferencesAgent, to run. Why? As we don't use the Embabel shell, Embabel doesn't analyze the prompt and compute the score to determine what exact agent to run. It currently only looks into the return type of all agents and picks the first one that matches the type defined in the AgentInvocation object. But both our agents have the same return type Domain.ConferenceApplications. Their goals, in the end, are the same: to apply for the conferences. With that, for our 2 agents, there is a 50% probability that Embabel will pick the wrong agent to run.

How can we fix that? For example, we can define the endpoint per agent. Let's do this:

@GetMapping(value = "/applyToConferencesWithExistingTalks", consumes = "text/plain")
public Domain.ConferenceApplications applyToConferencesWithExistingTalks(@RequestParam String prompt) {
    return this.invokeAgent(prompt,
       SearchForTalksAndApplyForConferencesAgent.AGENT_NAME);   
}

@GetMapping(value = "/applyToConferencesWithNewTalks", consumes = "text/plain")
public Domain.ConferenceApplications applyToConferencesWithNewTalks(@RequestParam String prompt) {
    return this.invokeAgent(prompt,
        CreateTalksAndApplyForConferencesAgent.AGENT_NAME); 
}

I additionally wrote 2 helper methods, invokeAgent and getAgentByName, to simplify the agent search by name and its subsequent invocation:

private Domain.ConferenceApplications invokeAgent(String prompt, String agentName) {        
     var inputs = Map.of("request", new UserInput(prompt));
     var agent = this.getAgentByName(agentName);
     var agentProcess = this.agentPlatform.createAgentProcess(agent,
 processOptions, inputs);
     return agentProcess.run().last(Domain.ConferenceApplications.class);
}

private Agent getAgentByName(String agentName) {
     var optionalAgent = this.agentPlatform.agents()
        .stream()
        .filter(a -> a.getName().equals(agentName))
        .findFirst();

     if( optionalAgent.isEmpty()) {
       throw new RuntimeException("agent with the name "+agentName+ " not found");
     }

    return optionalAgent.get();
}

Let's explain what happens in the invokeAgent method. We first invoke the getAgentByName method. It searches in the collection of agents that the agent platform knows for one that has a specified name. All classes annotated with Embabel's Agent annotation will be registered in the agent platform. Then we use the AgentPlatform object to create the AgentProcess and pass the found agent, the already described process options, and inputs to it. Then we run the AgentProcess and return its last output of type Domain.ConferenceApplications.class.

Now we send prompt number 1 (see above) to the applyToConferencesWithExistingTalks endpoint and prompt number 2 to the applyToConferencesWithNewTalks endpoint. We'll see something similar as a result, as we saw in part 2.

For example, here I use httpie to send prompt number 1:
http GET http://localhost:8080/applyToConferencesWithExistingTalks?prompt="Please provide me with the list of conferences including their IDs with Java topic happening in 2026 and 2027 with call for papers open today. Also provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference." Content-Type:text/plain.

Here is the response of the agent:

Having one endpoint per Embabel agent has its pros and cons. The biggest disadvantage is that a service like AgentCore Runtime exposes only one HTTP POST /invocations endpoint to send the prompt to the web application. This means that we'll need to think about how to design, implement, and deploy each Embabel agent on a separate AgentCore Runtime. Each agent belonging to the same web application will have a lot of shared logic with other agents. This is not very convenient. The biggest advantage is that you can scale each Embabel agent individually. AgentCore Runtime will do it for you.

It's worth checking in the upcoming versions of Embabel whether there are other options available to invoke the agent.

Conclusion

In this article, we covered how to develop a web application with Embabel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

EventBridge Scheduler DLQ'd My Successful Invokes

Danielle Heberling — Sat, 01 Aug 2026 18:00:00 +0000

Update (September 8, 2026): Async didn’t fix the timeouts. I still don’t know why they happen, but I’m leaving this post up for what I learned about async in AgentCore and Strands.

The DLQ alarm was already triggered, and no new job alerts meant I missed the duplicate emails. I disabled retries and removed the DLQ and alarm. I’m okay with risking a missed search here.

I now get emails for errors, matches, and no matches. No email means I should check whether the search or the notification failed.

A while back I wrote about letting an AI agent do your job searching. That post ended with a quirk I couldn't explain. EventBridge Scheduler would invoke my agent, the agent would run fine, I'd get the email, and the invocation would land in the dead letter queue anyway. Every single time.

I figured I was either doing something wrong or hitting an AgentCore bug. Turns out it was neither.

The Actual Problem

EventBridge Scheduler universal targets (the feature that lets Scheduler call almost any AWS API action directly, no Lambda in between) are synchronous. Scheduler makes the API call and waits for a response before deciding whether the invocation succeeded. The docs describe this as at-least-once delivery, where at least one delivery succeeds with a response from the target. I glossed over that last part.

My schedule was calling the Bedrock AgentCore invokeAgentRuntime action, which blocks until the agent finishes. A search takes 30 to 75 seconds. Scheduler gives up around the 30 second mark and marks the invocation failed. The agent, which has no idea any of this happened, keeps going, finishes the search, and publishes to SNS. I get my email. The "failed" invocation goes to the DLQ. Here's what one of those messages looks like:

Everything downstream worked. The only broken part was Scheduler's opinion of it.

About That 30 Seconds

I went looking for this in the docs and could not find it. The EventBridge Scheduler quotas page covers number of schedules, API request rates, and invocation throughput. Nothing about how long Scheduler waits for a target to respond.

The one timeout you can configure is MaximumEventAgeInSeconds in the retry policy. That's the maximum age of an event across retries, not a per-call limit. Different thing.

So treat ~30 seconds as an observed number, not a contract. In my opinion the safer bet is designing against "Scheduler will not wait long" rather than a specific value.

The Fix

Instead of making the search faster, I made the response faster. The entrypoint now kicks the search off as a background task and returns immediately, so Scheduler gets its response well inside the window and the agent runs for as long as it needs to (up to AgentCore's 8 hour session cap, which my searches are nowhere near).

AgentCore has first-class support for this. You register the background work with add_async_task, the runtime reports HealthyBusy on /ping while it's in flight, and the session stays alive instead of getting reaped. That last part matters because AgentCore terminates sessions after 15 minutes of idle, and without task tracking your background work looks exactly like idle. The async processing docs cover the pattern.

Here's the relevant part of the entrypoint:

# strong refs: asyncio only weakly references tasks
_background_tasks: set[asyncio.Task[Any]] = set()

async def _tracked_job_search(company: str, title: str, location: str) -> None:
    """Run the job search as a tracked async task so ping reports HealthyBusy."""
    task_id = app.add_async_task("job_search")
    try:
        await run_job_search(company, title, location)
    finally:
        app.complete_async_task(task_id)

@app.entrypoint
async def invoke(payload: dict[str, Any] | None = None) -> dict[str, Any]:
    """Main entrypoint for the agent invocation."""
    # ...payload validation...

    if payload.get("sync"):
        return await run_job_search(company, title, location)

    # Respond before EventBridge Scheduler's ~30s call timeout DLQs the invocation
    task = asyncio.create_task(_tracked_job_search(company, title, location))
    _background_tasks.add(task)
    task.add_done_callback(_background_tasks.discard)

    return {
        "status": "accepted",
        "search_criteria": {"company": company, "title": title, "location": location},
    }

Scheduler gets {"status": "accepted"} in well under a second. The search runs in the background. No more DLQ.

A Few Things to Watch For

The Strands call has to be awaited. I was calling agent(prompt) inside an async def entrypoint, which blocks the event loop. Survivable when the entrypoint blocked anyway. But once the runtime needs to answer health checks mid-search, a blocking call starves everything. Switching to await agent.invoke_async(prompt) was required, not a style preference.

Hold a reference to the task. asyncio only weakly references tasks from create_task, so an unreferenced task can get garbage collected mid-run. That's the _background_tasks set above, lifted straight from the pattern in the Python docs. It usually works fine right up until it doesn't.

Keep a synchronous path. Fire and forget is great for the scheduler and terrible for local development. A "sync": true flag in the payload runs the search inline and returns the full result. That's what I use for curl and the AgentCore playground.

One More Thing About Retries

In the original post I disabled retries, because Scheduler retrying a "failed" invocation meant duplicate agent runs and duplicate emails. The CDK target default is 185 retries with exponential backoff over up to 24 hours, so every scheduled search was set up to keep re-running (and re-billing) until I caught it.

With the async path, I turned retries back on (retryAttempts: 0 to 2). Now Scheduler gets an ack right away, so it won’t keep retrying while the search is still running. There’s still a small chance of a duplicate if the ack gets lost, but that’s a trade-off I’m comfortable with here.

There's a tradeoff here. Scheduler now only confirms your agent started. It says nothing about whether the search actually worked. If the search blows up 40 seconds in, Scheduler is happy and your DLQ stays empty. You need logs, metrics, or the notification itself to know the work finished. For me a missing email is a loud signal. If your silence is more ambiguous, plan for that.

If You're Hitting This

If you're invoking AgentCore Runtime from an EventBridge schedule and seeing DLQ messages for runs that clearly worked:

Check how long your agent actually takes. Anything over ~30 seconds is suspect.
Return an ack from your entrypoint instead of the result.
Wrap the real work in add_async_task / complete_async_task so the runtime knows to keep the session alive.
Make sure nothing in that background path blocks the event loop.

Code is in the job-search-agent repo and the change is PR #51 if you want to see the whole diff.

None of this is AgentCore specific. Any long-running universal target invoked from EventBridge Scheduler has the same problem. Agents just happen to be really good at being slow. 😅

If you've hit this with a different long-running target, or if you've found the actual timeout documented somewhere I missed, I'd love to hear about it.