DEV Community: AWS Heroes

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 4 Wait for condition

Vadym Kazulkin — Mon, 31 Aug 2026 14:00:00 +0000

Introduction

In part 2 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously.
Later, in part 3 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement wait and callback.

In this part, we'll explore the wait for condition operation.

Sample application with wait for condition

Let's explore the "wait for condition" operation. We get access to it through DurableContext. It repeatedly calls a check function until it signals done. Between polls, the Lambda suspends without consuming compute. State is checkpointed after each check, so progress survives interruptions.

I first copied the application that we created in part 3 into the aws-s3-files-lambda-durable-functions-with-wait-for-condition-java-25 repository. The only change we had to make was to reimplement the waitForUpcomingTalksApproval method of the AbstractAuthorContentExtractor class. Here we'll now use the "wait for condition" instead of the "wait for callback" operation.

This is how the code now looks:

public default UpcomingTalkApprovalStatus waitForUpcomingTalksApproval(DurableContext ctx, Author author, UpcomingTalks upcomingTalks) {

  var config = WaitForConditionConfig.<UpcomingTalkApprovalStatus>builder()                
     .waitStrategy(WaitStrategies.defaultStrategy())
     .initialState(new UpcomingTalkApprovalStatus(upcomingTalks, 
         UpcomingTalkApprovalStatus.Status.PENDING))
     .build();

  return ctx.waitForCondition(
     "wait-for-approval",
     UpcomingTalkApprovalStatus.class, 
         (currentStatus, stepCtx) -> {
         var latest = this.getUpcomingTalkApprovalStatus(author,upcomingTalks);
         return  
         UpcomingTalkApprovalStatus.Status.APPROVED.equals(latest.status())                     
         ? WaitForConditionResult.stopPolling(latest)
         : WaitForConditionResult.continuePolling(latest);
    },
    config);
}

private UpcomingTalkApprovalStatus getUpcomingTalkApprovalStatus(Author author, UpcomingTalks upcomingTalks) {
     // Generate random integers in range 0 to 3
     int r = RANDOM.nextInt(4);
     return r < 3 ? 
       new UpcomingTalkApprovalStatus(upcomingTalks, 
          UpcomingTalkApprovalStatus.Status.NOT_APPROVED) :
       new UpcomingTalkApprovalStatus(upcomingTalks, 
          UpcomingTalkApprovalStatus.Status.APPROVED);
}

Let's go step by step through this code. The ctx.waitForCondition invocation receives the current state and a StepContext, and returns a WaitForConditionResult. WaitForConditionResult contains the result type, in our case UpcomingTalkApprovalStatus :

WaitForConditionResult.stopPolling(value) - condition met (in our case, status "APPROVED"), return value as the final result.
WaitForConditionResult.continuePolling(value) - keep polling (bacause the status is "NOT_APPROVED"), pass value to the next check.

For the sake of simplicity, we haven't implemented the proper endpoint to poll the status. We provided a simple implementation in the getUpcomingTalkApprovalStatus method, which generates a random integer between 0 and 3. If 3 is generated, the "APPROVED" status will be returned, indicating that the condition is met and the polling should stop. If 0, 1, or 2 are generated, the operation will keep polling.

Next, let's look at how to configure WaitForConditionConfig. First, we pass the initialState with the status "PENDING," and this status is passed to the first check invocation. After that, we also configure the waitStrategy. The wait strategy controls the delay between polls. By default, waitForCondition uses an exponential backoff wait strategy (60 max attempts, 5s initial delay, 300s max delay, 1.5x backoff rate, FULL jitter). We can invoke the WaitStrategies.exponentialBackoff method to configure exponential backoff with different settings than the default ones. Alternatively, we can use the WaitStrategies.fixedDelay method. With this wait strategy, we need to configure:

fixedDelay - a constant delay between polls and
maxAttempts - a maximum number of attempts before throwing WaitForConditionFailedException. This exception is thrown when the maximum attempts are exceeded (thrown by the wait strategy).

We can also implement Custom Wait Strategies, which we won't cover in this article.

That's it. There are no changes in the Infrastructure as Code part that we need to make. Now we can build and package our application with mvn clean package and deploy it with sam deploy. The deployment process can take up to 10 minutes because of the creation and mounting of S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractor or AsyncAuthorContentExtractor function, and go to the "Test" tab.

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

Here we see that the operation with the name wait-for-approval and the subtype WaitForCondition succeeded after 3 attempts. The number of retries is 2. This means that it took 3 attempts to generate the integer 3. We defined the generation of this value as the condition to stop polling and return the result.

Let's also look at event history:

Here we see all events (durable operations) in the right order required for the execution of our Lambda durable function.

There is also the waitForConditionAsync operation for non-blocking polling that you can explore on your own. It starts polling but returns a DurableFuture immediately, allowing other operations to run concurrently.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement the wait for condition operation.

In the next part, we'll explore map and child context operations. Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why, in the subsequent parts, we'll move the logic for each step into a separate Lambda function. With that, we'll explore how to invoke another Lambda function within the durable step. We'll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

DynamoDB Streams is not an outbox

Allen Helton — Tue, 25 Aug 2026 16:39:57 +0000

Hopefully you didn't notice, but a couple of weeks ago I sent an issue of my newsletter that never actually got emailed. My dashboard looked fine and it was marked as published, but the email never hit my inbox (or any of yours 😬).

I didn't notice until two days later when I went to look at the weekly statistics and found nothing. Just a Sent status and a bunch of 0's on the screen.

A quick browse through the code didn't turn up anything obvious. Emails are sent via an async handler triggered by an EventBridge event while the status of the newsletter is marked separately in DynamoDB, a classic "db save and domain event publish" pattern. We've all done it, it looks something like this:

await ddb.send(new UpdateItemCommand({ ... }));
await eventBridge.send(new PutEventsCommand({ ... }));

Despite its simple appearance, this is a nasty anti-pattern. Formally known as the dual write problem, if those two actions aren't contained in a single transaction, you're asking for trouble. Why? I'm glad you asked.

Imagine the write succeeds, but the publish doesn't. Data is updated but the follow-on actions that react to the domain event never run (like your email never being sent 🙃). Things fall into a bad state and you (maybe) end up stumbling upon a problem days later.

Ok, so flip them. Put the event first and the data save second. But if the publish succeeds and the db save fails, then you risk acting on stale data, which depending on your domain, is much worse.

Before you ask, no, serverless doesn't make it better. In many cases, serverless makes us too laissez-faire to solve it the "write" way (get it?).

What you're probably doing

Let's say you already knew not to do the data save and the event publish at the same time. I bet I know what you're doing instead. It's a pattern AWS has recommended for years and is all over Serverless Land.

You do your save to DynamoDB, let the change data flow into DynamoDB Streams, process it in Lambda, then fire your domain events from the stream processor.

When it comes down to it, this is legitimately better, and it's the same thought process behind going storage-first. You persist it first, then process it asynchronously. That makes the database write the event trigger, so now your domain events are tightly coupled to the corresponding data change. Plus, DynamoDB Streams retries automatically (and even bisects batches when configured) if there's a processing failure.

Sounds like our dual write problem is solved, right? Wrong. Kind of. It's complicated. 😅

DynamoDB Streams records are change data capture, not domain events. The Lambda function you write to handle them receives a NewImage and OldImage record, and it's up to you to infer what happened upstream. Smells like leaky business logic to me. Before you know it, you have this thousand-line file that looks at 40 different data shapes to determine which event to publish and which transformation to perform. As someone who's found themselves in that hole before, believe me when I say that is a nightmare situation.

Let's take it a step further. A business operation could (and probably does) affect multiple DynamoDB records. When I publish my newsletter, I create a DynamoDB record with a short TTL for each email address that was successfully sent to. If I am processing entity changes in a stream handler, I'd be up to my ears in duplicate newsletter.sent events because I'm working on individual records and not an aggregate operation. That ultimately leads to workarounds with atomic counters and lastAction flags that make it overly complex with little value add.

And most importantly, you don't know what did or didn't publish. Streams retain data for 24 hours, and if your publisher was broken for a day because of an outage or a rogue IAM change, you're out of luck. Even if you opted for something like Kinesis, which has 365 days of data retention, that's not the point. These mechanisms are change data, not the intended effect on the system.

DynamoDB Streams are a wonderful piece of engineering that gives you retries, but not the durability of intent.

What you should do instead

Time for a bit of a mental shift. We need to prioritize domain events. If an event is important enough to publish, it's important enough to persist in a database.

This is where the outbox pattern comes into play. Instead of deriving your domain events from the data change in a downstream handler, you save the event in an outbox table in a transaction along with the data mutation. The transaction guarantees both the domain event and the data change will save together (or neither).

With DynamoDB, that could look like this:

const eventId = ulid();

await ddb.send(new TransactWriteItemsCommand({
  TransactItems: [
    {
      Update: {
        TableName: 'newsletter',
        Key: marshall({ pk: issueId, sk: 'issue' }),
        UpdateExpression: 'SET #status = :sent',
        ...
      }
    },
    {
      Put: {
        TableName: 'outbox',
        Item: marshall({
          pk: issueId,
          sk: `EVENT#${eventId}`,
          status: 'PENDING',
          pendingBucket: 'PENDING',              // GSI1 PK, removed on publish
          pendingAt: new Date().toISOString(),   // GSI1 SK
          source: 'newsletter.service',
          detailType: 'newsletter.sent',
          detail: { issueId, date, idempotencyKey: eventId }
        })
      }
    }
  ]
}));

Funnily enough, the architecture doesn't change with this pattern. You're still doing Lambda (or whatever compute) to DynamoDB to a stream to another Lambda function. But this time the downstream Lambda function is processing the event you saved, not the mutated data. It doesn't have to infer anything.

AWS does a fantastic job in their transactional outbox design pattern docs showing you how to setup and implement this pattern yourself. So I won't go into more of the code here. But I will address some questions I had the first time I built this.

What do I do with an event after it's published?

After you publish an event in your stream handler, it's up to you to either delete it from the outbox or mark it as published. Personally, I prefer to mark it as published and set a 30 day TTL. This means in the stream handler, we're hitting the dual write problem again because we are both publishing an event and updating the outbox with a published status.

This one is ok though 😉 and it brings up an important point we haven't covered yet. Idempotency is crucial for success with this pattern. The outbox pattern trades duplicate delivery in exchange for never losing durable content. If your publisher fails to send the event or update the database with a published status, that's fine! Do it again. The important part is that the intent wasn't lost.

Can I replace the stream handler function with EventBridge Pipes?

Fewer pieces, easier system to maintain, right? In this instance, the Lambda function is the better solution over Pipes. If you were only publishing an event to EventBridge, you could easily filter the stream to the relevant outbox records and forward it straight to a bus. But because you also need to write back to DynamoDB to update the status of the event, the function wins. A pipe can only perform a single operation (outside of enrichments and transformations), but a function can perform many.

Does this guarantee ordering?

Nope. The outbox pattern is designed for correctness only. Eventually consistent state across a distributed system. This does not tackle other hard distributed system problems like ordering or exactly-once delivery. Quite the opposite in fact. While Streams do process change data in the order they happen, it's per shard (not globally), and you can't guarantee ordering when it comes to retries. And as soon as you drop an event on EventBridge, you lose any sort of ordering.

How does this guarantee events aren't skipped?

If you implemented this exactly as described, it wouldn't. It gives you the normal retry path you're already used to, but still has that 24-hour clock because of the stream data retention. The important piece that's missing is a scheduled reconciler. It's usually another Lambda function that queries a sparse GSI for anything still pending from more than five minutes ago, which is long enough that the stream handler has had its shot, and republishes it. So your full flow would look something like this:

Remember this

There is no clever workaround to the dual write problem. You can't order the calls in a way that mitigates the risk, you can't retry around it, and you can't/shouldn't move it into a stream handler and hope.

But you can stop having two writes. Well, two writes, but one transaction. 😅

Remember, what we're solving with the outbox pattern is a correctness problem. If someone asks "how can you guarantee the system is eventually consistent, even under backpressure?" this is what you reach for. It took me years to arrive here naturally and I've been accidentally building holes in my systems longer than I'd care to admit.

This pattern does make your architecture more complicated, but the trade-offs are often worth it, especially at scale. In exchange for 2x the WCUs, you get a solid way to store and act upon intent in your system. Anyway, take a look at how you're publishing events and see if there are any unexplained missing events in your system. If there are, you know what to do. 😉 That's what I did and you should be reliably getting your newsletters from now on.

Happy coding!

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 3 Waits and callbacks

Vadym Kazulkin — Mon, 24 Aug 2026 14:30:00 +0000

Introduction

In part 2 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. In this part, we'll extend our application by adding and implementing wait and callback operations to it.

Sample application with wait for callback

Let's explore waits and callbacks. We get access to them through DurableContext. Waits are planned pauses where your function stops running and stops charging until it's time to continue. Use them for time periods, external callbacks, or polling for a condition. Callbacks suspend execution until an external system sends a result. Use this for human approvals, webhooks, or any event-driven workflow. In our sample application, we'll combine both and use waitForCallback. It simplifies callback handling by combining callback creation and submission in one operation.

I first copied the application that we created in part 2 into the aws-s3-files-lambda-durable-functions-with-wait-for-callback-java-25 repository. We'll work on it to extend our application. Please go through the content of part 2 to understand the sample application and the basic concepts of Lambda durable functions. We'll completely reuse the Infrastructure as Code part as well.

Let's add another, a bit artificial use case to our sample application. After searching for the author's YouTube videos and upcoming talks, the author should confirm those talks. For this, let's extend our AsyncAuthorContentExtractor Lambda function and add the following invocation:

var upcomingTalkApprovalStatus = this.waitForUpcomingTalksApproval(ctx, author, upcomingTalks);

Next, let's look at the implementation of the waitForUpcomingTalksApproval method in the AbstractAuthorContentExtractor class:

public default UpcomingTalkApprovalStatus waitForUpcomingTalksApproval(DurableContext ctx, Author author, UpcomingTalks upcomingTalks) {
    var waitCallbackConfig= WaitForCallbackConfig.builder()
           .callbackConfig(CallbackConfig.builder()
           .timeout(Duration.ofHours(1))
           .build())
       .build();

   return ctx.waitForCallback(
            "wait-for-approval",
             UpcomingTalkApprovalStatus.class,
             (callbackId, stepCtx) 
              -> this.sendApprovalRequest(callbackId, author, upcomingTalks),
             waitCallbackConfig);
}

private void sendApprovalRequest(String callbackId, Author author, UpcomingTalks upcomingTalks) {
       LOGGER.info("get approval for the talk of  "+ author+ " for the talks "+upcomingTalks + 
       " with the callback id "+callbackId);
}

Let's explain step by step what is happening here. We use DurableContext to create the waitForCallback operation. We give it a name and the return type. This return type corresponds to the type of the object that will be passed in the callback. We also provide the submitter function sendApprovalRequest, receiving the callback ID and a StepContext. In this function, we can perform the desired operation, like sending the talk approval request to the author. We won't implement this functionality but only log the callback ID, which we'll require later to send the callback. We can optionally pass the WaitForCallbackConfig. In our case, we only set the CallbackConfig with a timeout of 1 hour. This means that if no callback arrives within this period of time, the Lambda durable function will fail. We can also set the StepConfig. Please review part 2 for more information.

When we invoke the waitForCallback method, the invocation will block until the callback arrives. After its arrival, the payload of the callback is deserialized into the object of type UpcomingTalkApprovalStatus, and this object will be returned.

Now we can build and package our application with mvn clean package and deploy it with sam deploy. The deployment process can take up to 10 minutes because of the creation and mounting of S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AsyncAuthorContentExtractor function, and go to the "Test" tab.

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin",
  "email": "bla@bla.de"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

What we see here is that the first 2 steps (searching for YouTube videos and upcoming talks) have already been executed asynchronously. Now, our Lambda function is in the "wait for the approval" state. The invocation of the Lambda durable function is now completed. Why? Because waits suspend our function without incurring charges.

Now let's send the reply (callback) to our Lambda durable function. We first need to grab the callback ID from the log. There are several ways to send the callback (success or failure). We can use the Lambda CLI send-durable-execution-callback-success or execution-callback-failure commands. Alternatively, we can use the AWS SDK to do the same. Let's use the AWS Java SDK for it. First, we need to add the following dependency to pom.xml:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>lambda</artifactId>
</dependency>

We implemented the functionality to send the callback in the SendDurableExecutionCallback class:

public class SendDurableExecutionCallback {

  private final static String CALLBACK_ID=
"Ab9hZXiWYXJuOmF3czpsYW1iZGE6dXMtZWFzdC0xOjI2NTYzNDI1NzYxMDpmdW5jdGlvbjpBdXRob3JDb250ZW50RXh0cmFjdG9yOiRMQVRFU1QvZHVyYWJsZS1leGVjdXRpb24vc2VhcmNoRm9yVmFkeW0xMzUvMmM2NzRiOWEtNTQ0OC0zZDVjLWI3MzUtNTQ1NWM5ODA5MWUwYWl4JDA2N2JkNmRkLTViN2MtNGZmYi05NmY4LTRkM2Y1Yjk3MWE0Zf8";

  private final static LambdaClient LAMBDA_CLIENT = 
  LambdaClient.builder().region(Region.US_EAST_1).build();

  private final static ObjectMapper MAPPER= new ObjectMapper();

  public static void main(String[] args) {

    var approvedStatus= MAPPER.writeValueAsString(
         new UpcomingTalkApprovalStatus(UpcomingTalks.getDefaultUpcomingTalks(), "approved"));

    var response= LAMBDA_CLIENT.sendDurableExecutionCallbackSuccess(
                  builder -> builder.callbackId(CALLBACK_ID)
                  .result(SdkBytes.fromUtf8String(approvedStatus))
                  .build());
    }
}

First, we need to configure the callback ID (please replace it with your individual one). Then, we create the Lambda client. Next, we invoke its sendDurableExecutionCallbackSuccess method to send the successful callback. We pass the callback ID and payload as JSON. This payload should be deserializable into the object of type UpcomingTalkApprovalStatus (see above). Similarly, we can send the failure callback by using the sendDurableExecutionCallbackFailure operation. Please move this logic with all required dependencies to the service that sends the callbacks. Now let's send the callback (approximately 23 minutes later) and see what happens:

We observe that the callback was successfully received and processed. After it, the last writeAuthorContentToFile-step step was executed as well. But we also observe that execution of our Lambda durable function started from the beginning. What happened here exactly? After our function resumes from a pause or interruption, the system performs replay. During replay, our code runs from the beginning but skips over completed checkpoints, using stored results instead of re-executing completed operations. This replay mechanism ensures consistency while enabling long-running executions.

Here is the picture, which describes the concept that I took from this source:

There are also other individual operations that AWS Lambda Durable Execution SDK for Java supports:

wait - wait suspends the function and resumes after the specified duration. There is no charge during suspension.
waitAsync - starts the wait timer but returns a DurableFuture immediately, allowing other operations to run concurrently. The execution only suspends when you call .get() on the future (if the wait hasn't completed yet).
createCallback - callbacks suspend execution until an external system sends a result. Use this for human approvals, webhooks, or any event-driven workflow.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement wait and callback.

In the next part, we'll explore the "wait for condition" operation to poll a condition until it's met. Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why, in the subsequent parts, we'll move the logic for each step into a separate Lambda function. With that, we'll explore how to invoke another Lambda function within the durable step. We'll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 2 Durable synchronous and asynchronous steps

Vadym Kazulkin — Mon, 17 Aug 2026 13:00:00 +0000

Introduction

In part 1, we introduced what we'll cover in this series and our sample application. We also gave a short introduction to the services that we'll use throughout this series: AWS Lambda durable functions and Amazon S3 Files. In this part of the series, we'll explore how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously.

First implementation attempt of our sample application

Throughout this article series, we'll develop a simple application to show how to build multi-step applications and orchestrate workflows using AWS Lambda durable functions. This simple application provides the functionality to extract the content for the provided user. To this content belongs searching for YouTube videos of the authors and their upcoming talks. We'll store the result on Amazon S3 Files in JSON format. As this operation may potentially take time, we'll additionally provide access to the author content through the Amazon API Gateway:

We'll start with the search for YouTube videos and upcoming talks of the author synchronously. For the sake of simplicity, we won't implement any persistent layer and store the static content in memory. We'll store the result on Amazon S3 Files in JSON format.

You can find the code examples in my aws-s3-files-lambda-durable-functions-java-sdk repository. We'll go through all the examples in more detail in the subsequent articles.

First of all, let's declare some important dependencies in pom.xml. The most important one is the AWS Lambda Durable Execution SDK for Java:

<dependency>
    <groupId>software.amazon.lambda.durable</groupId>
    <artifactId>aws-durable-execution-sdk-java</artifactId>
    <version>2.0.0</version>
</dependency>

Why do we need such an SDK? Under the hood, durable functions are regular Lambda functions using a checkpoint/replay mechanism to track progress and support long-running operations through user-defined suspension points, commonly referred to as durable execution. After your function resumes from a pause or interruption, the system performs replay. During replay, your code runs from the beginning but skips over completed checkpoints, using stored results instead of re-executing completed operations. This replay mechanism ensures consistency while enabling long-running executions.

To harness this checkpoint-and-replay mechanism in your applications, Lambda provides a durable execution SDK. The SDK abstracts away the complexity of managing checkpoints and replay, exposing simple primitives called durable operations that you use in your code. The SDK integrates seamlessly with your existing Lambda development workflow.

To provide the implementation of our Lambda durable function, we need the class to extend DurableHandler and implement only the method handleRequest. Let's do it for our AuthorContentExtractor Lambda durable function:

public class AuthorContentExtractor extends DurableHandler<Author, AuthorContent> implements AbstractAuthorContentExtractor {

@Override
public AuthorContent handleRequest(Author author, DurableContext ctx) {
   var config = this.getStepConfig();       
   var upcomingTalks= ctx.step("searchForUpcomingTalks-step", UpcomingTalks.class, stepCtx -> this.searchForUpcomingTalks(), config);
   var youtubeVideos= ctx.step("searchForYouTubeVideos-step", YouTubeVideos.class, stepCtx -> this.searchForYouTubeVideos(), config);

   var authorContent = new AuthorContent(author, upcomingTalks, youtubeVideos);

   ctx.step("writeAuthorContentToFile-step", Void.class, stepCtx -> this.writeAuthorContentToFile(authorContent), config);      

    return authorContent;
   }
}

Let's go step by step through what's happening here. First of all, we also implement the AbstractAuthorContentExtractor interface. We place some common methods there that we'll use when improving our application.

Now let's introduce the concept of durable steps. Steps run business logic with built-in retries and automatic checkpointing. Each step saves its result, so your function resumes from the last completed step after an interruption. With the SDK, you wrap your Lambda event handler, which then provides a DurableContext alongside your event. This context gives you access to durable operations like steps. You write your function logic as normal sequential code, but instead of calling services directly, you wrap those calls in steps for automatic checkpointing and retries. When you need to pause execution, you add waits that suspend your function without incurring charges. We'll show it in the next part of the series. The SDK handles all the complexity of state management and replay behind the scenes, so your code remains clean and readable.

Here is the picture, which describes the concept that I took from this source:

We wrapped several operations into separate durable steps:

searchForUpcomingTalks-step
searchForYouTubeVideos-step
writeAuthorContentToFile-step

Let's look into the searchForUpcomingTalks-step first:

var upcomingTalks= ctx.step("searchForUpcomingTalks-step", UpcomingTalks.class, 
stepCtx -> this.searchForUpcomingTalks(), config);

To create a step, we need to give it a name and the result type. We also have to give it the function to execute, receiving a StepContext. The return type of the function should be the same as the result type. In our case, we execute the searchForUpcomingTalks function, which returns the static list of the upcoming talks. Optionally, we can also pass the StepConfig. In our case, we create it like this:

var stepConfig = StepConfig.builder()
    .semanticsPerRetry(StepSemantics.AT_LEAST_ONCE_PER_RETRY)                 
    .retryStrategy(RetryStrategies.exponentialBackoff(                                
      3, // max attempts
      Duration.ofSeconds(2),   // initial delay  
      Duration.ofSeconds(30),   // max delay
      2.0,                      // backoff multiplier
      JitterStrategy.FULL))
   .build();

We create the StepConfig by passing the following:

retry strategy: fixed delay, linear backoff, or exponential backoff. We pass the latter here.
semantics per retry: at-least-once delivery (default, which we also pass). The step may be re-executed if interrupted. START checkpoint is fire-and-forget. At-most-once delivery per retry attempt. START checkpoint is awaited before user code runs.
custom serializer for the step with serDes method. We don't provide it here, as we're happy with the default one.

searchForYouTubeVideos-step step works the same. After we have collected the result of the search for the upcoming talks and YouTube videos, we use the writeAuthorContentToFile-step step to serialize the result to JSON and write it to the file:

public default Void writeAuthorContentToFile(AuthorContent authorContent) {
  var authorContentAsJson = OBJECT_MAPPER.writeValueAsString(authorContent);
  var fileName= authorContent.author().firstName()+"-"+authorContent.author().lastName()+".json";
  Path path = Paths.get(WORKSPACE_MOUNT, fileName);
  byte[] strToBytes = authorContentAsJson.getBytes();
  Files.write(path, strToBytes);
  return null;
}

We use the standard Java Path API to do it. This is because we use Amazon S3 Files, and it supports POSIX. Strictly speaking, we could also use S3 here, but I wanted to show the functionality of S3 Files.

The directory where we store the file is passed as an environment variable WORKSPACE_MOUNT. In our case, it is /mnt/workspace. See the Infrastructure as Code explanation below, especially the LocalMountPath setting of the file system configuration for the Lambda function.

For infrastructure as code (IaC), I use AWS SAM, and you can find it here. To indicate that our Lambda function is durable, we need to provide some additional durable config properties:

  AuthorContentExtractorFunction:
    Type: AWS::Serverless::Function
    ...
    Properties:
      FunctionName: AuthorContentExtractor
      Handler: dev.vkazulkin.handler.AuthorContentExtractor::handleRequest
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
        ...

Execution timeout. The execution timeout controls how long a durable execution can run from start to completion. This is different from the Lambda function timeout, which controls how long a single function invocation can run. A durable execution can span multiple Lambda function invocations as it progresses through checkpoints, waits, and replays. The execution timeout applies to the total elapsed time of the durable execution, not to individual function invocations. Set the Execution timeout value in seconds (default: 86400 seconds / 24 hours, minimum: 60 seconds, maximum: 31536000 seconds / 1 year).
Retention period. The retention period controls how long Lambda retains execution history and checkpoint data after a durable execution completes. This data includes step results, execution state, and the complete checkpoint log. After the retention period expires, Lambda deletes the execution history and checkpoint data. You can no longer retrieve execution details or replay the execution. Set the Retention period value in days (default: 14 days, minimum: 1 day, maximum: 90 days).

For a very detailed explanation of how to create S3 Files and mount them to the Lambda function, please read the brilliant article Lambda Just Got a File System. I Put AI Agents on It by Eric Johnson. I mostly copied the IaC part from it and adjusted it to my needs. I only describe the main steps here:

First, we reference the networking stack, which we place in a separate file. Then we create:

VPC networking for Lambda functions using S3 Files.
VPC with private subnets (for Lambda + mount targets).
Security groups for NFS traffic.
Public subnet with a NAT gateway. We'll need them for internet access to Bedrock and Bedrock AgentCore Web Search Tool. In later parts, we'll use both services.

  NetworkingStack:
    Type: AWS::Serverless::Application
    Properties:
      Location: src/main/resources/stacks/network.yaml

Next, we need to create an S3 Bucket:

 WorkspaceBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName : vadym-s3-files-workspace
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled

Please don't forget to rename the S3 Bucket. After that, we need to create S3 Files:

  S3FileSystem:
    Type: AWS::S3Files::FileSystem
    Properties:
      Bucket: !GetAtt WorkspaceBucket.Arn
      RoleArn: !GetAtt S3FilesRole.Arn
      AcceptBucketWarning: true

Here we reference an already created S3 Bucket. We also need to profile the appropriate IAM Role (see the SAM template for the IaC). Next, we need to create 2 different mount targets for the S3 File. They should use private subnets in different availability zones. Here is an example of one such mount target:

  MountTargetA:
    Type: AWS::S3Files::MountTarget
    Properties:
      FileSystemId: !GetAtt S3FileSystem.FileSystemId
      SubnetId: !GetAtt NetworkingStack.Outputs.PrivateSubnetAId
      SecurityGroups:
        - !GetAtt NetworkingStack.Outputs.MountTargetSGId

Next, we need to create an S3 Files access point:

  S3FilesAccessPoint:
    Type: AWS::S3Files::AccessPoint
    Properties:
      FileSystemId: !GetAtt S3FileSystem.FileSystemId
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        Path: /lambda
        CreationPermissions:
          OwnerUid: '1000'
          OwnerGid: '1000'
          Permissions: '755'

The CreationPermissions property is crucial. It auto-creates the /lambda directory within our S3 bucket with the right ownership when a client first connects. Without it, the root directory is owned by root (UID 0), and Lambda (running as UID 1000 through the access point) can’t create subdirectories.

Lastly, we need to provide configuration and give permissions to our Lambda function to use S3 Files:

  AuthorContentExtractorFunction:
    Type: AWS::Serverless::Function
    DependsOn:
      - MountTargetA
      - MountTargetB
    Properties:
      FunctionName: AuthorContentExtractor
      Handler: dev.vkazulkin.handler.AuthorContentExtractor::handleRequest
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
      VpcConfig:
        SecurityGroupIds:
          - !GetAtt NetworkingStack.Outputs.LambdaSGId
        SubnetIds:
          - !GetAtt NetworkingStack.Outputs.PrivateSubnetAId
          - !GetAtt NetworkingStack.Outputs.PrivateSubnetBId
      FileSystemConfigs:
        - Arn: !GetAtt S3FilesAccessPoint.AccessPointArn
          LocalMountPath: /mnt/workspace
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Sid: MountS3Files
              Effect: Allow
              Action:
                - s3files:ClientMount
                - s3files:ClientWrite
                - s3files:ClientRootAccess
              Resource: !GetAtt S3FileSystem.FileSystemArn

Here we provide the VPC and file system configuration (including local mount path) and the policy for our Lambda function to use the S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractor function, and go to the "Test" tab:

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the "Durable execution" tab and can see all the execution details:

Input and output as JSON
Logger output
Details like durable operations (with individual inputs and outputs of each step) and event history. If something goes wrong, the step status will be set to failed.

In our case, we invoked the durable function directly from the Lambda console. As an alternative, we can also use the Lambda SDK for it. We can also put an API Gateway in front of it if you wish and even invoke the Lambda function asynchronously.

We can also see the JSON file with author content in the S3 Bucket that we created. Alternatively, I implemented a GetAuthorContentResult Lambda function with the name GetAuthorContentResult. This Lambda function takes the author's first and last name and streams the content of the JSON file from the S3 bucket. This JSON file is stored in the subdirectory /lambda of our created S3 bucket. I also created and put an API Gateway in front of this Lambda function. Please use the /result/{firstname}/{lastname} HTTP GET endpoint for it, for example /result/Vadym/Kazulkin. Be aware that it takes several minutes for the file to appear in the S3 Bucket after being written to S3 Files. If you need the result immediately, you can rewrite this Lambda to retrieve the result from S3 Files instead.

This was a very simple application to explore the durable steps with the AWS Lambda Durable Execution SDK for Java. Let's improve our application a bit. Until now, we searched for the upcoming talks and YouTube videos sequentially. But we can do it in parallel. For that, we can use the stepAsync operation instead of step. I provided a separate implementation, AsyncAuthorContentExtractor, and the Lambda durable function with the name AsyncAuthorContentExtractor for it:

var upcomingTalksFuture= ctx.stepAsync("searchForUpcomingTalks-async-step", UpcomingTalks.class, 
stepCtx -> this.searchForUpcomingTalks(), config);
var youtubeVideosFuture= ctx.stepAsync("searchForYouTubeVideos-async-step", YouTubeVideos.class, 
stepCtx -> this.searchForYouTubeVideos(), config);

var upcomingTalks=upcomingTalksFuture.get();
var youtubeVideos= youtubeVideosFuture.get();

Now, stepAsync invocation returns a DurableFuture of the result type - a future representing the step result. This operation is non-blocking; we only block when invoking the get method on the result. Everything else remains the same in this asynchronous implementation as it was in the synchronous one.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. In the next part, we'll extend our application by adding and implementing wait and callback operations to it. Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That's why in the later parts, we'll move the logic associated with each step into a separate Lambda function. We'll explore how to invoke another Lambda function within the durable step. We'll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 1 Introduction to the series and services

Vadym Kazulkin — Mon, 10 Aug 2026 13:00:00 +0000

Introduction to the series

Throughout this article series, we'll develop a simple application to show how to build multi-step applications and orchestrate workflows using AWS Lambda durable functions. For it, we'll explore the AWS Lambda Durable Execution SDK for Java. This simple application provides the functionality to extract the content for the provided user. This content involves searching for YouTube videos of the authors and their upcoming talks. We'll store the result on Amazon S3 Files in JSON format. As this operation may potentially take time, we'll additionally provide access to the author content through the Amazon API Gateway:

We'll start with the simple application to search for YouTube videos and upcoming talks of the author synchronously. First, we'll use durable steps and then later add waits and callbacks. We won't implement any persistent layer and store the static content in memory.
We'll then improve our application by adding asynchronous communication where it makes sense. For example, searching for YouTube videos and upcoming talks can be done in parallel.
We'll then further improve our application by converting the AWS durable function to a pure orchestrator. This means we'll move the business logic for searching for YouTube videos and upcoming talks into separate Lambda functions. With that, our Lambda durable function will invoke those separate Lambda functions in parallel and combine the results
Finally, we'll further extend our application to use Amazon Bedrock AgentCore Web Search Tool to search for the author content on the web. With that, our Lambda durable function becomes an orchestrator of the agentic AI workflows. As we'll use Java in this series, we'll explore how to run this business logic responsible for the Amazon Bedrock AgentCore Web Search Tool. As this tool is exposed as an MCP tool by the Amazon Bedrock AgentCore Gateway, we need a framework capable of providing AI functionality and supporting MCP. That's why we'll use Spring AI. As 2 operational environments for our agentic AI workflows, we'll explore an AWS Lambda function and Amazon Bedrock AgentCore Runtime.

You can find the code examples in my aws-s3-files-lambda-durable-functions-java-sdk repository. We'll go through all the examples in more detail in the subsequent articles.

Introduction to the services used throughout the series

Throughout this series, we'll use the following AWS services:

AWS Lambda durable functions. With Lambda durable functions, you can build resilient multi-step applications and AI workflows. These can execute for up to one year while maintaining reliable progress despite interruptions. When a durable function runs, this complete lifecycle is called a durable execution, which uses checkpoints to track progress and automatically recover from failures through replay, re-executing from the beginning while skipping completed work. Within each function, you use durable operations as fundamental building blocks. Steps execute business logic with built-in retries and progress tracking. Waits suspend execution without incurring compute charges, making them ideal for long-running processes. These processes include human-in-the-loop workflows or polling external dependencies.
Amazon S3 Files. S3 Files is a shared file system that connects any AWS compute resource directly with your data in Amazon S3. It provides fast, direct access to all of your S3 data as files with full file system semantics and low-latency performance, without your data ever leaving S3. Every file-based application, agent, and team can access and work with your S3 data as a file system using the tools they already depend on. Built using Amazon EFS, S3 Files gives you the performance and simplicity of a file system with the scalability, durability, and cost-effectiveness of S3. You can read, write, and organize data using file and directory operations. At the same time, S3 Files manages the synchronization of changes between your bucket and file system.
We'll also introduce in the later parts of this series the following Amazon Bedrock AgentCore services or features: Gateway, Web Search Tool, and Runtime.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Building AI Agents with Embabel, Spring AI and Amazon Bedrock AgentCore - Part 3 Develop Agents as web application with Embabel

Vadym Kazulkin — Mon, 03 Aug 2026 13:00:00 +0000

Introduction

In part 2, we developed our first AI agents with the Embabel shell. We learned many concepts of this framework, such as agents, actions, and goals. In this article, we'll slightly adjust our application to convert it into a web application.

Implement AI agents as a web application with Embabel

You can look at my GitHub embabel-1.x-conference-app-agent-local repository for the example. We'll completely reuse the implementation of both agents, the CreateTalksAndApplyForConferencesAgent and the SearchForTalksAndApplyForConferencesAgent that we implemented in part 2.

Those agents are capable of responding to the following prompts:

"Please provide me with the list of conferences, including their IDs, with Java topics happening in 2027, with the call for papers open today. Also, provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference." SearchForTalksAndApplyForConferencesAgent is responsible for responding to this prompt.
"Please create a talk with a cool title (max 60 characters long) and description (max 300 characters long) about using Spring AI on the Amazon Bedrock AgentCore service. Then provide me with the list of conferences, including their IDs, with Java topics happening in 2026 and 2027, with the call for papers open today. Finally, for each conference, apply individually for it with the talk just created." CreateTalksAndApplyForConferencesAgent is responsible for responding to this prompt.

First, let's check that we declared the following dependencies in pom.xml:

 <dependency>
    <groupId>com.embabel.agent</groupId>
    <artifactId>embabel-agent-starter-bedrock</artifactId>
    <version>${embabel-agent.version}</version>
 </dependency>
 <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

Also, make sure not to declare the embabel-agent-starter-shell dependency, as we don't use Embabel shell in this article.

All we need to do now is to additionally implement the EmbabelAgentController controller.

Let's first implement a generic endpoint that will accept both prompts. Let's first autowire and create some objects coming from Embabel that we'll use for this:

@RestController
public class EmbabelAgentController {

private final AgentPlatform agentPlatform;
private final AgentInvocation<Domain.ConferenceApplications> invocation;

private final ProcessOptions processOptions = new ProcessOptions()
     .withVerbosity(new Verbosity()
     .withShowPrompts(true)
     .withShowLlmResponses(true)
     .withShowPlanning(true)
     .withDebug(true));


public EmbabelAgentController(AgentPlatform agentPlatform) {
      this.agentPlatform = agentPlatform;

      this.invocation = AgentInvocation
            .builder(agentPlatform)
        .options(processOptions)
        .build(Domain.ConferenceApplications.class);
  }
...

}

First, we autowire the Embabel's AgentPlatform object and use it to create the AgentInvocation object. We additionally pass the ProcessOption object that is configured to show planning, LLM responses, and prompts. We also pass the type of the response - the domain object returned after the agents achieve their goals; in our case, the Domain.ConferenceApplications class:

Now, let's implement the endpoint itself:

@GetMapping(value = "/generic", consumes = "text/plain")
public Domain.ConferenceApplications genericPrompt(@RequestParam String prompt) {
    var inputs = Map.of("request", new UserInput(prompt)); 
    return this.invocation.run(inputs)
         .last(Domain.ConferenceApplications.class);    
}

Here we wrap the prompt into Embabel's UserInput object that we put into the map with all inputs required to run the agent. Then we use the AgentInvocation object to run the agent with the specified input synchronously. We can also invoke runAsync to run the agent asynchronously. Finally, we retrieve the result of the agent invocation with the specified type.

Let's build the application with mvn clean package and run it locally with mvn spring-boot:run. We'll send the prompt number 1 to this endpoint. What we'll see is that Embabel picked the wrong agent, CreateTalksAndApplyForConferencesAgent, to run. Why? As we don't use the Embabel shell, Embabel doesn't analyze the prompt and compute the score to determine what exact agent to run. It currently only looks into the return type of all agents and picks the first one that matches the type defined in the AgentInvocation object. But both our agents have the same return type Domain.ConferenceApplications. Their goals, in the end, are the same: to apply for the conferences. With that, for our 2 agents, there is a 50% probability that Embabel will pick the wrong agent to run.

How can we fix that? For example, we can define the endpoint per agent. Let's do this:

@GetMapping(value = "/applyToConferencesWithExistingTalks", consumes = "text/plain")
public Domain.ConferenceApplications applyToConferencesWithExistingTalks(@RequestParam String prompt) {
    return this.invokeAgent(prompt,
       SearchForTalksAndApplyForConferencesAgent.AGENT_NAME);   
}

@GetMapping(value = "/applyToConferencesWithNewTalks", consumes = "text/plain")
public Domain.ConferenceApplications applyToConferencesWithNewTalks(@RequestParam String prompt) {
    return this.invokeAgent(prompt,
        CreateTalksAndApplyForConferencesAgent.AGENT_NAME); 
}

I additionally wrote 2 helper methods, invokeAgent and getAgentByName, to simplify the agent search by name and its subsequent invocation:

private Domain.ConferenceApplications invokeAgent(String prompt, String agentName) {        
     var inputs = Map.of("request", new UserInput(prompt));
     var agent = this.getAgentByName(agentName);
     var agentProcess = this.agentPlatform.createAgentProcess(agent,
 processOptions, inputs);
     return agentProcess.run().last(Domain.ConferenceApplications.class);
}

private Agent getAgentByName(String agentName) {
     var optionalAgent = this.agentPlatform.agents()
        .stream()
        .filter(a -> a.getName().equals(agentName))
        .findFirst();

     if( optionalAgent.isEmpty()) {
       throw new RuntimeException("agent with the name "+agentName+ " not found");
     }

    return optionalAgent.get();
}

Let's explain what happens in the invokeAgent method. We first invoke the getAgentByName method. It searches in the collection of agents that the agent platform knows for one that has a specified name. All classes annotated with Embabel's Agent annotation will be registered in the agent platform. Then we use the AgentPlatform object to create the AgentProcess and pass the found agent, the already described process options, and inputs to it. Then we run the AgentProcess and return its last output of type Domain.ConferenceApplications.class.

Now we send prompt number 1 (see above) to the applyToConferencesWithExistingTalks endpoint and prompt number 2 to the applyToConferencesWithNewTalks endpoint. We'll see something similar as a result, as we saw in part 2.

For example, here I use httpie to send prompt number 1:
http GET http://localhost:8080/applyToConferencesWithExistingTalks?prompt="Please provide me with the list of conferences including their IDs with Java topic happening in 2026 and 2027 with call for papers open today. Also provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference." Content-Type:text/plain.

Here is the response of the agent:

Having one endpoint per Embabel agent has its pros and cons. The biggest disadvantage is that a service like AgentCore Runtime exposes only one HTTP POST /invocations endpoint to send the prompt to the web application. This means that we'll need to think about how to design, implement, and deploy each Embabel agent on a separate AgentCore Runtime. Each agent belonging to the same web application will have a lot of shared logic with other agents. This is not very convenient. The biggest advantage is that you can scale each Embabel agent individually. AgentCore Runtime will do it for you.

It's worth checking for the upcoming version of Embabel whether there are other options available to invoke the agent.

Conclusion

In this article, we covered how to develop a web application with Embabel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

EventBridge Scheduler DLQ'd My Successful Invokes

Danielle Heberling — Sat, 01 Aug 2026 18:00:00 +0000

A while back I wrote about letting an AI agent do your job searching. That post ended with a quirk I couldn't explain. EventBridge Scheduler would invoke my agent, the agent would run fine, I'd get the email, and the invocation would land in the dead letter queue anyway. Every single time.

I figured I was either doing something wrong or hitting an AgentCore bug. Turns out it was neither.

The Actual Problem

EventBridge Scheduler universal targets (the feature that lets Scheduler call almost any AWS API action directly, no Lambda in between) are synchronous. Scheduler makes the API call and waits for a response before deciding whether the invocation succeeded. The docs describe this as at-least-once delivery, where at least one delivery succeeds with a response from the target. I glossed over that last part.

My schedule was calling the Bedrock AgentCore invokeAgentRuntime action, which blocks until the agent finishes. A search takes 30 to 75 seconds. Scheduler gives up around the 30 second mark and marks the invocation failed. The agent, which has no idea any of this happened, keeps going, finishes the search, and publishes to SNS. I get my email. The "failed" invocation goes to the DLQ. Here's what one of those messages looks like:

Everything downstream worked. The only broken part was Scheduler's opinion of it.

About That 30 Seconds

I went looking for this in the docs and could not find it. The EventBridge Scheduler quotas page covers number of schedules, API request rates, and invocation throughput. Nothing about how long Scheduler waits for a target to respond.

The one timeout you can configure is MaximumEventAgeInSeconds in the retry policy. That's the maximum age of an event across retries, not a per-call limit. Different thing.

So treat ~30 seconds as an observed number, not a contract. In my opinion the safer bet is designing against "Scheduler will not wait long" rather than a specific value.

The Fix

Instead of making the search faster, I made the response faster. The entrypoint now kicks the search off as a background task and returns immediately, so Scheduler gets its response well inside the window and the agent runs for as long as it needs to (up to AgentCore's 8 hour session cap, which my searches are nowhere near).

AgentCore has first-class support for this. You register the background work with add_async_task, the runtime reports HealthyBusy on /ping while it's in flight, and the session stays alive instead of getting reaped. That last part matters because AgentCore terminates sessions after 15 minutes of idle, and without task tracking your background work looks exactly like idle. The async processing docs cover the pattern.

Here's the relevant part of the entrypoint:

# strong refs: asyncio only weakly references tasks
_background_tasks: set[asyncio.Task[Any]] = set()

async def _tracked_job_search(company: str, title: str, location: str) -> None:
    """Run the job search as a tracked async task so ping reports HealthyBusy."""
    task_id = app.add_async_task("job_search")
    try:
        await run_job_search(company, title, location)
    finally:
        app.complete_async_task(task_id)

@app.entrypoint
async def invoke(payload: dict[str, Any] | None = None) -> dict[str, Any]:
    """Main entrypoint for the agent invocation."""
    # ...payload validation...

    if payload.get("sync"):
        return await run_job_search(company, title, location)

    # Respond before EventBridge Scheduler's ~30s call timeout DLQs the invocation
    task = asyncio.create_task(_tracked_job_search(company, title, location))
    _background_tasks.add(task)
    task.add_done_callback(_background_tasks.discard)

    return {
        "status": "accepted",
        "search_criteria": {"company": company, "title": title, "location": location},
    }

Scheduler gets {"status": "accepted"} in well under a second. The search runs in the background. No more DLQ.

A Few Things to Watch For

The Strands call has to be awaited. I was calling agent(prompt) inside an async def entrypoint, which blocks the event loop. Survivable when the entrypoint blocked anyway. But once the runtime needs to answer health checks mid-search, a blocking call starves everything. Switching to await agent.invoke_async(prompt) was required, not a style preference.

Hold a reference to the task. asyncio only weakly references tasks from create_task, so an unreferenced task can get garbage collected mid-run. That's the _background_tasks set above, lifted straight from the pattern in the Python docs. It usually works fine right up until it doesn't.

Keep a synchronous path. Fire and forget is great for the scheduler and terrible for local development. A "sync": true flag in the payload runs the search inline and returns the full result. That's what I use for curl and the AgentCore playground.

One More Thing About Retries

In the original post I disabled retries, because Scheduler retrying a "failed" invocation meant duplicate agent runs and duplicate emails. The CDK target default is 185 retries with exponential backoff over up to 24 hours, so every scheduled search was set up to keep re-running (and re-billing) until I caught it.

With the async path, I turned retries back on (retryAttempts: 0 to 2). Now Scheduler gets an ack right away, so it won’t keep retrying while the search is still running. There’s still a small chance of a duplicate if the ack gets lost, but that’s a trade-off I’m comfortable with here.

There's a tradeoff here. Scheduler now only confirms your agent started. It says nothing about whether the search actually worked. If the search blows up 40 seconds in, Scheduler is happy and your DLQ stays empty. You need logs, metrics, or the notification itself to know the work finished. For me a missing email is a loud signal. If your silence is more ambiguous, plan for that.

If You're Hitting This

If you're invoking AgentCore Runtime from an EventBridge schedule and seeing DLQ messages for runs that clearly worked:

Check how long your agent actually takes. Anything over ~30 seconds is suspect.
Return an ack from your entrypoint instead of the result.
Wrap the real work in add_async_task / complete_async_task so the runtime knows to keep the session alive.
Make sure nothing in that background path blocks the event loop.

Code is in the job-search-agent repo and the change is PR #51 if you want to see the whole diff.

None of this is AgentCore specific. Any long-running universal target invoked from EventBridge Scheduler has the same problem. Agents just happen to be really good at being slow. 😅

If you've hit this with a different long-running target, or if you've found the actual timeout documented somewhere I missed, I'd love to hear about it.

# Blog #9: I Gave My Minecraft Mod a Serverless Brain — And It Decides How Many Allies to Summon Based on How Screwed You Are

Carlos Cortez 🇵🇪 [AWS Hero] — Sat, 01 Aug 2026 16:13:30 +0000

I Gave My Minecraft Mod a Serverless Brain — And It Decides How Many Allies to Summon Based on How Screwed You Are

Cloud Swords Mod — Blog 9

I've been building a Minecraft mod that teaches cloud computing through gameplay. Swords named after AWS services. Energy systems that work like network infrastructure. Villagers that trade cloud certifications.

But version 0.3.8 is where things got weird. I connected the mod to an actual AI agent running in the cloud. Now when you swing the Sword of Lambda, instead of spawning a fixed number of minions — an AI evaluates your combat situation and decides how many allies you need.

The game never freezes. The AI never breaks balance. And the whole thing runs on a serverless backend that costs essentially nothing.

Here's how I built it.

The Items That Make It Work

Tome of API Key

Every cloud connection starts with authentication. In the mod, that's the Tome of API Key — a rare artifact found in dungeon chests (8% drop rate in bastions, end cities, nether fortresses).

When you first pick it up, it generates a unique API key (csk-XXXX format) and shows 100 charges. It glows with enchant shimmer. It's your passport to the cloud.

// On first inventory tick, generate unique key
if (!stack.getNbt().contains("api_key")) {
    String key = "csk-" + UUID.randomUUID().toString().substring(0, 8);
    stack.getNbt().putString("api_key", key);
    stack.getNbt().putInt("charges", 100);
}

The tooltip shows your key preview, remaining charges, and connection status. When charges hit zero, the cloud refuses your invocations — just like hitting a rate limit on a real API.

Cloud API Requester (Block)

The Cloud API Requester is the bridge between Minecraft and AWS. It's a crafted block (Iron + Circuit Board + Redstone + Ender Pearl + Data Core) with two slots:

Your Tome of API Key
The weapon you want to cloud-enable

Drop both in, wait ~8 seconds (the "deployment timer"), and your weapon gets marked with cloud_enabled=true in its NBT data. The tooltip now shows "☁ Cloud Enabled" in cyan.

This is the CI/CD pipeline of the mod. You're deploying your weapon to the cloud.

Cloud-Enabled Weapons

Once cloud-enabled, the Sword of Lambda changes behavior completely. Instead of spawning a hardcoded number of minions, it:

Sends your combat context to the backend (health, nearby mobs, biome, dimension)
Shows "☁ Cloud invocation sent..." in the actionbar
Waits for the AI's decision
Applies it: "☁ Cloud decided: 5 minions (Critical threat — max invocations)"

If you're NOT cloud-enabled, the sword works exactly as before. Backward compatible. No cloud required.

The Architecture

┌─────────────────────────────────────────────────────────────┐
│                     MINECRAFT CLIENT                         │
│                                                             │
│  Player swings Sword of Lambda (cloud-enabled)              │
│       │                                                     │
│       ▼                                                     │
│  CloudCombatHandler builds context:                         │
│  {player, weapon, health, biome, dimension, nearby_mobs}   │
│       │                                                     │
│       ▼                                                     │
│  CloudApiClient.sendEvent(context)  ← async, non-blocking  │
└───────┬─────────────────────────────────────────────────────┘
        │ POST /event
        ▼
┌─────────────────────────────────────────────────────────────┐
│                    BACKEND (CDK Stack)                        │
│                                                             │
│  API Gateway (rate limited: 100/day, 10/sec)                │
│       │                                                     │
│       ▼                                                     │
│  Lambda: IngestEvent                                        │
│  - Receives context                                         │
│  - Calls Strands Agent (or rule-based fallback)             │
│  - Stores decision in DynamoDB (TTL: 60s)                   │
│  - Returns {request_id}                                     │
│                                                             │
│  DynamoDB: CloudSwordsResults                               │
│  - Partition key: request_id                                │
│  - TTL auto-cleanup                                         │
│                                                             │
│  Lambda: PollResult                                         │
│  - GET /result/{request_id}                                 │
│  - Returns decision when ready                              │
└───────┬─────────────────────────────────────────────────────┘
        │ Poll every 2s
        ▼
┌─────────────────────────────────────────────────────────────┐
│  CloudApiClient receives decision                           │
│       │                                                     │
│       ▼                                                     │
│  CloudCombatHandler.applyDecision()                         │
│  - Summon "☁ Cloud Invocation" minions (named, cyan)        │
│  - Or apply buff (regeneration, resistance, strength)       │
│  - Or do nothing ("No action needed")                       │
└─────────────────────────────────────────────────────────────┘

The Async Pattern: Fire-and-Forget + Polling

The game can NEVER freeze waiting for a network response. So I used the same pattern you'd use in any real-time application:

Fire: Send the event, get back a request_id immediately
Forget: Continue gameplay, don't block
Poll: Every 2 seconds (via ServerTickEvents), check if the result is ready
Apply: When ready, queue the callback for the server thread

// CloudApiClient — non-blocking
public void sendEvent(JsonObject context) {
    httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofString())
        .thenAccept(response -> {
            String requestId = parseRequestId(response.body());
            startPolling(requestId);
        });
}

// Polling on server tick (every 40 ticks = 2 seconds)
ServerTickEvents.END_SERVER_TICK.register(server -> {
    // Execute queued callbacks on server thread
    Runnable callback;
    while ((callback = pendingCallbacks.poll()) != null) {
        callback.run();
    }
});

The threading fix was critical. Java's HttpClient runs callbacks on its own async thread — but Minecraft entity spawning MUST happen on the server thread. Without the ConcurrentLinkedQueue bridge, buffs wouldn't apply and entities would crash the game.

The AI Agent: Strands + Ollama Cloud

The brain is a Strands Agent running gpt-oss:20b via Ollama Cloud (free tier — no Bedrock cost for development):

from strands import Agent
from strands.models.ollama import OllamaModel

SYSTEM_PROMPT = """You are a combat advisor for a Minecraft RPG mod.
Given player context, decide ONE action.

Available actions (respond ONLY with valid JSON):
- {"action": "summon", "count": 1-5, "duration": 5-15, "reason": "brief"}
- {"action": "buff", "effect": "regeneration|resistance|strength|speed", "duration": 3-10, "reason": "brief"}
- {"action": "none", "reason": "brief"}

Rules:
- If health < 8 and many mobs: summon max allies or buff resistance
- If health > 15 and few mobs: action none or summon 1
- Scale response to threat level
- ONLY output the JSON object, nothing else"""

model = OllamaModel(
    host="https://api.ollama.com",
    model_id="gpt-oss:20b",
)
agent = Agent(model=model, system_prompt=SYSTEM_PROMPT)

The agent receives context like:

{
  "player": "Carlos",
  "weapon": "sword_of_lambda",
  "health": 6,
  "nearby_mobs": 8,
  "biome": "nether_wastes",
  "dimension": "minecraft:the_nether"
}

And responds:

{"action": "summon", "count": 5, "duration": 10, "reason": "Critical threat"}

The system prompt enforces bounded actions — the AI can't break game balance because it can only choose from a fixed set of actions with capped values.

The CDK Stack

The production backend is a single CDK stack:

class CloudSwordsStack(Stack):
    def __init__(self, scope, id, **kwargs):
        super().__init__(scope, id, **kwargs)

        # DynamoDB for async results
        results_table = dynamodb.Table(self, "DecisionResults",
            partition_key={"name": "request_id", "type": "STRING"},
            time_to_live_attribute="ttl",
            billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
        )

        # Lambda: receive event, decide, store
        ingest_fn = _lambda.Function(self, "IngestEvent",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="handler.lambda_handler",
            code=_lambda.Code.from_asset("lambda/ingest"),
        )

        # Lambda: poll result
        poll_fn = _lambda.Function(self, "PollResult",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="handler.lambda_handler",
            code=_lambda.Code.from_asset("lambda/poll"),
        )

        # API Gateway with usage plan (rate limiting = gameplay mechanic)
        api = apigw.RestApi(self, "CloudSwordsApi")
        plan = api.add_usage_plan("Basic",
            throttle={"rate_limit": 10, "burst_limit": 20},
            quota={"limit": 100, "period": apigw.Period.DAY},
        )

The usage plan isn't just infrastructure — it's a gameplay mechanic. 100 requests per day means you can't spam the cloud. You have to be strategic about when you invoke it. Just like real API budgets.

Local Development: The Mock Server

For development, I run a mock server that integrates the Strands Agent directly:

# mock_server.py — localhost:8777
agent = None
try:
    from combat_advisor import create_agent, decide
    agent = create_agent()
    print("✓ Strands Agent loaded")
except:
    print("⚠ Agent unavailable, using rule-based fallback")

def decide_action(body):
    if agent:
        return decide(agent, body)
    # Fallback: simple rules
    if body["health"] < 8 and body["nearby_mobs"] > 5:
        return {"action": "summon", "count": 5, "duration": 10, "reason": "Critical"}
    return {"action": "summon", "count": 1, "duration": 5, "reason": "Standard"}

The mock server logs every interaction to interactions.json — which becomes the eval dataset for Blog 10.

Cloud Concepts Taught Through Gameplay

Game Mechanic	Cloud Concept
Tome of API Key	API keys + authentication
100 charges	Rate limiting / usage plans
Cloud API Requester (8s timer)	CI/CD deployment pipeline
`cloud_enabled=true` NBT	Feature flags
POST /event → request_id	Async event processing
Poll every 2s	Eventual consistency
AI decides action	Serverless compute (Lambda)
Bounded actions (1-5 minions)	Guardrails / IAM boundaries
CDK stack	Infrastructure as Code
Mock server	Local development environment

What I Learned

Threading in Minecraft is brutal — anything touching entities must be on the server thread. Async HTTP + game logic = careful queue management.
Rate limiting as gameplay works — players naturally strategize around limited cloud invocations. It teaches resource management without a lecture.
AI guardrails through system prompts — bounding the action space (1-5 minions, specific buffs) means the AI literally cannot break the game. The worst it can do is summon 5 minions when 3 would suffice.
Fallback patterns matter — if the cloud is down, the sword still works with local logic. Graceful degradation, taught through a video game.

What's Next

The AI makes decisions. But are they good decisions? In Blog 10, I build an evaluation framework that judges the agent's combat advice — using another AI as the judge. Meta? Absolutely. Useful? Surprisingly yes.

Code:

I'm Carlos Cortez — cloud engineer, gamer, and apparently someone who connects Minecraft to AI agents for fun. This is Breaking the Cloud.

LinkedIn · GitHub · Dev.to

I Built a Cloud Caravan Merchants Progression System as a cloud Marketplace

Carlos Cortez 🇵🇪 [AWS Hero] — Sat, 01 Aug 2026 16:10:01 +0000

Every cloud platform has a marketplace. AWS has the AWS Marketplace, Azure has its own, GCP too. They're places where you find specialized vendors offering services you can't build yourself — and the more you buy, the more they offer you.

We built that in Minecraft: Cloud Caravan Merchants. Three specialized traders that travel with donkeys carrying loot chests, offer 5 base trades each, and evolve into Elite merchants when you complete their missions — unlocking premium trades including Elytra, Tridents, and Totems of Undying.

🎮 The learning angle: The Elite progression system mirrors how cloud marketplace vendors work — start with basic offerings, prove you're a serious customer (complete trades), and unlock enterprise-tier products. The 3 merchant types map to cloud specializations: infrastructure (Packet Runner), platform (Protocol Broker), and application layer (Quantum Courier).

3 Merchants — 3 Cloud Layers

Merchant	Color	Specialty	Cloud Layer
Packet Runner	Cyan	Raw materials & resources	Infrastructure (IaaS)
Protocol Broker	Orange	Equipment & upgrades	Platform (PaaS)
Quantum Courier	Purple	Runes, tomes & magic	Application (SaaS)

Each spawns with a donkey carrying a loot chest with starter items. It's a welcome package — like free tier credits when you sign up for a cloud platform.

The Spawn Egg — Full Caravan Unit

The custom spawn egg doesn't just spawn a merchant. It spawns the entire unit:

Merchant entity (custom type, wandering trader model)
Donkey (tamed, with chest)
Loot in the chest (via loot table)
Leash connecting donkey to merchant

@Override
public ActionResult useOnBlock(ItemUsageContext context) {
    // 1. Spawn merchant
    CloudMerchantEntity merchant = merchantType.create(world);
    merchant.setPosition(pos.getX() + 0.5, pos.getY(), pos.getZ() + 0.5);
    world.spawnEntity(merchant);

    // 2. Spawn donkey with chest
    DonkeyEntity donkey = EntityType.DONKEY.create(world);
    donkey.setTame(true);
    donkey.setHasChest(true);

    // 3. Fill chest via NBT (inventory must be properly sized)
    NbtCompound donkeyNbt = new NbtCompound();
    donkey.writeNbt(donkeyNbt);
    donkeyNbt.putBoolean("ChestedHorse", true);
    NbtList itemsList = new NbtList();
    // ... add loot items to NBT
    donkey.readNbt(donkeyNbt);
    world.spawnEntity(donkey);

    // 4. Leash
    donkey.attachLeash(merchant, true);
}

The Donkey Chest Bug — A War Story

Fun debugging story: donkeys in 1.20.1 have a SimpleInventory field called items. When you call setHasChest(true), it's supposed to expand the inventory from 2 slots (saddle + armor) to 17 slots (+ 15 chest slots). But the expansion happens in onChestedStatusChanged(), which recreates the inventory.

If you try to put items in the chest BEFORE the inventory is properly sized, you get ArrayIndexOutOfBoundsException. If you put them AFTER spawning, the inventory might not be synced to the client yet.

The solution: write the chest contents via NBT. Set ChestedHorse: true and Items: [...] in the NBT compound, then call readNbt(). This forces the entity to reconstruct its inventory with the correct size AND the items already in place.

Elite Progression — The Loyalty Program

Complete all 5 base trades (use them until sold out) and the merchant evolves:

Base (5 trades) → Elite ★ (7 trades) → Elite ★★ (9 trades) → Elite ★★★ (11 trades)

Each promotion:

Adds 2 new trades with premium items
Changes the merchant's name to gold with stars
Triggers a Totem of Undying particle burst
Adds ambient END_ROD particles

private void checkElitePromotion() {
    int requiredTrades = 5 + (eliteLevel * 2);
    if (completedTrades >= requiredTrades && eliteLevel < 3) {
        eliteLevel++;
        applyEliteName(); // "★★ Protocol Broker ★★"
        addEliteTrades(this.getOffers(), eliteLevel);
        // Celebration particles
        world.spawnParticles(ParticleTypes.TOTEM_OF_UNDYING, ...);
    }
}

Elite Trades — Premium Vanilla Items

Cycle	Packet Runner	Protocol Broker	Quantum Courier
★	Quantum Shards, Spyglass	Netherlink Plating, Enchanted Golden Apple	Deployed Runes, Ender Eyes
★★	Cloud Edge Pearls, Echo Shards	Cloudsteel Plating, Trident	Grimoire of Architect, XP Bottles ×16
★★★	Nether Star, Totem of Undying	Netherite ×2, Elytra	Quantum Runes ×16, End Crystals

The endgame trades offer items that are normally very hard to obtain (Elytra, Trident) in exchange for Cloud Credits and rare materials. It's the cloud marketplace premium tier.

Natural Spawning — The Caravan Arrives

Every ~1 hour of real time, there's a 15% chance a caravan spawns 48-96 blocks from a random player:

private static void spawnCaravan(ServerWorld world, BlockPos pos, Random random) {
    int caravanSize = 1 + random.nextInt(3); // 1, 2, or 3 merchants
    // Shuffle types to avoid duplicates
    List<MerchantType> available = new ArrayList<>(List.of(MerchantType.values()));
    Collections.shuffle(available);

    for (int i = 0; i < caravanSize; i++) {
        // Spawn merchant + donkey at offset positions
        spawnUnit(world, pos, available.get(i), i * 2.5);
    }
}

A full caravan (3 merchants) gives you access to all specializations at once. It's like a cloud conference — all the vendors in one place.

Never Despawn — Persistent Infrastructure

Unlike vanilla wandering traders that disappear after 40 minutes, Cloud Merchants are permanent:

this.setPersistent();
this.setDespawnDelay(Integer.MAX_VALUE);

Once you find (or spawn) a merchant, they stay forever. They're your infrastructure — you invested in them, they stick around.

Custom Textures — Visual Identity

Each merchant uses a recolored wandering trader texture generated with Python PIL:

def recolor(data, hue_shift):
    """Shift hue of non-skin pixels"""
    for pixel in data:
        if not is_skin_tone(pixel):
            h, s, v = rgb_to_hsv(pixel)
            h = (h + hue_shift) % 1.0
            pixel = hsv_to_rgb(h, s, v)

Packet Runner: cyan shift (networking)
Protocol Broker: orange shift (fire/security)
Quantum Courier: purple shift (quantum/magic)

Rune Table Recipes — Mod Interop

We also added recipes for the Rune Crafting Altar (from the Runes mod):

Rune	Altar Input	Output	Savings
Packet	Cloud Shard + Redstone	×2	-1 Lapis, +1 output
Protocol	Packet Rune + Blazing Shard	×2	-1 Gold, +1 output
Quantum	Protocol Rune + Quantum Shard	×2	-1 Ender Pearl, +1 output

This is mod interop — using another mod's crafting system to provide alternative recipes. Like using a third-party service that's cheaper than building your own.

Lessons Learned

Challenge	Solution
Donkey chest inventory size	Write items via NBT, not direct slot access
Entity renderer missing = crash	Register renderer in ClientModInitializer
Wandering trader despawn timer	Override with MAX_VALUE despawnDelay
Spawn egg model missing	`template_spawn_egg` parent model
Trade completion tracking	NBT persistence (CompletedTrades, EliteLevel)

Series Wrap-Up

Over 8 posts, we went from "what if AWS services were Minecraft swords" to a full tech-magic mod with:

7 cloud-themed swords with unique abilities
28 spells across 9 spellbooks
4 armor sets with set bonuses
Multiblock machines with BFS energy networks
7 villager professions with a Cloud Credits economy
3 Caravan Merchants with Elite progression
Full compatibility with 650+ mod modpacks

The mod teaches cloud computing by making you experience the concepts. You don't read about Auto Scaling — you feel your damage stack. You don't study fault tolerance — you dodge attacks with Quantum Armor. You don't memorize service tiers — you progress through them.

That's the power of game-based learning. And we built it all with AI as a co-pilot.

The complete mod is on GitHub. Clone it, build it, play it.

Connect with me:

I'm Carlos Cortez, this is Breaking the Cloud, and this series proved that the best way to learn cloud is to play it. See you in the next adventure! 🎮☁️⚔️

Designing an Armor Progression That Mirrors Cloud Certification Levels

Carlos Cortez 🇵🇪 [AWS Hero] — Sat, 01 Aug 2026 16:09:22 +0000

In cloud computing, you level up through certifications: Practitioner → Associate → Professional → Specialty. Each level unlocks more capabilities and proves deeper expertise. We built the same thing as armor sets in Minecraft.

4 armor sets, each with a set bonus, a dodge mechanic, and a clear progression path that mirrors going from cloud beginner to architect.

🎮 The learning angle: Dataflow Armor gives Speed (network optimization). Netherlink gives Fire Resistance (firewall protection). Quantum gives damage resistance (fault tolerance). Cloudsteel gives flight (cloud-native freedom). The set bonuses ARE the cloud concepts.

4 Armor Sets — 4 Cloud Maturity Levels

Set	Tier	Armor	Toughness	Cloud Level
Dataflow	Post-Iron	18	1.5	Practitioner — basic networking
Netherlink	Post-Diamond	20	2.5	Associate — security focus
Quantum	Post-Netherite	24	3.5	Professional — fault tolerance
Cloudsteel	Endgame	24	4.0	Specialty — full mastery

Each set has a unique visual identity:

Dataflow: Cyan/teal with circuit line patterns + bright nodes
Netherlink: Dark red/orange with ember glow edges
Quantum: Purple with glitch pixels (digital distortion)
Cloudsteel: White/silver with cloud wisps + luminous edges

Set Bonuses — Cloud Capabilities

When you wear all 4 pieces of a set, you get passive effects:

Set	Effects	Dodge %	Cloud Concept
Dataflow	Speed I + Haste I	10%	Network optimization — faster throughput
Netherlink	Fire Resistance + Strength I (Nether)	13%	Firewall — immune to fire attacks
Quantum	Resistance I	15%	Fault tolerance — reduced impact
Cloudsteel	Slow Falling + Jump Boost II	20%	Cloud-native — freedom of movement

The Dodge System

Every set has a chance to completely nullify incoming damage. When triggered:

Portal particles burst around the player
Enderman teleport sound plays
Damage is set to zero

public class QuantumDodgeHandler {
    public static void register() {
        // ALLOW_DAMAGE event — return false to cancel damage
        DamageEvents.ALLOW.register((entity, source, amount) -> {
            if (!(entity instanceof PlayerEntity player)) return true;

            float dodgeChance = getSetDodgeChance(player);
            if (dodgeChance > 0 && player.getRandom().nextFloat() < dodgeChance) {
                // Dodge! Cancel damage
                spawnDodgeParticles(player);
                return false;
            }
            return true;
        });
    }
}

Higher-tier armor = higher dodge chance. It's probabilistic fault tolerance — like how multi-AZ deployments reduce (but don't eliminate) the chance of downtime.

Crafting Progression — The Certification Path

You don't jump straight to Cloudsteel. The progression uses the Smithing Table:

Iron Armor + Dataflow Plating + Cloud Shard → Dataflow Armor
Diamond Armor + Netherlink Plating + Blazing Shard → Netherlink Armor
Netherite Armor + Quantum Plating + Quantum Shard → Quantum Armor
Quantum Armor + Cloudsteel Plating + Cloud Edge Pearl → Cloudsteel Armor

Each tier requires the previous tier's armor as base. You can't skip levels — just like certifications build on each other.

Platings (4 ingots each)

III
I_I  →  1 Plating
___

Simple shaped recipe. 4 platings per set (helmet, chest, legs, boots) × 4 ingots = 16 ingots per full set.

Accessories Integration — Optional Dependencies

The mod works with or without the Accessories mod installed. When present, the Cloud Engineer Lens can be equipped in the face trinket slot instead of taking the helmet slot.

public class AccessoriesCompat {
    public static boolean isEquipped(PlayerEntity player, Item item) {
        // Runtime check — only call if mod is loaded
        var component = AccessoriesCapability.get(player);
        if (component == null) return false;
        return component.isEquipped(item);
    }
}

Detection priority: Accessories slot → Trinkets slot → Helmet → Offhand → Mainhand.

This is the optional dependency pattern — your mod works standalone, but integrates with other mods when available. Like how a cloud service works alone but integrates with others via APIs.

Spell Book Slot — Cross-System Integration

All 18 tomes (9 base + 9 empowered) implement a SpellBookItem interface and register in the trinkets:charm/spell_book tag. This means they can be equipped in the Spell Book accessory slot from other magic mods.

Cross-mod compatibility isn't just nice to have — it's essential for modpack inclusion. Same as cloud services: if your service doesn't integrate with the ecosystem, nobody uses it.

Nether Star Duplication — Endgame Recipe

4 Cloud Edge Pearls + 4 Data-Infused Quantum Shards + 1 Nether Star → 2 Nether Stars

An endgame recipe that avoids Wither farming. Expensive inputs, but saves time. It's like Reserved Instances — pay more upfront in materials, save time in the long run.

Lessons Learned

Challenge	Solution
Set bonus detection every tick is expensive	Check every 20 ticks (1 second), not every tick
Dodge feels unfair in PvP	Balance with low percentages (10-20%)
Accessories as optional dep	`compileOnly` + runtime class check
Armor textures	Recolor vanilla iron/netherite with unique effects
Progression too fast	Require previous tier as base (sequential)

What's Next

We have swords, spells, machines, and armor. But what about NPCs that sell you things? In the final post of this series, we'll build Cloud Caravan Merchants — wandering traders with donkeys, loot chests, and an Elite progression system. Because every cloud platform needs a marketplace.

Connect with me:

I'm Carlos Cortez, this is Breaking the Cloud, and today we armored up with cloud certifications. See you in the next one! 🛡️

I Built a Magic System Where Every Spell is a Cloud Operation

Carlos Cortez 🇵🇪 [AWS Hero] — Sat, 01 Aug 2026 15:13:45 +0000

What does casting a spell have in common with calling an API? More than you'd think. Both consume resources (mana/runes vs. API credits), both have cooldowns (cast time vs. rate limits), both have tiers (spell levels vs. service tiers), and both produce effects at a distance.

So we built Cloud Spellbooks — 9 tomes, 28 spells, 5 rune types, and a full upgrade path. Every spell is named after a cloud operation, and every mechanic maps to how cloud services actually work.

🎮 The learning angle: When you cast "Auto Scale" and it does AoE damage that scales with targets, you're experiencing what Auto Scaling does. When "Region Failover" fully heals you with a 5-minute cooldown, you understand why failover is a last resort with a long recovery time. The game mechanics ARE the cloud concepts.

The Architecture: Spell Engine Integration

We didn't build a spell system from scratch. We integrated with Spell Engine — a Fabric mod that provides data-driven spell registration, the Binding Table mechanic, and compatibility with other magic mods.

Our spells are defined as JSON:

{
  "school": "arcane",
  "range": 6.0,
  "cast": { "duration": 0, "channel_ticks": 0 },
  "cost": { "item_id": "cloudswords:packet_rune", "count": 1 },
  "cooldown": 3.0,
  "impact": [
    { "action": { "type": "spell_engine:damage", "damage": { "spell_power_coefficient": 0.5 } } },
    { "action": { "type": "spell_engine:status_effect", "status_effect": { "effect_id": "minecraft:glowing", "duration": 100 } } }
  ]
}

This is Infrastructure as Code for magic. Define your spell's behavior declaratively, and the engine handles execution.

9 Tomes, 3 Tiers — Like Service Tiers

Tier	Tomes	Rune Cost	Spell Power	Cloud Analogy
T1	Hacker's Notebook, Codex of Uptime	Packet Rune	+2	Free tier — basic operations
T2	Pipeline, SRE Codex, Cloud Captain, DevOps	Protocol Rune	+4	Standard tier — production workloads
T3	Serverless Bible, Grimoire of Architect, Zero Downtime	Quantum Rune	+7	Enterprise tier — mission-critical

Each tier requires a more expensive rune to cast. Just like cloud services — basic operations are cheap, enterprise features cost more.

28 Spells — Cloud Operations as Game Mechanics

Here are some highlights showing how cloud concepts become spells:

Spell	Tome	Effect	Cloud Concept
Ping	Hacker's Notebook	Damage + Glowing	Network ping — reveals target
DDoS Flood	Hacker's Notebook	Triple rapid hit	Distributed denial — overwhelming force
Firewall Burst	Codex of Uptime	Resistance + Fire Res	Firewall — blocks incoming
Auto Scale	Tome of Pipeline	AoE damage radius 6	Auto Scaling — handles multiple targets
Incident Response	SRE Codex	Regen III + Absorption	Incident response — emergency healing
Load Balance	Cloud Captain	Strength + Speed to allies	Load balancer — distributes power
CI/CD Sprint	DevOps Manifesto	Speed III + Haste II	CI/CD — everything faster
Cloud Nuke	Serverless Bible	Massive AoE	Nuclear option — destroy everything
Quantum Tunnel	Grimoire of Architect	Teleport 20 blocks	Quantum networking — instant transfer
Region Failover	Zero Downtime	Full heal + Res IV	Failover — complete recovery, long CD

The cooldowns tell a story too: "Ping" is 3 seconds (quick diagnostic), "Region Failover" is 5 minutes (disaster recovery is expensive and slow).

Rune System — API Credits

Runes are consumed when you cast spells. They're your API credits:

Rune	Tier	Crafting	Analogy
Packet Rune	T1	Cloud Shard + Redstone + Lapis	Basic API call
Protocol Rune	T2	Packet Rune + Blazing Shard + Gold	Standard API call
Quantum Rune	T3	Protocol Rune + Quantum Shard + Ender Pearl	Enterprise API call

Higher-tier spells consume more expensive runes. You can't spam "Cloud Nuke" because Quantum Runes are expensive to craft — just like you can't spam expensive API calls without budget consequences.

Upgrade Paths — Service Evolution

Hacker's Notebook → Tome of the Pipeline → The Serverless Bible
Codex of Uptime → The SRE Codex → Grimoire of Zero Downtime
                → Tome of the Cloud Captain → Grimoire of the Architect
                → DevOps Manifesto (independent)

Upgrades use the Smithing Table: Tome + Compiled/Deployed Rune + vanilla item. The path from T1 to T3 mirrors how cloud services evolve — you start with basic tools and upgrade to enterprise solutions.

Empowered Tomes — Premium Tier

Each tome has an "Empowered" variant (+2 extra spell power) crafted with rare materials:

T1 Empowered: + Amethyst Shard
T2 Empowered: + Blaze Rod
T3 Empowered: + Nether Star

It's like going from standard to reserved instances — same service, better performance, higher upfront cost.

Right-Click Casting — The UX

public abstract class SpellTome extends Item {
    @Override
    public TypedActionResult<ItemStack> use(World world, PlayerEntity player, Hand hand) {
        // 1. Check rune in inventory
        if (!hasRune(player, getRequiredRune())) {
            player.sendMessage(Text.literal("§c✗ Missing " + getRuneName()));
            return TypedActionResult.fail(stack);
        }
        // 2. Check cooldown
        if (isOnCooldown(player)) return TypedActionResult.fail(stack);
        // 3. Consume rune
        consumeRune(player, getRequiredRune());
        // 4. Cast spell
        castSpell(world, player);
        // 5. Apply cooldown
        applyCooldown(player);
        return TypedActionResult.success(stack);
    }
}

Simple flow: verify resources → consume → execute → cooldown. It's the same pattern as any API call: authenticate → charge → process → rate limit.

Lessons Learned

Challenge	Solution
28 spells is a lot of JSON	Data-driven approach — one template, many configs
Balancing cooldowns	Map to real-world operation frequency
Rune economy	Make T1 cheap, T3 expensive — natural gating
Spell Engine compatibility	Use `is_proxy: true` for Binding Table
Visual identity per tome	Isometric textures with unique colors + symbols

What's Next

Magic is great, but it needs infrastructure to support it. In the next post, we'll build a multiblock Data Melter with BFS energy transfer through cables — because even in a fantasy world, you need power grids.

Connect with me:

I'm Carlos Cortez, this is Breaking the Cloud, and today we cast cloud operations as spells. See you in the next one! ✨

Monthly Amazon Location Service Updates - 2026.07

Yasunori Kirimoto — Sat, 01 Aug 2026 02:10:06 +0000

Monthly Amazon Location Service Updates - 2026.07

This is a summary of the July updates for Amazon Location Service.

2026.07 Updates

Amazon Location Service enhances Places APIs with new address and search options
This release introduces the AddressNamesMode parameter to control how address components are returned, the AddressTranslations parameter to return place name translations in 50+ languages, and the TravelMode parameter to optimize search results for users in motion. It also adds the DriveThrough attribute indicating whether a place offers drive-through service, and a Parsing.AdditionalInfo response field in the Geocode API that details how the input address was interpreted.

Other Info

Amazon Location Service Demo
Official Amazon Location Service demo.

Amazon Location Service Developer Guide
Official Amazon Location Service Documentation.

AWS Geospatial
Official AWS Geospatial samples.

maplibregljs-amazon-location-service-starter
Build environment to get started with Amazon Location Service.

dev.to
Articles on Amazon Location Service.

tags - Amazon Location Service
tags - Try
Notes on Amazon Location Service. (Japanese)

Yasunori Kirimoto for AWS Heroes

Jul 11

Monthly Amazon Location Service Updates - 2026.06

#amazonlocationservice #amazonlocationserviceupdates

1 min read