DEV Community: AWS Heroes

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 1 Introduction to the series and services

Vadym Kazulkin — Mon, 10 Aug 2026 13:00:00 +0000

Introduction to the series

Throughout this article series, we'll develop a simple application to show how to build multi-step applications and orchestrate workflows using AWS Lambda durable functions. For it, we'll explore the AWS Lambda Durable Execution SDK for Java. This simple application provides the functionality to extract the content for the provided user. This content involves searching for YouTube videos of the authors and their upcoming talks. We'll store the result on Amazon S3 Files in JSON format. As this operation may potentially take time, we'll additionally provide access to the author content through the Amazon API Gateway:

We'll start with the simple application to search for YouTube videos and upcoming talks of the author synchronously. First, we'll use durable steps and then later add waits and callbacks. We won't implement any persistent layer and store the static content in memory.
We'll then improve our application by adding asynchronous communication where it makes sense. For example, searching for YouTube videos and upcoming talks can be done in parallel.
We'll then further improve our application by converting the AWS durable function to a pure orchestrator. This means we'll move the business logic for searching for YouTube videos and upcoming talks into separate Lambda functions. With that, our Lambda durable function will invoke those separate Lambda functions in parallel and combine the results
Finally, we'll further extend our application to use Amazon Bedrock AgentCore Web Search Tool to search for the author content on the web. With that, our Lambda durable function becomes an orchestrator of the agentic AI workflows. As we'll use Java in this series, we'll explore how to run this business logic responsible for the Amazon Bedrock AgentCore Web Search Tool. As this tool is exposed as an MCP tool by the Amazon Bedrock AgentCore Gateway, we need a framework capable of providing AI functionality and supporting MCP. That's why we'll use Spring AI. As 2 operational environments for our agentic AI workflows, we'll explore an AWS Lambda function and Amazon Bedrock AgentCore Runtime.

You can find the code examples in my aws-s3-files-lambda-durable-functions-java-sdk repository. We'll go through all the examples in more detail in the subsequent articles.

Introduction to the services used throughout the series

Throughout this series, we'll use the following AWS services:

AWS Lambda durable functions. With Lambda durable functions, you can build resilient multi-step applications and AI workflows. These can execute for up to one year while maintaining reliable progress despite interruptions. When a durable function runs, this complete lifecycle is called a durable execution, which uses checkpoints to track progress and automatically recover from failures through replay, re-executing from the beginning while skipping completed work. Within each function, you use durable operations as fundamental building blocks. Steps execute business logic with built-in retries and progress tracking. Waits suspend execution without incurring compute charges, making them ideal for long-running processes. These processes include human-in-the-loop workflows or polling external dependencies.
Amazon S3 Files. S3 Files is a shared file system that connects any AWS compute resource directly with your data in Amazon S3. It provides fast, direct access to all of your S3 data as files with full file system semantics and low-latency performance, without your data ever leaving S3. Every file-based application, agent, and team can access and work with your S3 data as a file system using the tools they already depend on. Built using Amazon EFS, S3 Files gives you the performance and simplicity of a file system with the scalability, durability, and cost-effectiveness of S3. You can read, write, and organize data using file and directory operations. At the same time, S3 Files manages the synchronization of changes between your bucket and file system.
We'll also introduce in the later parts of this series the following Amazon Bedrock AgentCore services or features: Gateway, Web Search Tool, and Runtime.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Building AI Agents with Embabel, Spring AI and Amazon Bedrock AgentCore - Part 3 Develop Agents as web application with Embabel

Vadym Kazulkin — Mon, 03 Aug 2026 13:00:00 +0000

Introduction

In part 2, we developed our first AI agents with the Embabel shell. We learned many concepts of this framework, such as agents, actions, and goals. In this article, we'll slightly adjust our application to convert it into a web application.

Implement AI agents as a web application with Embabel

You can look at my GitHub embabel-1.x-conference-app-agent-local repository for the example. We'll completely reuse the implementation of both agents, the CreateTalksAndApplyForConferencesAgent and the SearchForTalksAndApplyForConferencesAgent that we implemented in part 2.

Those agents are capable of responding to the following prompts:

"Please provide me with the list of conferences, including their IDs, with Java topics happening in 2027, with the call for papers open today. Also, provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference." SearchForTalksAndApplyForConferencesAgent is responsible for responding to this prompt.
"Please create a talk with a cool title (max 60 characters long) and description (max 300 characters long) about using Spring AI on the Amazon Bedrock AgentCore service. Then provide me with the list of conferences, including their IDs, with Java topics happening in 2026 and 2027, with the call for papers open today. Finally, for each conference, apply individually for it with the talk just created." CreateTalksAndApplyForConferencesAgent is responsible for responding to this prompt.

First, let's check that we declared the following dependencies in pom.xml:

 <dependency>
    <groupId>com.embabel.agent</groupId>
    <artifactId>embabel-agent-starter-bedrock</artifactId>
    <version>${embabel-agent.version}</version>
 </dependency>
 <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

Also, make sure not to declare the embabel-agent-starter-shell dependency, as we don't use Embabel shell in this article.

All we need to do now is to additionally implement the EmbabelAgentController controller.

Let's first implement a generic endpoint that will accept both prompts. Let's first autowire and create some objects coming from Embabel that we'll use for this:

@RestController
public class EmbabelAgentController {

private final AgentPlatform agentPlatform;
private final AgentInvocation<Domain.ConferenceApplications> invocation;

private final ProcessOptions processOptions = new ProcessOptions()
     .withVerbosity(new Verbosity()
     .withShowPrompts(true)
     .withShowLlmResponses(true)
     .withShowPlanning(true)
     .withDebug(true));


public EmbabelAgentController(AgentPlatform agentPlatform) {
      this.agentPlatform = agentPlatform;

      this.invocation = AgentInvocation
            .builder(agentPlatform)
        .options(processOptions)
        .build(Domain.ConferenceApplications.class);
  }
...

}

First, we autowire the Embabel's AgentPlatform object and use it to create the AgentInvocation object. We additionally pass the ProcessOption object that is configured to show planning, LLM responses, and prompts. We also pass the type of the response - the domain object returned after the agents achieve their goals; in our case, the Domain.ConferenceApplications class:

Now, let's implement the endpoint itself:

@GetMapping(value = "/generic", consumes = "text/plain")
public Domain.ConferenceApplications genericPrompt(@RequestParam String prompt) {
    var inputs = Map.of("request", new UserInput(prompt)); 
    return this.invocation.run(inputs)
         .last(Domain.ConferenceApplications.class);    
}

Here we wrap the prompt into Embabel's UserInput object that we put into the map with all inputs required to run the agent. Then we use the AgentInvocation object to run the agent with the specified input synchronously. We can also invoke runAsync to run the agent asynchronously. Finally, we retrieve the result of the agent invocation with the specified type.

Let's build the application with mvn clean package and run it locally with mvn spring-boot:run. We'll send the prompt number 1 to this endpoint. What we'll see is that Embabel picked the wrong agent, CreateTalksAndApplyForConferencesAgent, to run. Why? As we don't use the Embabel shell, Embabel doesn't analyze the prompt and compute the score to determine what exact agent to run. It currently only looks into the return type of all agents and picks the first one that matches the type defined in the AgentInvocation object. But both our agents have the same return type Domain.ConferenceApplications. Their goals, in the end, are the same: to apply for the conferences. With that, for our 2 agents, there is a 50% probability that Embabel will pick the wrong agent to run.

How can we fix that? For example, we can define the endpoint per agent. Let's do this:

@GetMapping(value = "/applyToConferencesWithExistingTalks", consumes = "text/plain")
public Domain.ConferenceApplications applyToConferencesWithExistingTalks(@RequestParam String prompt) {
    return this.invokeAgent(prompt,
       SearchForTalksAndApplyForConferencesAgent.AGENT_NAME);   
}

@GetMapping(value = "/applyToConferencesWithNewTalks", consumes = "text/plain")
public Domain.ConferenceApplications applyToConferencesWithNewTalks(@RequestParam String prompt) {
    return this.invokeAgent(prompt,
        CreateTalksAndApplyForConferencesAgent.AGENT_NAME); 
}

I additionally wrote 2 helper methods, invokeAgent and getAgentByName, to simplify the agent search by name and its subsequent invocation:

private Domain.ConferenceApplications invokeAgent(String prompt, String agentName) {        
     var inputs = Map.of("request", new UserInput(prompt));
     var agent = this.getAgentByName(agentName);
     var agentProcess = this.agentPlatform.createAgentProcess(agent,
 processOptions, inputs);
     return agentProcess.run().last(Domain.ConferenceApplications.class);
}

private Agent getAgentByName(String agentName) {
     var optionalAgent = this.agentPlatform.agents()
        .stream()
        .filter(a -> a.getName().equals(agentName))
        .findFirst();

     if( optionalAgent.isEmpty()) {
       throw new RuntimeException("agent with the name "+agentName+ " not found");
     }

    return optionalAgent.get();
}

Let's explain what happens in the invokeAgent method. We first invoke the getAgentByName method. It searches in the collection of agents that the agent platform knows for one that has a specified name. All classes annotated with Embabel's Agent annotation will be registered in the agent platform. Then we use the AgentPlatform object to create the AgentProcess and pass the found agent, the already described process options, and inputs to it. Then we run the AgentProcess and return its last output of type Domain.ConferenceApplications.class.

Now we send prompt number 1 (see above) to the applyToConferencesWithExistingTalks endpoint and prompt number 2 to the applyToConferencesWithNewTalks endpoint. We'll see something similar as a result, as we saw in part 2.

For example, here I use httpie to send prompt number 1:
http GET http://localhost:8080/applyToConferencesWithExistingTalks?prompt="Please provide me with the list of conferences including their IDs with Java topic happening in 2026 and 2027 with call for papers open today. Also provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference." Content-Type:text/plain.

Here is the response of the agent:

Having one endpoint per Embabel agent has its pros and cons. The biggest disadvantage is that a service like AgentCore Runtime exposes only one HTTP POST /invocations endpoint to send the prompt to the web application. This means that we'll need to think about how to design, implement, and deploy each Embabel agent on a separate AgentCore Runtime. Each agent belonging to the same web application will have a lot of shared logic with other agents. This is not very convenient. The biggest advantage is that you can scale each Embabel agent individually. AgentCore Runtime will do it for you.

It's worth checking for the upcoming version of Embabel whether there are other options available to invoke the agent.

Conclusion

In this article, we covered how to develop a web application with Embabel.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

EventBridge Scheduler DLQ'd My Successful Invokes

Danielle Heberling — Sat, 01 Aug 2026 18:00:00 +0000

A while back I wrote about letting an AI agent do your job searching. That post ended with a quirk I couldn't explain. EventBridge Scheduler would invoke my agent, the agent would run fine, I'd get the email, and the invocation would land in the dead letter queue anyway. Every single time.

I figured I was either doing something wrong or hitting an AgentCore bug. Turns out it was neither.

The Actual Problem

EventBridge Scheduler universal targets (the feature that lets Scheduler call almost any AWS API action directly, no Lambda in between) are synchronous. Scheduler makes the API call and waits for a response before deciding whether the invocation succeeded. The docs describe this as at-least-once delivery, where at least one delivery succeeds with a response from the target. I glossed over that last part.

My schedule was calling the Bedrock AgentCore invokeAgentRuntime action, which blocks until the agent finishes. A search takes 30 to 75 seconds. Scheduler gives up around the 30 second mark and marks the invocation failed. The agent, which has no idea any of this happened, keeps going, finishes the search, and publishes to SNS. I get my email. The "failed" invocation goes to the DLQ. Here's what one of those messages looks like:

Everything downstream worked. The only broken part was Scheduler's opinion of it.

About That 30 Seconds

I went looking for this in the docs and could not find it. The EventBridge Scheduler quotas page covers number of schedules, API request rates, and invocation throughput. Nothing about how long Scheduler waits for a target to respond.

The one timeout you can configure is MaximumEventAgeInSeconds in the retry policy. That's the maximum age of an event across retries, not a per-call limit. Different thing.

So treat ~30 seconds as an observed number, not a contract. In my opinion the safer bet is designing against "Scheduler will not wait long" rather than a specific value.

The Fix

Instead of making the search faster, I made the response faster. The entrypoint now kicks the search off as a background task and returns immediately, so Scheduler gets its response well inside the window and the agent runs for as long as it needs to (up to AgentCore's 8 hour session cap, which my searches are nowhere near).

AgentCore has first-class support for this. You register the background work with add_async_task, the runtime reports HealthyBusy on /ping while it's in flight, and the session stays alive instead of getting reaped. That last part matters because AgentCore terminates sessions after 15 minutes of idle, and without task tracking your background work looks exactly like idle. The async processing docs cover the pattern.

Here's the relevant part of the entrypoint:

# strong refs: asyncio only weakly references tasks
_background_tasks: set[asyncio.Task[Any]] = set()

async def _tracked_job_search(company: str, title: str, location: str) -> None:
    """Run the job search as a tracked async task so ping reports HealthyBusy."""
    task_id = app.add_async_task("job_search")
    try:
        await run_job_search(company, title, location)
    finally:
        app.complete_async_task(task_id)

@app.entrypoint
async def invoke(payload: dict[str, Any] | None = None) -> dict[str, Any]:
    """Main entrypoint for the agent invocation."""
    # ...payload validation...

    if payload.get("sync"):
        return await run_job_search(company, title, location)

    # Respond before EventBridge Scheduler's ~30s call timeout DLQs the invocation
    task = asyncio.create_task(_tracked_job_search(company, title, location))
    _background_tasks.add(task)
    task.add_done_callback(_background_tasks.discard)

    return {
        "status": "accepted",
        "search_criteria": {"company": company, "title": title, "location": location},
    }

Scheduler gets {"status": "accepted"} in well under a second. The search runs in the background. No more DLQ.

A Few Things to Watch For

The Strands call has to be awaited. I was calling agent(prompt) inside an async def entrypoint, which blocks the event loop. Survivable when the entrypoint blocked anyway. But once the runtime needs to answer health checks mid-search, a blocking call starves everything. Switching to await agent.invoke_async(prompt) was required, not a style preference.

Hold a reference to the task. asyncio only weakly references tasks from create_task, so an unreferenced task can get garbage collected mid-run. That's the _background_tasks set above, lifted straight from the pattern in the Python docs. It usually works fine right up until it doesn't.

Keep a synchronous path. Fire and forget is great for the scheduler and terrible for local development. A "sync": true flag in the payload runs the search inline and returns the full result. That's what I use for curl and the AgentCore playground.

One More Thing About Retries

In the original post I disabled retries, because Scheduler retrying a "failed" invocation meant duplicate agent runs and duplicate emails. The CDK target default is 185 retries with exponential backoff over up to 24 hours, so every scheduled search was set up to keep re-running (and re-billing) until I caught it.

With the async path, I turned retries back on (retryAttempts: 0 to 2). Now Scheduler gets an ack right away, so it won’t keep retrying while the search is still running. There’s still a small chance of a duplicate if the ack gets lost, but that’s a trade-off I’m comfortable with here.

There's a tradeoff here. Scheduler now only confirms your agent started. It says nothing about whether the search actually worked. If the search blows up 40 seconds in, Scheduler is happy and your DLQ stays empty. You need logs, metrics, or the notification itself to know the work finished. For me a missing email is a loud signal. If your silence is more ambiguous, plan for that.

If You're Hitting This

If you're invoking AgentCore Runtime from an EventBridge schedule and seeing DLQ messages for runs that clearly worked:

Check how long your agent actually takes. Anything over ~30 seconds is suspect.
Return an ack from your entrypoint instead of the result.
Wrap the real work in add_async_task / complete_async_task so the runtime knows to keep the session alive.
Make sure nothing in that background path blocks the event loop.

Code is in the job-search-agent repo and the change is PR #51 if you want to see the whole diff.

None of this is AgentCore specific. Any long-running universal target invoked from EventBridge Scheduler has the same problem. Agents just happen to be really good at being slow. 😅

If you've hit this with a different long-running target, or if you've found the actual timeout documented somewhere I missed, I'd love to hear about it.

# Blog #9: I Gave My Minecraft Mod a Serverless Brain — And It Decides How Many Allies to Summon Based on How Screwed You Are

Carlos Cortez 🇵🇪 [AWS Hero] — Sat, 01 Aug 2026 16:13:30 +0000

I Gave My Minecraft Mod a Serverless Brain — And It Decides How Many Allies to Summon Based on How Screwed You Are

Cloud Swords Mod — Blog 9

I've been building a Minecraft mod that teaches cloud computing through gameplay. Swords named after AWS services. Energy systems that work like network infrastructure. Villagers that trade cloud certifications.

But version 0.3.8 is where things got weird. I connected the mod to an actual AI agent running in the cloud. Now when you swing the Sword of Lambda, instead of spawning a fixed number of minions — an AI evaluates your combat situation and decides how many allies you need.

The game never freezes. The AI never breaks balance. And the whole thing runs on a serverless backend that costs essentially nothing.

Here's how I built it.

The Items That Make It Work

Tome of API Key

Every cloud connection starts with authentication. In the mod, that's the Tome of API Key — a rare artifact found in dungeon chests (8% drop rate in bastions, end cities, nether fortresses).

When you first pick it up, it generates a unique API key (csk-XXXX format) and shows 100 charges. It glows with enchant shimmer. It's your passport to the cloud.

// On first inventory tick, generate unique key
if (!stack.getNbt().contains("api_key")) {
    String key = "csk-" + UUID.randomUUID().toString().substring(0, 8);
    stack.getNbt().putString("api_key", key);
    stack.getNbt().putInt("charges", 100);
}

The tooltip shows your key preview, remaining charges, and connection status. When charges hit zero, the cloud refuses your invocations — just like hitting a rate limit on a real API.

Cloud API Requester (Block)

The Cloud API Requester is the bridge between Minecraft and AWS. It's a crafted block (Iron + Circuit Board + Redstone + Ender Pearl + Data Core) with two slots:

Your Tome of API Key
The weapon you want to cloud-enable

Drop both in, wait ~8 seconds (the "deployment timer"), and your weapon gets marked with cloud_enabled=true in its NBT data. The tooltip now shows "☁ Cloud Enabled" in cyan.

This is the CI/CD pipeline of the mod. You're deploying your weapon to the cloud.

Cloud-Enabled Weapons

Once cloud-enabled, the Sword of Lambda changes behavior completely. Instead of spawning a hardcoded number of minions, it:

Sends your combat context to the backend (health, nearby mobs, biome, dimension)
Shows "☁ Cloud invocation sent..." in the actionbar
Waits for the AI's decision
Applies it: "☁ Cloud decided: 5 minions (Critical threat — max invocations)"

If you're NOT cloud-enabled, the sword works exactly as before. Backward compatible. No cloud required.

The Architecture

┌─────────────────────────────────────────────────────────────┐
│                     MINECRAFT CLIENT                         │
│                                                             │
│  Player swings Sword of Lambda (cloud-enabled)              │
│       │                                                     │
│       ▼                                                     │
│  CloudCombatHandler builds context:                         │
│  {player, weapon, health, biome, dimension, nearby_mobs}   │
│       │                                                     │
│       ▼                                                     │
│  CloudApiClient.sendEvent(context)  ← async, non-blocking  │
└───────┬─────────────────────────────────────────────────────┘
        │ POST /event
        ▼
┌─────────────────────────────────────────────────────────────┐
│                    BACKEND (CDK Stack)                        │
│                                                             │
│  API Gateway (rate limited: 100/day, 10/sec)                │
│       │                                                     │
│       ▼                                                     │
│  Lambda: IngestEvent                                        │
│  - Receives context                                         │
│  - Calls Strands Agent (or rule-based fallback)             │
│  - Stores decision in DynamoDB (TTL: 60s)                   │
│  - Returns {request_id}                                     │
│                                                             │
│  DynamoDB: CloudSwordsResults                               │
│  - Partition key: request_id                                │
│  - TTL auto-cleanup                                         │
│                                                             │
│  Lambda: PollResult                                         │
│  - GET /result/{request_id}                                 │
│  - Returns decision when ready                              │
└───────┬─────────────────────────────────────────────────────┘
        │ Poll every 2s
        ▼
┌─────────────────────────────────────────────────────────────┐
│  CloudApiClient receives decision                           │
│       │                                                     │
│       ▼                                                     │
│  CloudCombatHandler.applyDecision()                         │
│  - Summon "☁ Cloud Invocation" minions (named, cyan)        │
│  - Or apply buff (regeneration, resistance, strength)       │
│  - Or do nothing ("No action needed")                       │
└─────────────────────────────────────────────────────────────┘

The Async Pattern: Fire-and-Forget + Polling

The game can NEVER freeze waiting for a network response. So I used the same pattern you'd use in any real-time application:

Fire: Send the event, get back a request_id immediately
Forget: Continue gameplay, don't block
Poll: Every 2 seconds (via ServerTickEvents), check if the result is ready
Apply: When ready, queue the callback for the server thread

// CloudApiClient — non-blocking
public void sendEvent(JsonObject context) {
    httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofString())
        .thenAccept(response -> {
            String requestId = parseRequestId(response.body());
            startPolling(requestId);
        });
}

// Polling on server tick (every 40 ticks = 2 seconds)
ServerTickEvents.END_SERVER_TICK.register(server -> {
    // Execute queued callbacks on server thread
    Runnable callback;
    while ((callback = pendingCallbacks.poll()) != null) {
        callback.run();
    }
});

The threading fix was critical. Java's HttpClient runs callbacks on its own async thread — but Minecraft entity spawning MUST happen on the server thread. Without the ConcurrentLinkedQueue bridge, buffs wouldn't apply and entities would crash the game.

The AI Agent: Strands + Ollama Cloud

The brain is a Strands Agent running gpt-oss:20b via Ollama Cloud (free tier — no Bedrock cost for development):

from strands import Agent
from strands.models.ollama import OllamaModel

SYSTEM_PROMPT = """You are a combat advisor for a Minecraft RPG mod.
Given player context, decide ONE action.

Available actions (respond ONLY with valid JSON):
- {"action": "summon", "count": 1-5, "duration": 5-15, "reason": "brief"}
- {"action": "buff", "effect": "regeneration|resistance|strength|speed", "duration": 3-10, "reason": "brief"}
- {"action": "none", "reason": "brief"}

Rules:
- If health < 8 and many mobs: summon max allies or buff resistance
- If health > 15 and few mobs: action none or summon 1
- Scale response to threat level
- ONLY output the JSON object, nothing else"""

model = OllamaModel(
    host="https://api.ollama.com",
    model_id="gpt-oss:20b",
)
agent = Agent(model=model, system_prompt=SYSTEM_PROMPT)

The agent receives context like:

{
  "player": "Carlos",
  "weapon": "sword_of_lambda",
  "health": 6,
  "nearby_mobs": 8,
  "biome": "nether_wastes",
  "dimension": "minecraft:the_nether"
}

And responds:

{"action": "summon", "count": 5, "duration": 10, "reason": "Critical threat"}

The system prompt enforces bounded actions — the AI can't break game balance because it can only choose from a fixed set of actions with capped values.

The CDK Stack

The production backend is a single CDK stack:

class CloudSwordsStack(Stack):
    def __init__(self, scope, id, **kwargs):
        super().__init__(scope, id, **kwargs)

        # DynamoDB for async results
        results_table = dynamodb.Table(self, "DecisionResults",
            partition_key={"name": "request_id", "type": "STRING"},
            time_to_live_attribute="ttl",
            billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
        )

        # Lambda: receive event, decide, store
        ingest_fn = _lambda.Function(self, "IngestEvent",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="handler.lambda_handler",
            code=_lambda.Code.from_asset("lambda/ingest"),
        )

        # Lambda: poll result
        poll_fn = _lambda.Function(self, "PollResult",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="handler.lambda_handler",
            code=_lambda.Code.from_asset("lambda/poll"),
        )

        # API Gateway with usage plan (rate limiting = gameplay mechanic)
        api = apigw.RestApi(self, "CloudSwordsApi")
        plan = api.add_usage_plan("Basic",
            throttle={"rate_limit": 10, "burst_limit": 20},
            quota={"limit": 100, "period": apigw.Period.DAY},
        )

The usage plan isn't just infrastructure — it's a gameplay mechanic. 100 requests per day means you can't spam the cloud. You have to be strategic about when you invoke it. Just like real API budgets.

Local Development: The Mock Server

For development, I run a mock server that integrates the Strands Agent directly:

# mock_server.py — localhost:8777
agent = None
try:
    from combat_advisor import create_agent, decide
    agent = create_agent()
    print("✓ Strands Agent loaded")
except:
    print("⚠ Agent unavailable, using rule-based fallback")

def decide_action(body):
    if agent:
        return decide(agent, body)
    # Fallback: simple rules
    if body["health"] < 8 and body["nearby_mobs"] > 5:
        return {"action": "summon", "count": 5, "duration": 10, "reason": "Critical"}
    return {"action": "summon", "count": 1, "duration": 5, "reason": "Standard"}

The mock server logs every interaction to interactions.json — which becomes the eval dataset for Blog 10.

Cloud Concepts Taught Through Gameplay

Game Mechanic	Cloud Concept
Tome of API Key	API keys + authentication
100 charges	Rate limiting / usage plans
Cloud API Requester (8s timer)	CI/CD deployment pipeline
`cloud_enabled=true` NBT	Feature flags
POST /event → request_id	Async event processing
Poll every 2s	Eventual consistency
AI decides action	Serverless compute (Lambda)
Bounded actions (1-5 minions)	Guardrails / IAM boundaries
CDK stack	Infrastructure as Code
Mock server	Local development environment

What I Learned

Threading in Minecraft is brutal — anything touching entities must be on the server thread. Async HTTP + game logic = careful queue management.
Rate limiting as gameplay works — players naturally strategize around limited cloud invocations. It teaches resource management without a lecture.
AI guardrails through system prompts — bounding the action space (1-5 minions, specific buffs) means the AI literally cannot break the game. The worst it can do is summon 5 minions when 3 would suffice.
Fallback patterns matter — if the cloud is down, the sword still works with local logic. Graceful degradation, taught through a video game.

What's Next

The AI makes decisions. But are they good decisions? In Blog 10, I build an evaluation framework that judges the agent's combat advice — using another AI as the judge. Meta? Absolutely. Useful? Surprisingly yes.

Code:

I'm Carlos Cortez — cloud engineer, gamer, and apparently someone who connects Minecraft to AI agents for fun. This is Breaking the Cloud.

LinkedIn · GitHub · Dev.to

I Built a Cloud Caravan Merchants Progression System as a cloud Marketplace

Carlos Cortez 🇵🇪 [AWS Hero] — Sat, 01 Aug 2026 16:10:01 +0000

Every cloud platform has a marketplace. AWS has the AWS Marketplace, Azure has its own, GCP too. They're places where you find specialized vendors offering services you can't build yourself — and the more you buy, the more they offer you.

We built that in Minecraft: Cloud Caravan Merchants. Three specialized traders that travel with donkeys carrying loot chests, offer 5 base trades each, and evolve into Elite merchants when you complete their missions — unlocking premium trades including Elytra, Tridents, and Totems of Undying.

🎮 The learning angle: The Elite progression system mirrors how cloud marketplace vendors work — start with basic offerings, prove you're a serious customer (complete trades), and unlock enterprise-tier products. The 3 merchant types map to cloud specializations: infrastructure (Packet Runner), platform (Protocol Broker), and application layer (Quantum Courier).

3 Merchants — 3 Cloud Layers

Merchant	Color	Specialty	Cloud Layer
Packet Runner	Cyan	Raw materials & resources	Infrastructure (IaaS)
Protocol Broker	Orange	Equipment & upgrades	Platform (PaaS)
Quantum Courier	Purple	Runes, tomes & magic	Application (SaaS)

Each spawns with a donkey carrying a loot chest with starter items. It's a welcome package — like free tier credits when you sign up for a cloud platform.

The Spawn Egg — Full Caravan Unit

The custom spawn egg doesn't just spawn a merchant. It spawns the entire unit:

Merchant entity (custom type, wandering trader model)
Donkey (tamed, with chest)
Loot in the chest (via loot table)
Leash connecting donkey to merchant

@Override
public ActionResult useOnBlock(ItemUsageContext context) {
    // 1. Spawn merchant
    CloudMerchantEntity merchant = merchantType.create(world);
    merchant.setPosition(pos.getX() + 0.5, pos.getY(), pos.getZ() + 0.5);
    world.spawnEntity(merchant);

    // 2. Spawn donkey with chest
    DonkeyEntity donkey = EntityType.DONKEY.create(world);
    donkey.setTame(true);
    donkey.setHasChest(true);

    // 3. Fill chest via NBT (inventory must be properly sized)
    NbtCompound donkeyNbt = new NbtCompound();
    donkey.writeNbt(donkeyNbt);
    donkeyNbt.putBoolean("ChestedHorse", true);
    NbtList itemsList = new NbtList();
    // ... add loot items to NBT
    donkey.readNbt(donkeyNbt);
    world.spawnEntity(donkey);

    // 4. Leash
    donkey.attachLeash(merchant, true);
}

The Donkey Chest Bug — A War Story

Fun debugging story: donkeys in 1.20.1 have a SimpleInventory field called items. When you call setHasChest(true), it's supposed to expand the inventory from 2 slots (saddle + armor) to 17 slots (+ 15 chest slots). But the expansion happens in onChestedStatusChanged(), which recreates the inventory.

If you try to put items in the chest BEFORE the inventory is properly sized, you get ArrayIndexOutOfBoundsException. If you put them AFTER spawning, the inventory might not be synced to the client yet.

The solution: write the chest contents via NBT. Set ChestedHorse: true and Items: [...] in the NBT compound, then call readNbt(). This forces the entity to reconstruct its inventory with the correct size AND the items already in place.

Elite Progression — The Loyalty Program

Complete all 5 base trades (use them until sold out) and the merchant evolves:

Base (5 trades) → Elite ★ (7 trades) → Elite ★★ (9 trades) → Elite ★★★ (11 trades)

Each promotion:

Adds 2 new trades with premium items
Changes the merchant's name to gold with stars
Triggers a Totem of Undying particle burst
Adds ambient END_ROD particles

private void checkElitePromotion() {
    int requiredTrades = 5 + (eliteLevel * 2);
    if (completedTrades >= requiredTrades && eliteLevel < 3) {
        eliteLevel++;
        applyEliteName(); // "★★ Protocol Broker ★★"
        addEliteTrades(this.getOffers(), eliteLevel);
        // Celebration particles
        world.spawnParticles(ParticleTypes.TOTEM_OF_UNDYING, ...);
    }
}

Elite Trades — Premium Vanilla Items

Cycle	Packet Runner	Protocol Broker	Quantum Courier
★	Quantum Shards, Spyglass	Netherlink Plating, Enchanted Golden Apple	Deployed Runes, Ender Eyes
★★	Cloud Edge Pearls, Echo Shards	Cloudsteel Plating, Trident	Grimoire of Architect, XP Bottles ×16
★★★	Nether Star, Totem of Undying	Netherite ×2, Elytra	Quantum Runes ×16, End Crystals

The endgame trades offer items that are normally very hard to obtain (Elytra, Trident) in exchange for Cloud Credits and rare materials. It's the cloud marketplace premium tier.

Natural Spawning — The Caravan Arrives

Every ~1 hour of real time, there's a 15% chance a caravan spawns 48-96 blocks from a random player:

private static void spawnCaravan(ServerWorld world, BlockPos pos, Random random) {
    int caravanSize = 1 + random.nextInt(3); // 1, 2, or 3 merchants
    // Shuffle types to avoid duplicates
    List<MerchantType> available = new ArrayList<>(List.of(MerchantType.values()));
    Collections.shuffle(available);

    for (int i = 0; i < caravanSize; i++) {
        // Spawn merchant + donkey at offset positions
        spawnUnit(world, pos, available.get(i), i * 2.5);
    }
}

A full caravan (3 merchants) gives you access to all specializations at once. It's like a cloud conference — all the vendors in one place.

Never Despawn — Persistent Infrastructure

Unlike vanilla wandering traders that disappear after 40 minutes, Cloud Merchants are permanent:

this.setPersistent();
this.setDespawnDelay(Integer.MAX_VALUE);

Once you find (or spawn) a merchant, they stay forever. They're your infrastructure — you invested in them, they stick around.

Custom Textures — Visual Identity

Each merchant uses a recolored wandering trader texture generated with Python PIL:

def recolor(data, hue_shift):
    """Shift hue of non-skin pixels"""
    for pixel in data:
        if not is_skin_tone(pixel):
            h, s, v = rgb_to_hsv(pixel)
            h = (h + hue_shift) % 1.0
            pixel = hsv_to_rgb(h, s, v)

Packet Runner: cyan shift (networking)
Protocol Broker: orange shift (fire/security)
Quantum Courier: purple shift (quantum/magic)

Rune Table Recipes — Mod Interop

We also added recipes for the Rune Crafting Altar (from the Runes mod):

Rune	Altar Input	Output	Savings
Packet	Cloud Shard + Redstone	×2	-1 Lapis, +1 output
Protocol	Packet Rune + Blazing Shard	×2	-1 Gold, +1 output
Quantum	Protocol Rune + Quantum Shard	×2	-1 Ender Pearl, +1 output

This is mod interop — using another mod's crafting system to provide alternative recipes. Like using a third-party service that's cheaper than building your own.

Lessons Learned

Challenge	Solution
Donkey chest inventory size	Write items via NBT, not direct slot access
Entity renderer missing = crash	Register renderer in ClientModInitializer
Wandering trader despawn timer	Override with MAX_VALUE despawnDelay
Spawn egg model missing	`template_spawn_egg` parent model
Trade completion tracking	NBT persistence (CompletedTrades, EliteLevel)

Series Wrap-Up

Over 8 posts, we went from "what if AWS services were Minecraft swords" to a full tech-magic mod with:

7 cloud-themed swords with unique abilities
28 spells across 9 spellbooks
4 armor sets with set bonuses
Multiblock machines with BFS energy networks
7 villager professions with a Cloud Credits economy
3 Caravan Merchants with Elite progression
Full compatibility with 650+ mod modpacks

The mod teaches cloud computing by making you experience the concepts. You don't read about Auto Scaling — you feel your damage stack. You don't study fault tolerance — you dodge attacks with Quantum Armor. You don't memorize service tiers — you progress through them.

That's the power of game-based learning. And we built it all with AI as a co-pilot.

The complete mod is on GitHub. Clone it, build it, play it.

Connect with me:

I'm Carlos Cortez, this is Breaking the Cloud, and this series proved that the best way to learn cloud is to play it. See you in the next adventure! 🎮☁️⚔️

Designing an Armor Progression That Mirrors Cloud Certification Levels

Carlos Cortez 🇵🇪 [AWS Hero] — Sat, 01 Aug 2026 16:09:22 +0000

In cloud computing, you level up through certifications: Practitioner → Associate → Professional → Specialty. Each level unlocks more capabilities and proves deeper expertise. We built the same thing as armor sets in Minecraft.

4 armor sets, each with a set bonus, a dodge mechanic, and a clear progression path that mirrors going from cloud beginner to architect.

🎮 The learning angle: Dataflow Armor gives Speed (network optimization). Netherlink gives Fire Resistance (firewall protection). Quantum gives damage resistance (fault tolerance). Cloudsteel gives flight (cloud-native freedom). The set bonuses ARE the cloud concepts.

4 Armor Sets — 4 Cloud Maturity Levels

Set	Tier	Armor	Toughness	Cloud Level
Dataflow	Post-Iron	18	1.5	Practitioner — basic networking
Netherlink	Post-Diamond	20	2.5	Associate — security focus
Quantum	Post-Netherite	24	3.5	Professional — fault tolerance
Cloudsteel	Endgame	24	4.0	Specialty — full mastery

Each set has a unique visual identity:

Dataflow: Cyan/teal with circuit line patterns + bright nodes
Netherlink: Dark red/orange with ember glow edges
Quantum: Purple with glitch pixels (digital distortion)
Cloudsteel: White/silver with cloud wisps + luminous edges

Set Bonuses — Cloud Capabilities

When you wear all 4 pieces of a set, you get passive effects:

Set	Effects	Dodge %	Cloud Concept
Dataflow	Speed I + Haste I	10%	Network optimization — faster throughput
Netherlink	Fire Resistance + Strength I (Nether)	13%	Firewall — immune to fire attacks
Quantum	Resistance I	15%	Fault tolerance — reduced impact
Cloudsteel	Slow Falling + Jump Boost II	20%	Cloud-native — freedom of movement

The Dodge System

Every set has a chance to completely nullify incoming damage. When triggered:

Portal particles burst around the player
Enderman teleport sound plays
Damage is set to zero

public class QuantumDodgeHandler {
    public static void register() {
        // ALLOW_DAMAGE event — return false to cancel damage
        DamageEvents.ALLOW.register((entity, source, amount) -> {
            if (!(entity instanceof PlayerEntity player)) return true;

            float dodgeChance = getSetDodgeChance(player);
            if (dodgeChance > 0 && player.getRandom().nextFloat() < dodgeChance) {
                // Dodge! Cancel damage
                spawnDodgeParticles(player);
                return false;
            }
            return true;
        });
    }
}

Higher-tier armor = higher dodge chance. It's probabilistic fault tolerance — like how multi-AZ deployments reduce (but don't eliminate) the chance of downtime.

Crafting Progression — The Certification Path

You don't jump straight to Cloudsteel. The progression uses the Smithing Table:

Iron Armor + Dataflow Plating + Cloud Shard → Dataflow Armor
Diamond Armor + Netherlink Plating + Blazing Shard → Netherlink Armor
Netherite Armor + Quantum Plating + Quantum Shard → Quantum Armor
Quantum Armor + Cloudsteel Plating + Cloud Edge Pearl → Cloudsteel Armor

Each tier requires the previous tier's armor as base. You can't skip levels — just like certifications build on each other.

Platings (4 ingots each)

III
I_I  →  1 Plating
___

Simple shaped recipe. 4 platings per set (helmet, chest, legs, boots) × 4 ingots = 16 ingots per full set.

Accessories Integration — Optional Dependencies

The mod works with or without the Accessories mod installed. When present, the Cloud Engineer Lens can be equipped in the face trinket slot instead of taking the helmet slot.

public class AccessoriesCompat {
    public static boolean isEquipped(PlayerEntity player, Item item) {
        // Runtime check — only call if mod is loaded
        var component = AccessoriesCapability.get(player);
        if (component == null) return false;
        return component.isEquipped(item);
    }
}

Detection priority: Accessories slot → Trinkets slot → Helmet → Offhand → Mainhand.

This is the optional dependency pattern — your mod works standalone, but integrates with other mods when available. Like how a cloud service works alone but integrates with others via APIs.

Spell Book Slot — Cross-System Integration

All 18 tomes (9 base + 9 empowered) implement a SpellBookItem interface and register in the trinkets:charm/spell_book tag. This means they can be equipped in the Spell Book accessory slot from other magic mods.

Cross-mod compatibility isn't just nice to have — it's essential for modpack inclusion. Same as cloud services: if your service doesn't integrate with the ecosystem, nobody uses it.

Nether Star Duplication — Endgame Recipe

4 Cloud Edge Pearls + 4 Data-Infused Quantum Shards + 1 Nether Star → 2 Nether Stars

An endgame recipe that avoids Wither farming. Expensive inputs, but saves time. It's like Reserved Instances — pay more upfront in materials, save time in the long run.

Lessons Learned

Challenge	Solution
Set bonus detection every tick is expensive	Check every 20 ticks (1 second), not every tick
Dodge feels unfair in PvP	Balance with low percentages (10-20%)
Accessories as optional dep	`compileOnly` + runtime class check
Armor textures	Recolor vanilla iron/netherite with unique effects
Progression too fast	Require previous tier as base (sequential)

What's Next

We have swords, spells, machines, and armor. But what about NPCs that sell you things? In the final post of this series, we'll build Cloud Caravan Merchants — wandering traders with donkeys, loot chests, and an Elite progression system. Because every cloud platform needs a marketplace.

Connect with me:

I'm Carlos Cortez, this is Breaking the Cloud, and today we armored up with cloud certifications. See you in the next one! 🛡️

I Built a Magic System Where Every Spell is a Cloud Operation

Carlos Cortez 🇵🇪 [AWS Hero] — Sat, 01 Aug 2026 15:13:45 +0000

What does casting a spell have in common with calling an API? More than you'd think. Both consume resources (mana/runes vs. API credits), both have cooldowns (cast time vs. rate limits), both have tiers (spell levels vs. service tiers), and both produce effects at a distance.

So we built Cloud Spellbooks — 9 tomes, 28 spells, 5 rune types, and a full upgrade path. Every spell is named after a cloud operation, and every mechanic maps to how cloud services actually work.

🎮 The learning angle: When you cast "Auto Scale" and it does AoE damage that scales with targets, you're experiencing what Auto Scaling does. When "Region Failover" fully heals you with a 5-minute cooldown, you understand why failover is a last resort with a long recovery time. The game mechanics ARE the cloud concepts.

The Architecture: Spell Engine Integration

We didn't build a spell system from scratch. We integrated with Spell Engine — a Fabric mod that provides data-driven spell registration, the Binding Table mechanic, and compatibility with other magic mods.

Our spells are defined as JSON:

{
  "school": "arcane",
  "range": 6.0,
  "cast": { "duration": 0, "channel_ticks": 0 },
  "cost": { "item_id": "cloudswords:packet_rune", "count": 1 },
  "cooldown": 3.0,
  "impact": [
    { "action": { "type": "spell_engine:damage", "damage": { "spell_power_coefficient": 0.5 } } },
    { "action": { "type": "spell_engine:status_effect", "status_effect": { "effect_id": "minecraft:glowing", "duration": 100 } } }
  ]
}

This is Infrastructure as Code for magic. Define your spell's behavior declaratively, and the engine handles execution.

9 Tomes, 3 Tiers — Like Service Tiers

Tier	Tomes	Rune Cost	Spell Power	Cloud Analogy
T1	Hacker's Notebook, Codex of Uptime	Packet Rune	+2	Free tier — basic operations
T2	Pipeline, SRE Codex, Cloud Captain, DevOps	Protocol Rune	+4	Standard tier — production workloads
T3	Serverless Bible, Grimoire of Architect, Zero Downtime	Quantum Rune	+7	Enterprise tier — mission-critical

Each tier requires a more expensive rune to cast. Just like cloud services — basic operations are cheap, enterprise features cost more.

28 Spells — Cloud Operations as Game Mechanics

Here are some highlights showing how cloud concepts become spells:

Spell	Tome	Effect	Cloud Concept
Ping	Hacker's Notebook	Damage + Glowing	Network ping — reveals target
DDoS Flood	Hacker's Notebook	Triple rapid hit	Distributed denial — overwhelming force
Firewall Burst	Codex of Uptime	Resistance + Fire Res	Firewall — blocks incoming
Auto Scale	Tome of Pipeline	AoE damage radius 6	Auto Scaling — handles multiple targets
Incident Response	SRE Codex	Regen III + Absorption	Incident response — emergency healing
Load Balance	Cloud Captain	Strength + Speed to allies	Load balancer — distributes power
CI/CD Sprint	DevOps Manifesto	Speed III + Haste II	CI/CD — everything faster
Cloud Nuke	Serverless Bible	Massive AoE	Nuclear option — destroy everything
Quantum Tunnel	Grimoire of Architect	Teleport 20 blocks	Quantum networking — instant transfer
Region Failover	Zero Downtime	Full heal + Res IV	Failover — complete recovery, long CD

The cooldowns tell a story too: "Ping" is 3 seconds (quick diagnostic), "Region Failover" is 5 minutes (disaster recovery is expensive and slow).

Rune System — API Credits

Runes are consumed when you cast spells. They're your API credits:

Rune	Tier	Crafting	Analogy
Packet Rune	T1	Cloud Shard + Redstone + Lapis	Basic API call
Protocol Rune	T2	Packet Rune + Blazing Shard + Gold	Standard API call
Quantum Rune	T3	Protocol Rune + Quantum Shard + Ender Pearl	Enterprise API call

Higher-tier spells consume more expensive runes. You can't spam "Cloud Nuke" because Quantum Runes are expensive to craft — just like you can't spam expensive API calls without budget consequences.

Upgrade Paths — Service Evolution

Hacker's Notebook → Tome of the Pipeline → The Serverless Bible
Codex of Uptime → The SRE Codex → Grimoire of Zero Downtime
                → Tome of the Cloud Captain → Grimoire of the Architect
                → DevOps Manifesto (independent)

Upgrades use the Smithing Table: Tome + Compiled/Deployed Rune + vanilla item. The path from T1 to T3 mirrors how cloud services evolve — you start with basic tools and upgrade to enterprise solutions.

Empowered Tomes — Premium Tier

Each tome has an "Empowered" variant (+2 extra spell power) crafted with rare materials:

T1 Empowered: + Amethyst Shard
T2 Empowered: + Blaze Rod
T3 Empowered: + Nether Star

It's like going from standard to reserved instances — same service, better performance, higher upfront cost.

Right-Click Casting — The UX

public abstract class SpellTome extends Item {
    @Override
    public TypedActionResult<ItemStack> use(World world, PlayerEntity player, Hand hand) {
        // 1. Check rune in inventory
        if (!hasRune(player, getRequiredRune())) {
            player.sendMessage(Text.literal("§c✗ Missing " + getRuneName()));
            return TypedActionResult.fail(stack);
        }
        // 2. Check cooldown
        if (isOnCooldown(player)) return TypedActionResult.fail(stack);
        // 3. Consume rune
        consumeRune(player, getRequiredRune());
        // 4. Cast spell
        castSpell(world, player);
        // 5. Apply cooldown
        applyCooldown(player);
        return TypedActionResult.success(stack);
    }
}

Simple flow: verify resources → consume → execute → cooldown. It's the same pattern as any API call: authenticate → charge → process → rate limit.

Lessons Learned

Challenge	Solution
28 spells is a lot of JSON	Data-driven approach — one template, many configs
Balancing cooldowns	Map to real-world operation frequency
Rune economy	Make T1 cheap, T3 expensive — natural gating
Spell Engine compatibility	Use `is_proxy: true` for Binding Table
Visual identity per tome	Isometric textures with unique colors + symbols

What's Next

Magic is great, but it needs infrastructure to support it. In the next post, we'll build a multiblock Data Melter with BFS energy transfer through cables — because even in a fantasy world, you need power grids.

Connect with me:

I'm Carlos Cortez, this is Breaking the Cloud, and today we cast cloud operations as spells. See you in the next one! ✨

Monthly Amazon Location Service Updates - 2026.07

Yasunori Kirimoto — Sat, 01 Aug 2026 02:10:06 +0000

Monthly Amazon Location Service Updates - 2026.07

This is a summary of the July updates for Amazon Location Service.

2026.07 Updates

Amazon Location Service enhances Places APIs with new address and search options
This release introduces the AddressNamesMode parameter to control how address components are returned, the AddressTranslations parameter to return place name translations in 50+ languages, and the TravelMode parameter to optimize search results for users in motion. It also adds the DriveThrough attribute indicating whether a place offers drive-through service, and a Parsing.AdditionalInfo response field in the Geocode API that details how the input address was interpreted.

Other Info

Amazon Location Service Demo
Official Amazon Location Service demo.

Amazon Location Service Developer Guide
Official Amazon Location Service Documentation.

AWS Geospatial
Official AWS Geospatial samples.

maplibregljs-amazon-location-service-starter
Build environment to get started with Amazon Location Service.

dev.to
Articles on Amazon Location Service.

tags - Amazon Location Service
tags - Try
Notes on Amazon Location Service. (Japanese)

Yasunori Kirimoto for AWS Heroes

Jul 11

Monthly Amazon Location Service Updates - 2026.06

#amazonlocationservice #amazonlocationserviceupdates

1 min read

Send your events later

Allen Helton — Wed, 29 Jul 2026 20:26:23 +0000

I build a lot of automated workflows. Across the suite of apps that powers Ready, Set, Cloud, I have a dozen or so mature, long-running automations that do some work, wait for something, then do more work. A great example is my newsletter. I often write it on Fridays and publish it on Mondays. After I open a PR for an issue, my backend service does some initial work, but it waits until Monday to send the emails and start analytics tracking.

In the past, I've used the Step Functions Wait state for this because it's the exact use case it was designed for. But as part of a big refactor I did to separate Ready, Set, Cloud logic from the generic newsletter logic, I needed to find a simpler way to schedule work - ideally as part of my CI/CD pipeline.

I thought using the AWS SDK in a script to create a one-time EventBridge schedule that published an event at a designated time was my best bet. But that brings in some implications with IAM and passing roles to the scheduler in order to invoke a PutEvents call. It ended up being way too messy for a simple CI script.

But I had an epiphany that immediately made me think, "I should write about this" because it's very simple but not always obvious.

The surprisingly-simple-yet-not-obvious solution

I moved the EventBridge Scheduler event publish behind an EventBridge rule. Anything that can call PutEvents can now ask for a deferred event.

It's essentially a one-time loop with a customizable wait step in it. Here's an example payload of an event to get my static site to rebuild because future-dated content doesn't render automatically.

{
  "Source": "readysetcloud.app",
  "DetailType": "Schedule Event",
  "Detail": {
    "at": "2026-08-03T09:00:00",
    "timezone": "America/Chicago",
    "name": "newsletter-rebuild-227",
    "event": {
      "detailType": "Trigger Site Rebuild",
      "detail": { "issueNumber": 227 }
    }
  }
}

Publishing the above event will create a schedule that triggers at 9 a.m. CDT on August 3. The schedule will publish a Trigger Site Rebuild event back onto the bus, which is wired up to a rule that kicks off a build in Amplify. Nifty, right?

Just a little bit of logic

While I normally try to skip Lambda in this type of integration, I needed it for a couple of reasons. One, you can't set the EventBridge Scheduler as a target of a rule 🙃. And even if you could, I would have lost my ability to maintain schedules. If I needed to reschedule an event for a different time, I would have been out of luck.

So I wrote a Lambda function as the target that will upsert the schedule, validate the event (WHICH YOU SHOULD ALWAYS DO), and transform the provided time to the format the scheduler expects.

As a bonus, the Lambda function also recognizes when the scheduled date is in the past, and will publish the event immediately or throw it away, depending on a parameter optionally provided in the event payload.

So in this case, it made a lot of sense to add the function in the middle, which increases the complexity of the build, but the tradeoff is a richer experience.

Build your own platform

A few years ago, I made what I called my serverless toolbox, with some tools I could use across all of my apps that ran in the same AWS account. What I didn't know then was that I was essentially building a platform for all of my applications to run.

I've since abandoned the toolbox and have turned my attention fully toward building a platform my apps run on. It's the peak of DRY (Don't Repeat Yourself) and in this case, is incredibly useful. I've added this scheduling primitive to my platform repo so it can be used everywhere - my newsletter service, blog service, and all the other services I've built that power my website.

I only need to solve this problem once. Time and time again I've reimplemented things that worked in the past only to find myself with 4 versions of the same thing and at least one of them stale. So I make an intentional effort to consolidate the shared work and make the entry point to it as simple as possible (what's simpler than calling PutEvents?!).

Anyway, this is intentionally short and sweet. I was excited by the idea of delaying an event with nothing more than PutEvents, and wanted to share. For the full source code, you can check out the function here or view the entire PR to my platform app.

Happy coding!

Building AI Agents with Embabel, Spring AI and Amazon Bedrock AgentCore - Part 2 Develop Agents with Embabel shell

Vadym Kazulkin — Mon, 27 Jul 2026 15:35:30 +0000

Introduction

In part 1, we addressed some challenges when developing AI agents with Spring AI frameworks only. We start exploring how Embabel is capable of addressing and solving those challenges. Throughout this series, we'll start improving the agent developed in the Provide MCP tools for Conference application via AgentCore Gateway article using the Embabel framework. First, we'll use the Embabel shell, then the web application. You can look at my GitHub embabel-1.x-conference-app-agent-local repository for the example. I updated them to use the recently released Embabel version 1.0.0.

Implement AI agents with Embabel shell

In this article, we'll implement 2 AI agents capable of responding to the following prompts:

"Please provide me with the list of conferences, including their IDs, with Java topics happening in 2027, with the call for papers open today. Also, provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference."
"Please create a talk with a cool title (max 60 characters long) and description (max 300 characters long) about using Spring AI on the Amazon Bedrock AgentCore service. Then provide me with the list of conferences, including their IDs, with Java topics happening in 2026 and 2027, with the call for papers open today. Finally, for each conference, apply individually for it with the talk just created."

As our sample application also supports only searching for the conference as an attendee, not as a speaker, you can similarly implement those agents.

Before we start, we need to understand the following key concepts of the Embabel framework:

Actions: Steps an agent takes
Goals: What an agent is trying to achieve
Conditions: Conditions to assess before executing an action or determining that a goal has been achieved. Conditions are reassessed after each action is executed.
Domain model: Objects underpinning the flow and informing Actions, Goals and Conditions.
Plan: A sequence of actions to achieve a goal. Plans are dynamically formulated by the system, not the programmer. The system replans after the completion of each action, allowing it to adapt to new information as well as observe the effects of the previous action. This is effectively an OODA loop.

Now let's first define the actions for our agent - steps an agent takes.
We can split our first agent into the following actions:

Request: "Please provide me with the list of conferences, including their IDs, with Java topics happening in 2027, with the call for papers open today". LLM responds to it.
Request: "Provide me with the list of my talks with this topic in the title". LLM responds to it.
Request: "For each conference and talk retrieved, apply individually for the conference". This request means that the LLM uses its responses from the first 2 requests as the request and responds to it.

And we can split our second agent into the following actions:

Request: "Please create a talk with a cool title (max 60 characters long) and description (max 300 characters long) about using Spring AI on the Amazon Bedrock AgentCore service". LLM responds to it.
Request: "Provide me with the list of my talks with this topic in the title". LLM responds to it.
Request: "For each conference and talk, apply individually for the conference". This request means that the LLM uses its responses from the first 2 requests as the request and responds to it.

In this particular case, steps 2 and 3 are mainly the same for our 2 agents. The difference lies in step 1, where we either retrieve an existing talk(s) or create a new one. Your case might be different, so the design of the agents and their actions might also be different.

First, let's declare some important dependencies in pom.xml to use the Embabel framework (I use version 1.0.0) and its shell:

 <dependency>
    <groupId>com.embabel.agent</groupId>
    <artifactId>embabel-agent-starter-bedrock</artifactId>
    <version>${embabel-agent.version}</version>
 </dependency>
 <dependency>
    <groupId>com.embabel.agent</groupId>
    <artifactId>embabel-agent-starter-shell</artifactId>
    <version>${embabel-agent.version}</version>
 </dependency>

Also, make sure not to declare the spring-boot-starter-web dependency, as we'll implement a web application in the next article.

The EmbabelConferenceApplication is the main entry point of our application. Here we also load Amazon Bedrock models, which we declared in the additional-bedrock.yaml:

@Configuration
class AdditionalBedrockModels {
@Bean
   BedrockModelLoader bedrockModels() {
   return new BedrockModelLoader(new DefaultResourceLoader(), "classpath:models/additional-bedrock.yaml");
   }
}

In the Domain class, we declared a domain object (Java record) for each request and response, which corresponds to the agent actions described above. For example, for the action to search for the talks, those domain objects look like this:

// Request to LLM to create the talk
public record ConferenceSearchRequest(String topic, LocalDate startDate, 
  LocalDate endDate, LocalDate callForPapersOpenOnThistDate) 
}


// Response from LLM for the talk creation request
public record Conference (Integer conferenceId, String name, Set<String> topics, String homepage, LocalDate startDate, LocalDate endDate, LocalDate callForPapersStartDate, LocalDate callForPapersEndDate,  String city, String linkToCallforPapers) {
}

Now let's design those 2 agents. We defined the common actions for both of them in the AbstractConferenceAgent class. We defined individual agent-specific actions in the CreateTalksAndApplyForConferencesAgent and SearchForTalksAndApplyForConferencesAgent classes, respectively.

Let's explain the talk search agent action from the SearchForTalksAndApplyForConferencesAgent class in more detail. First of all, we annotate each agent with Embabel's Agent annotation, for example:

@Agent(name=SearchForTalksAndApplyForConferencesAgent.AGENT_NAME , description = "search for the talk(s), search for the conference(s) by the given criteria (all, by the topic, by the date range, and by call for papers being open on some date), and apply for them with the found talks")
public final class SearchForTalksAndApplyForConferencesAgent extends AbstractConferenceAgent {
 ....
}

Here we give the agent a name and description.

The first question is how we can break down a big unstructured prompt into the 3 actions that we described above? Let's start with the action to search for the talk first. The answer is: we can ask LLMs to do this job. Even cheap models are already capable of doing this. Here is how this looks for the action: "Please provide me with the list of conferences, including their IDs, with Java topics happening in 2027, with the call for papers open today"

@Action
Domain.TalkSearchRequest extractTalkSearchRequest(UserInput userInput, OperationContext context) {
return  context.ai()
   .withLlm(LlmOptions.withModel("us.amazon.nova-pro-v1:0"))
     .createObject("""
Create a talk search request from this user input, extracting the optional parameters like talk title substring, conference start and end date, and whether the call for papers is still open on a certain date. 
Don't include any other information in the request.
Here is the user input:
     %s""".formatted(userInput.getContent()), Domain.TalkSearchRequest.class);
}

Let's explain step by step what's happening here. The extractTalkSearchRequest method is an action, which the Agent executes. That's why we annotated it with the Embabel Action annotation. This method also takes 2 parameters, both coming from the Embabel framework:

UserInput object, which represents the unstructured user prompt
OperationContext, which gives us access to the AI functionality

We first invoke the ai() method of the context object and then provide the LLM to do the job. We can either pass the concrete model AI or use other options that are available, like using the default LLM, LLM by role, and others. As a result, we get back the Embabel implementation of the PromptRunnerOperations. Or, we can declare such LLMs in the application.properties, for example :

embabel.models.default-llm=us.anthropic.claude-sonnet-4-6
embabel.models.llms.best=us.anthropic.claude-sonnet-4-6
embabel.models.llms.balanced=us.amazon.nova-pro-v1:0

By invoking the createObject() method on the PromptRunnerOperations implementation, we pass the user prompt and instruction to the LLM on how to split this prompt into one subtask or action. It should only contain the information about the talk search criteria. Additionally, we pass the domain object, in this case, the object of type Domain.TalkSearchRequest. Embabel will make the best effort to map the LLM response to this domain object, which additionally provides us with type safety.

The next step is to implement the talk search itself. For this, we define another action in the SearchForTalksAndApplyForConferencesAgent class:

@Action
Domain.Talks talkSearch(Domain.TalkSearchRequest talkSearchRequest, Ai ai) {        
return config.speaker()     
    .promptRunner(ai)            
    .withPromptContributors(List.of(talkSearchRequest))
    .withToolGroup(this.getMcpToolGroupByName("get-talks-by-"))
    .createObject("Search for the talks with the given criteria", Domain.Talks.class);
}

Please note that the object of the Domain.TalkSearchRequest type, which is the output of the previous action, is now the input of this action. Also, we inject the Ai object coming from the Embabel framework this time directly. Let's first ignore the config.speaker invocation, which we'll explain later. Here we pass the prompt, which contains the properties required to search for the talk. When with createObject() method invocation, we once again pass the prompt and the domain object, which should match the outcome, in this case, the object of type Domain.Talks.

To respond to this prompt, the agent needs tools. Please review the articles Develop local MCP client for Conference application and Provide MCP tools for Conference application via AgentCore Gateway to understand the whole setup. I implemented very similar code to initialize the MCP client connected to the Bedrock AgentCore Gateway in the McpToolService class. MCP client is then exposed as a Spring bean in the EmbabelConferenceApplication class:

@Bean 
@Primary
public McpSyncClient geMcpClient() {
    return this.mcpToolService.getMcpClient();
}

By invoking the withToolGroup(this.getMcpToolGroupByName("get-talks-by-")) method on the PromptRunner (see above), we achieved another goal that we described in part 1. We not only split the prompt into individual actions, but are now also able to provide exact MCP tool(s) required to execute it. Embabel provides the ToolGroup abstraction for this. In this case, we give the LLM only the tools whose names consist of the get-talks-by- substring. I implemented the MCP tool filtering in the AbstractConferenceAgent class:

protected ToolGroup getMcpToolGroupByName(String name) {
    var  toolGroupDescription=  ToolGroupDescription.Companion.invoke(
         "A collection of tools to interact with 
         the MCP conference search service",
         "location",""
   );

  var noOp= ToolCallContextMcpMetaConverter.Companion.noOp();   

  return new McpToolGroup(
          toolGroupDescription,
          "Vadym",
          name,
          Set.of(ToolGroupPermission.INTERNET_ACCESS),
          List.of(mcpClient),
          callback -> filterMcpTool(callback, name),
          noOp
      );
}

private boolean filterMcpTool (ToolCallback toolCallback, String name) {
    // not passing name means return true -> pass all tools
      if(name==null) return true;
      return toolCallback.getToolDefinition().name().contains(name);
 }

The downside is that if we were to rename the MCP tools, we'd need to adjust their names here also. The alternative is to use semantic search for the tools. The alternative is to use semantic search for the tools.

Now let's come back to the config.speaker() method invocation, which we saw in the talkSearch action. Embabel gives the possibility to create different roles and attach different LLMs to them. In our case, we have 2 roles: conference attendee and conference speaker. We can define these roles either programmatically or in the application.properties:

conference.attendee.llm.role=best
conference.attendee.persona.role=Conference attendee
conference.attendee.persona.goal=Conduct a thorough conference search with the information (topics, conference dates) provided so I decide which to attend.
conference.attendee.persona.backstory=As a Conference Attendee Specialist, your mission is to find a suitable conference that matches the given request


conference.speaker.llm.role=best
conference.speaker.persona.role=Conference speaker
conference.speaker.persona.goal=Conduct the search for the talks or create the talk with the given criteria with the conference application for those talks
conference.speaker.persona.backstory=As a Conference Speaker Specialist, your mission is to search for the talk or create the talk with the given criteria with the conference application for those talks

We also link each role to the LLM model by referencing the previously defined LLM strategy (in our case, by the name best). See the LLM definition once again:

embabel.models.default-llm=us.anthropic.claude-sonnet-4-6
embabel.models.llms.best=us.anthropic.claude-sonnet-4-6
embabel.models.llms.balanced=us.amazon.nova-pro-v1:0

In our case, the best LLM strategy will use the us.anthropic.claude-sonnet-4-6 model.
Next, we defined ConferenceConfig to have access to the roles and actors:

@Validated
@ConfigurationProperties(prefix = "conference")
public record ConferenceConfig(Actor<RoleGoalBackstory> attendee,   
    Actor<RoleGoalBackstory> speaker) {
}

The conference prefix, together with Actor definitions, determines the mapping to the roles. For example, Actor speaker matched the conference.speaker role definition in the application.properties.

Finally, we injected the ConferenceConfig object (as config) into the constructor of each agent.

Similarly, we split the initial prompt into other individual steps or actions for searching for conferences. See the extractConferenceSearchRequest and conferenceSearch agents in the AbstractConferenceAgent class. Also, the last step (applying for conferences with talks) looks similar. See the applyForConference action in the same class. There are 2 things to notice here:

We don't ask the LLM to split the prompt. This is because the input to this action is the output of 2 actions before: talkSearch and conferenceSearch, and 2 domain objects, Domain.Conferences and Domain.Talks respectively. Both become the input to the PromptContributors.
This agent is additionally annotated with the AchievesGoal annotation to outline that this action execution is the final step that the agent should take in the workflow. See the complete code:

@Action
@AchievesGoal(description = "apply for the conferences with their IDs and talk IDs and provide status of the application")
    Domain.ConferenceApplications applyForConference(Domain.Conferences conferences, Domain.Talks talks, Ai ai) {
     return config.speaker()
         .promptRunner(ai)
         .withPromptContributors(List.of(conferences, talks))
         .withToolGroup(this.getMcpToolGroupByName("apply-to-conference"))
         .createObject("Apply for the conference with the given criteria", Domain.ConferenceApplications.class);
 }

We're completely done with the SearchForTalksAndApplyForConferencesAgent agent.

The implementation of the agent CreateTalksAndApplyForConferencesAgent looks very similar. The biggest difference is the first step: instead of searching for the existing talks, the talk is created. See the extractTalkCreationRequest and createTalks actions.

Now come the biggest questions:

How is Embabel capable of identifying which agent to execute?
How is Embabel capable of identifying the order of actions to execute for the individual agent? We only defined the last action to finally achieve the goal properly by annotation.

Let's build the application with mvn clean package and run it locally with mvn spring-boot:run. We see that Embabel starts the shell, and we can execute the prompt by running:

x 'Please provide me with the list of conferences, including their IDs, with Java topics happening in 2027, with the call for papers open today. Also, provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference.'

Let's see what happens next:

Embabel identifies 2 agents, takes their description to compute the score, and chooses a concrete agent to execute based on the user input (prompt). I assume Embabel also uses an LLM for it. As a result, it computed: As a result, it computed :

SearchForExistingTalksAndApplyForConferencesAgent: 0,97
CreateNewTalksAndApplyForConferencesAgent: 0,05

So, the SearchForExistingTalksAndApplyForConferencesAgent agent is clearly a winner.

Next, we see that Embabel formulates a plan to achieve the goal:

The sequence of agent actions to execute is a correct one :

Embabel could also come up with the plan to first search for the conferences, then for the talks, and then to apply. Both plans are valid. How does Embabel create such a plan? The system, not the programmer, dynamically formulates plans. The system replans after the completion of each action, allowing it to adapt to new information as well as observe the effects of the previous action. This is effectively an OODA loop. Emabebl analyzes the input and output types of each agent that we defined. This also helps to formulate the plan.

We see how Embabel executes those individual actions in the identified order. It logs all structured outputs of each action. The final output looks like this:

It found 2 talks and 1 conference to apply those talks that matched the criteria that we defined in the prompt. Finally, Embabel provides the statistics of the LLM used, the number of input (prompt) and output (completion) tokens spent. We also see the total cost of the agent execution. Embabel maintains the LLM pricing database. I don't expect this information to be very accurate, but it provides some indication. We also see the MCP tool names that the LLM used during the agent execution, their response times, and the number of failures.

If we execute the prompt x 'Please create a talk with a cool title (max 60 characters long) and description (max 300 characters long) about using Spring AI on the Amazon Bedrock AgentCore service. Then provide me with the list of conferences, including their IDs, with Java topics happening in 2026 and 2027, with the call for papers open today. Finally, for each conference, apply individually for it with the talk just created.', we'll see that Embabel chooses to execute the CreateNewTalksAndApplyForConferencesAgent agent :

It then formulates the plan and executes the actions to achieve the goal:

Finally, it provides the same statistics, which we previously explained:

There are many more features that Embabel provides, like retries, which we'll explore in later articles.

Conclusion

In this article, we developed our first AI agents with the Embabel shell. We learned many concepts of this framework, such as agents, actions, and goals. In the next article, we'll slightly adjust our application to convert it into a web application.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Beyond The Single-Agent Ceiling: Scale Out With MCP Agent Teams

Guy — Fri, 24 Jul 2026 22:10:08 +0000

Most people's experience with AI is a conversation with one assistant. ChatGPT, Claude, and similar products present one conversational partner. You ask it a question, it reasons, perhaps calls a few tools, and gives you an answer.

That experience creates a natural architectural instinct: if one agent is useful, make that one agent more capable. Give it better instructions. Connect more tools. Add more context. Increase its token budget. Upgrade the model.

That is the right place to start.

It is not an infinitely scalable place to finish.

At some point, the single agent is no longer doing one coherent job. Its instructions contain a job description, an organization chart, a workflow engine, a memory policy, a delegation policy, a review checklist, and a recovery procedure. Its context contains the request, the plan, intermediate evidence, failed attempts, tool results, and the agent's own conclusions about all of them. Its tool list spans several domains. It is asked to investigate, calculate, edit, criticize, validate, and finally approve its own work.

The architecture still looks simple because there is only one box labeled "agent." The complexity did not disappear. It moved inside the box, where it became harder to see, test, and govern.

This article is about what happens after that point.

The previous article reduced an agent to three parts: instructions, an LLM, and MCP tools. That remains the correct foundation. This article extends it with two arguments:

A single agent has a capability ceiling. For sufficiently complex, decomposable work, a team can scale beyond that ceiling through specialization, separate contexts, parallelism, and independent validation.
A team pays a coordination tax. But that tax exists in a single-agent system too, hidden in an increasingly confusing prompt and volatile context. MCP lets us externalize it into explicit, reusable collaboration infrastructure.

The practical pattern is to scale up one agent while simplicity remains an advantage, then scale out into a team when implicit coordination becomes the bottleneck.

What Is MCP? (The 30-Second Version)

The Model Context Protocol (MCP, spec 2025-11-25) is the interface layer between AI clients and external systems through tools, prompts, and resources. Throughout this series, we have treated MCP servers as the governed capability layer for enterprise AI: thin, remote, mostly stateless interfaces to internal systems and SaaS systems.

An agent is an MCP client with:

instructions describing its job and boundaries
an LLM providing reasoning and language capabilities
selected MCP tools connecting it to the outside world

An agent team does not replace that model. It composes it recursively:

a specialized agent can be exposed as an MCP tool
another agent can discover and call that tool
shared collaboration services can also be exposed through MCP

The result is not a new species of software. It is a scale-out architecture built from the same capability model.

A Single Agent Is The Right Place To Start

Teams introduce real cost. A coordinator has to decompose work, select collaborators, wait for results, resolve conflicts, and synthesize an answer. Multiple agents consume more tokens and create more failure boundaries. Shared state needs ownership, permissions, and lifecycle rules.

For many tasks, none of that is justified.

Use one agent when the work is:

short
mostly sequential
contained within one context
served by a small, coherent toolset
easy to validate directly
not valuable enough to justify additional inference and infrastructure cost

Before adding agents, scale up the single agent:

improve its instructions
use a stronger model where the economics make sense
reduce and clarify its tools
move deterministic computation into MCP servers
package repeatable workflows as prompts
give long-running work an explicit task lifecycle

This follows the same principle used throughout the series: do not add probabilistic orchestration where a deterministic capability would be better. If server code can reliably perform a calculation or enforce a known workflow, use server code. A second agent is not a substitute for a well-designed tool.

But scaling up eventually reaches diminishing returns.

The Single-Agent Ceiling

The ceiling is not a single model limit. Several pressures compound.

Context saturation
One agent must carry the plan, evidence, intermediate results, conversation history, and its own reasoning in one working context. A larger context window increases capacity, but it does not guarantee that every relevant detail receives the right attention at the right time.

Tool overload
A generalist agent needs more tools. The tool-design article showed that tool performance can fall off a cliff as the choice set grows. Dividing work across specialists lets each agent see a smaller, more relevant capability surface.

Role collision
Planning, execution, criticism, approval, and final communication require different behavior. Encoding all of them into one instruction set creates conflicting objectives: move quickly but verify everything, explore alternatives but remain concise, propose a solution but distrust it.

Sequential throughput
One agent explores one trajectory at a time. When a problem contains several independent research paths or checks, a team can work on them concurrently using separate contexts.

Path dependence
An early mistaken assumption shapes later reasoning. The same agent reviewing its own work sees the conclusion through the context that produced it.

Weak self-verification
Telling an agent to "double-check your answer" is useful, but it does not create an independent reviewer, an authoritative state boundary, or a requirement that validation succeed before completion.

These pressures explain why adding more instructions can make an agent worse. Each new rule may be sensible in isolation, but the complete prompt becomes a crowded control plane for responsibilities that should have explicit owners and interfaces.

The Agent Scaling Dilemma

Database architecture has long faced a similar decision: scale up one system while that remains economical, then scale out when the growing system crosses a complexity or capacity boundary. The same curve applies to agents.

The diagram is a conceptual architecture curve, not benchmark data.

The single-agent line begins lower. One prompt, one loop, and one deployment are hard to beat for a prototype. During initial growth, better instructions, a better model, and a curated toolset extend its useful range.

The team line begins higher because collaboration infrastructure must exist before the team can work well. Agents need contracts, shared artifacts, state ownership, memory boundaries, validation, status tracking, and governance.

The crossover arrives when keeping those concerns implicit inside one agent becomes more expensive and less reliable than representing them explicitly. Beyond that point, the team architecture can add specialized capacity without continually expanding one agent's prompt, context, and tool surface.

This is the important distinction:

A team is not cheaper because coordination disappears. A team becomes scalable because coordination is made explicit, reusable, and governable.

Teams Scale Reasoning Out

Scale-out is useful when a problem can be divided into bounded responsibilities.

Different agents can have:

different instructions
different tools and permissions
different working contexts
different artifact ownership
different validation responsibilities
even different models chosen for the economics of their role

A research agent can explore one branch while another explores a second branch. A finance agent can receive only finance tools and policy. A reviewer can inspect an artifact without inheriting every assumption made by its author. A coordinator can work from concise findings instead of carrying every raw document in its own context.

There is growing evidence that this helps for the right task shapes. Anthropic reported that its multi-agent research system outperformed its single-agent baseline by 90.2% on an internal research evaluation, especially for breadth-first queries with independent directions. The same report also says the multi-agent system used about 15 times as many tokens as ordinary chat and was a poor fit for work with tightly coupled dependencies. (Anthropic engineering)

A controlled study across 180 agent configurations found a similar boundary: centralized coordination improved performance substantially on parallelizable tasks in its benchmarks, while every tested multi-agent topology degraded performance on sequential reasoning tasks. (Towards a Science of Scaling Agent Systems)

The conclusion is not that more agents are always better. It is that teams raise the ceiling when the work benefits from partitioning, parallel exploration, specialization, or independent checking.

The Coordination Tax Exists Either Way

Multi-agent systems are often criticized for adding coordination overhead. That criticism is correct but incomplete.

A complex single agent also coordinates. It decides which responsibility it is performing, tracks intermediate state, remembers failed approaches, checks its evidence, manages its tool results, and decides whether its own answer is valid. The difference is that all of this happens implicitly inside probabilistic reasoning.

The choice is therefore not:

coordination
no coordination

It is:

implicit coordination inside one prompt and context
explicit coordination through contracts, artifacts, memory, tasks, and ownership

Concern	Hidden Inside One Agent	Externalized In A Team
Responsibility	Sections of one large instruction prompt	Small role-specific instructions and agent contracts
Delegation	Internal reasoning about what to do next	Typed calls to selected collaborators
Working state	Conversation and model context	Named, durable artifacts with explicit owners
Prior attempts	Whatever remains salient in context	Append-only hypothesis or decision logs
Validation	The author checks its own conclusion	An independently instructed reviewer or referee
Mutation	Any reasoning step may silently revise the working answer	A designated writer validates and records changes
Long-running work	The caller waits or invents hidden state	Task identifiers, status, retry, cancellation, and results
Learning	More instructions added to the prompt	Scoped, searchable team memory
Failure analysis	Reconstruct the model's reasoning after the fact	Inspect calls, files, logs, status, and validation results

Making coordination explicit does not make it free. It makes it addressable.

The engineering checklist becomes concrete:

How is work decomposed?
Which agent owns each responsibility?
What is the input and output contract?
Where do intermediate artifacts live?
Who may update authoritative state?
How are duplicate work and conflicting changes prevented?
How are progress, retries, timeouts, and cancellation represented?
Who resolves contradictory findings?
What information may be remembered, for how long, and for whom?
How do identity, authorization, and audit context propagate?
How are token, latency, and infrastructure budgets enforced?

Those questions are difficult to answer when their only implementation is prose buried in one system prompt. Once they are implemented as collaboration services, the investment can support many teams rather than one carefully tuned agent.

The Three MCP Services An Agent Team Needs

In an MCP-native platform, three shared services cover most of the collaboration substrate:

                            ┌─────────────────────┐
                            │  coordinator agent  │
                            └──────────┬──────────┘
                                       │
                 ┌─────────────────────┼─────────────────────┐
                 │                     │                     │
                 ▼                     ▼                     ▼
          ┌─────────────┐       ┌─────────────┐       ┌─────────────────┐
          │  team-mcp   │       │   team-fs   │       │   team-memory   │
          │ delegation  │       │  artifacts  │       │ semantic recall │
          └──────┬──────┘       └──────┬──────┘       └────────┬────────┘
                 │                     │                       │
          specialized agents      shared case files      reusable lessons
                 │                     │                       │
                 └─────────────────────┼───────────────────────┘
                                       │
                                       ▼
                             domain MCP servers
                                       │
                                       ▼
                             systems of record

The names matter less than the separation of responsibilities.

`team-mcp`: The Coordination Plane

team-mcp exposes selected agents as MCP tools.

A coordinator might see:

finance_checker
policy_reviewer
research_agent
independent_validator

Each agent has the same things a good MCP tool has:

a clear name and description
a typed input contract
a structured output contract
explicit error behavior
task or artifact references where appropriate
scoped permissions
observable invocation boundaries

From the caller's perspective, delegation remains ordinary tool use. The coordinator reads descriptions, selects the right capability, supplies the required context or artifact references, and reads the result.

This does not mean every agent should see every other agent. That would recreate the tool-overload problem at the team layer. team-mcp should expose a small, role-appropriate roster to each caller. A lead agent may see administrative specialists; an evidence agent may see domain specialists; a writer may see only validated artifacts and the validator.

The team topology becomes a governed capability graph rather than an unrestricted group chat.

`team-fs`: The Artifact Plane

Agents often collaborate on durable work products and documents:

plans
evidence
reports
spreadsheets
generated code
structured state
append-only decision logs
validation results

Passing all of that through tool-call payloads wastes context and creates brittle handoff chains. team-fs gives the team a shared workspace. Agents exchange stable file or artifact references instead of repeatedly copying entire documents through the coordinator.

The filesystem should not be an ungoverned scratch directory. Useful conventions include:

stable file names or artifact URIs
an explicit owner for authoritative files
single-writer rules where consistency matters
append-only logs for decisions and rejected changes
versioning or optimistic concurrency for shared updates
access control by team, task, and role
cleanup and retention policies

The names of shared files effectively become a small business API for the team. evidence.md, current_plan.json, contradictions.log, and final_report.md communicate purpose more reliably than a collection of anonymous text blobs in model context.

`team-memory`: The Semantic Context Plane

Shared memory exists for information that should remain useful beyond one call or artifact:

operating lessons
recurring failure patterns
durable team conventions
previously validated approaches
task history that is useful for later retrieval

Semantic search lets an agent retrieve the small subset relevant to its present work rather than loading the team's entire history.

But memory is not a system of record. It may be stale, incomplete, contaminated, or retrieved in the wrong context. A robust memory service needs:

scope: agent, user, task, or team
provenance: who or what wrote the memory
timestamps and retention
permissions
confidence or validation status
deletion and correction

Current task facts should normally remain in task state, artifacts, or systems of record. Memory can recall that "board updates require supporting evidence." It should not be treated as authoritative evidence that "Alice selected the science-fiction movie."

That separation prevents semantic memory from quietly becoming a shadow database.

Systems Of Record And Tasks Still Matter

The three collaboration services do not replace operational systems.

Customer state belongs in the CRM. Work status may belong in a ticket or case system. Approvals belong in an approval system. Financial truth belongs in financial systems. Agents should access those sources through governed domain MCP servers rather than recreate them in team memory or loose files.

Tasks cut across the architecture. When delegated work becomes long-running, retryable, approval-gated, or independently auditable, the caller should receive a task handle rather than block on a large hidden conversation. The experimental Tasks capability introduced in the MCP 2025-11-25 specification provides one protocol model for durable state, polling, deferred results, and cancellation. (MCP Tasks)

The durable task says what is happening. team-fs holds the work products. team-memory holds reusable learning. Systems of record remain authoritative for the business.

A Team Of Agents Can Look Like An MCP Server

With that infrastructure in place, the recursive model from the previous article becomes practical:

Inside one governed MCP-native platform, a team of agents can be presented as an MCP server that exposes selected specialized agents as tools.

The important word is presented. The team is not literally only a server process. Behind that surface are agent runtimes, shared files, memory, tasks, domain systems, permissions, and observability. MCP gives all of them a consistent interface boundary.

This is why a dedicated agent-to-agent protocol may be unnecessary inside one MCP-native platform:

discovery already exists through tool descriptions
invocation already exists through tool calls
structured results and output schemas already exist
tasks can represent deferred work
resources and artifact links can carry context
security, testing, and observability can remain on the MCP surface

Cross-organization or independently administered agent ecosystems may have broader interoperability requirements. But inside one governed platform, the burden of proof should be on adding a second protocol, not on reusing the capability layer already in place.

A Puzzle That Exposes The Ceiling

Logic-grid puzzles provide a useful demonstration because they have exact answers and can grow in controlled increments.

Consider a movie-night puzzle. A small version has three people and three categories: which movie each person watched, which snack they ate, and where they sat. A harder version adds people, streaming platforms, arrival times, drinks, more clues, and longer chains of dependencies.

The canonical test ladder grows from:

Difficulty	People	Categories	What Changes
Very easy	3	3	Basic smoke test
Easy	4	4	More assignments and platform ownership
Medium	4	5	Ordered arrival constraints
Hard	5	5	A fifth person and longer evidence chains
Very hard	5	6	Full stress case with cross-category dependencies

The standalone agent receives one long instruction set covering intake, clue parsing, hypothesis testing, grid updates, validation, and final explanation. It can solve the smaller puzzles.

As the puzzle grows, a more dangerous failure appears than merely returning "I don't know." The agent can announce that it solved the puzzle while its answer violates a clue, assigns a value twice, or leaves an unsupported conclusion. Asking the same agent to review its work does not reliably fix the problem because the proposed solution and the assumptions that produced it share the same context.

The team uses the same underlying model capabilities, but distributes responsibility:

a Lead Investigator chooses the next bounded question and coordinates work
a Hypothesis Log records proposed, accepted, rejected, and repeated ideas
an Evidence Clerk checks deductions against the numbered clues
a Board Manager is the only agent allowed to update the authoritative grid
a Referee independently checks every clue, uniqueness, completeness, and evidentiary support
a Final Report Writer produces the answer only after the Referee validates it

The collaboration infrastructure is not incidental to the result.

Through team-mcp, the Lead Investigator calls these responsibilities as tools. Through team-fs, the team shares explicit case artifacts:

clues.md
constraints.md
puzzle_grid.json
hypotheses.log
update_requests.log
contradictions.log
final_solution.md
final_proof.md

Through team-memory, the team can retain reusable operating lessons such as "every board update requires clue IDs or an evidence summary." It deliberately does not store current grid assignments or temporary guesses as semantic memory. Current case facts belong in the case files.

The single agent is asked to remember, reason, mutate state, and validate itself. The team turns those responsibilities into separate contracts with an inspectable chain of evidence.

That is the capability crossover in concrete form. The individual agents are not magically more intelligent than the standalone agent. The system becomes more capable because it adds bounded contexts, explicit ownership, controlled mutation, durable state, and independent validation.

A later article will cover the full experiment: the team design, prompts, puzzle ladder, failure traces, repeated evaluations, and the cost of achieving the higher completion rate.

Measure The Crossover, Do Not Assume It

A multi-agent architecture should earn its overhead against a strong single-agent baseline.

For the puzzle experiment, a run succeeds only when:

the final answer exactly matches the gold solution
every clue is satisfied
no category contains duplicate assignments
no person has a missing assignment

The most revealing measurement distinguishes claimed success from actual success:

false_completion = declared_success && !actual_success

That captures the case where an agent crosses its capability ceiling without recognizing that it crossed it.

A serious comparison should also record:

exact task-completion rate
unsupported conclusions
validation failures
retries and rejected updates
wall-clock latency
token usage
model and infrastructure cost
tool calls and delegated agent calls

Run each puzzle multiple times because model behavior is probabilistic. Compare both quality and economics. A team that improves completion by one percentage point at fifteen times the cost may be a poor design for a low-value task. The same cost may be trivial for a high-value investigation that the single agent cannot complete reliably.

The purpose of the crossover is to make the decision measurable, not ideological.

When To Scale Out

An agent team is a strong candidate when the work has several of these properties:

independent branches can run in parallel
the evidence exceeds one useful working context
specialists need different tools, policies, or permissions
intermediate artifacts have clear owners
independent validation materially improves correctness
the work is long-running, retryable, or resumable
different models can be used economically for different roles
the value of a correct result justifies higher inference cost

Keep one agent when:

the task is short and sequential
every step depends tightly on the previous step
all participants would need the same complete context
coordination would cost more than the work
one agent already meets the target completion rate

And use neither architecture for work that should be deterministic. If a known algorithm, database query, workflow engine, constraint solver, or ordinary service can produce the answer reliably, put that capability behind an MCP server and let agents call it.

A practical decision sequence is:

Can deterministic software do the work? Build a tool or workflow.
Can one agent do it reliably with a bounded context and toolset? Use one agent.
Does the remaining work benefit from specialization, parallelism, durable collaboration, or independent verification? Build a team.

The goal is not to maximize agent count. It is to choose the smallest architecture that reliably completes the task.

Security And Governance Multiply With The Team

Exposing an agent as a tool does not weaken the security requirements from earlier articles. It adds new places to apply them.

For every agent capability, ask:

who may call it?
which tools and files may it access?
whose identity does it act on behalf of?
which other agents may it delegate to?
which outputs may be written to shared state?
what prevents recursive delegation from running without a bound?
how are untrusted instructions and artifacts contained?
what is logged for reconstruction and audit?

Shared collaboration services need their own controls:

team-mcp enforces the allowed collaboration graph and delegation budgets
team-fs enforces role-based paths, ownership, and mutation rules
team-memory enforces scope, provenance, retention, and deletion
domain MCP servers continue to enforce business authorization closest to the data

Testing also moves up one level. Test each agent contract independently, then test the complete team on real scenarios. A specialist that performs well in isolation can still fail as a team dependency if its description is ambiguous, its output is difficult to consume, or the coordinator calls it at the wrong time.

Multi-agent systems do not escape MCP discipline. They multiply the reasons to apply it well.

Supervisors, Sub-Agents, and Swarms

Once the architecture is explicit, much of the market vocabulary becomes easier to place.

Supervisor/Router agent
An agent instructed to coordinate, review, or route work, with selected collaborators exposed as tools.

Sub-agent
A specialized agent called by another agent through a bounded capability contract.

Swarm
A label for a larger or more decentralized group of agents, often with overlapping capabilities and looser routing rules to execute in parallel.

The patterns may be useful. The names do not remove the need to define responsibilities, state, validation, permissions, and failure behavior. A vague swarm with no ownership is not more scalable than an overloaded single agent. It has merely distributed the ambiguity.

Agent Teams Are Also Preparing For Human Teammates

Tasks, artifacts, evidence, ownership, and validation make an agent team legible to humans as a useful side effect. A person can inspect a case file, approve a task, correct an artifact, or review a validation result without needing access to an agent's hidden prompt state.

That does not make this a human-agent collaboration article. The next step requires its own treatment:

which decisions should remain human
where human domain expertise outperforms model reasoning
how agents should escalate uncertainty
how authority and accountability work in a mixed team
how interfaces help people supervise without becoming coordination bottlenecks

A later article will extend this architecture into hybrid teams in which human experts and AI agents contribute different capabilities through the same tasks, artifacts, and governed interfaces.

The Team Model In One View

Carry these principles forward:

Start with one agent.
It has the lowest coordination cost and is the right default for bounded work.
Do not confuse a simple diagram with a simple system.
Coordination hidden inside one prompt and context is still coordination.
Scale up before scaling out.
Improve instructions, models, tools, workflows, and deterministic services first.
Teams raise the ceiling for the right task shapes.
Specialization, parallel contexts, explicit ownership, and independent validation enable harder work.
The coordination tax exists either way.
Teams make it explicit so it can be tested, observed, governed, and reused.
Use team-mcp for delegation.
Expose a small, role-appropriate set of specialized agents as typed capabilities.
Use team-fs for working artifacts.
Give important state stable names, clear owners, mutation rules, and history.
Use team-memory for reusable learning.
Keep semantic recall separate from authoritative task and business state.
Measure the crossover.
Compare completion, false confidence, latency, tokens, and cost against a strong single-agent baseline.
Use the smallest architecture that completes the task reliably.
Sometimes that is deterministic software, sometimes one agent, and sometimes a governed team.

The scalable idea is not "more agents." It is the separation of responsibilities and state behind explicit interfaces.

Scale up while simplicity wins. Scale out when coordination becomes the bottleneck.

Continue the Series

This article extended the capability-first model from one agent to a governed team. The rest of the series provides the layers underneath it and the next steps beyond it.

Need the single-agent foundation? Read Build MCP Servers Before Agents.
Need better capability boundaries? Read Tool Design and Prompts and Resources.
Need release and security discipline? Read Testing MCP Servers and Security for MCP Servers.
Need governed flexibility for the long tail requests? Read Code Mode for MCP.
Want the evidence behind the crossover? A future article will present the complete grid-puzzle team, evaluation ladder, and single-agent comparison.
What follows agent-only teams? A later article will cover hybrid teams of human experts and AI agents.

Build MCP Servers Before Agents: The Enterprise Shift AI Teams Need

Guy — Thu, 23 Jul 2026 17:27:54 +0000

A team builds a support agent in an agent framework. Another team builds a finance agent. A third builds a sales operations agent. Each demo is impressive. Each agent has a prompt, a model, some memory, a few tools, and a nice workflow graph. Then the enterprise reality arrives.

The support agent has its own private Salesforce connector. The finance agent has a different connector to the same customer data. The sales operations agent defines "North America" differently from both of them. Each team handles authentication a little differently. Each team logs different things. Each team wraps the same APIs again. Six months later, the company does not have an agent platform. It has a convoy of bespoke machines held together by prompts, glue code, and optimism.

That is the wrong foundation.

The current market narrative tells teams to start with the agent: pick a framework, write the instructions, choose a model, add tools, and ship. That works for demos. It is backward for enterprises.

The reusable asset in enterprise AI is usually not the agent. It is the MCP server (AKA connector).

Agents should come later, as lightweight clients built from three parts:

instructions: what the agent should do, what it should not do, and how it should behave — much like instructions you would give a human employee. These are defined by business people and updated frequently as the business learns.
LLM: the reasoning and language component, usually from a major vendor such as OpenAI or Anthropic.
tools: capabilities exposed through MCP servers, with all the design, security, testing, and governance work we covered in the earlier articles.

That is the shift this article is about. And to make it concrete, this time we will not stop at the diagram — we will build a complete, running agent and show that the "hard part" was never the agent at all.

What Is MCP? (The 30-Second Version)

The Model Context Protocol (MCP, spec 2025-11-25) is the interface layer between AI clients and external systems through tools, prompts, and resources. In enterprise deployments, MCP servers should be thin, remote, mostly stateless interface layers over internal systems and SaaS platforms. They are the AI-facing equivalent of web servers and mobile backends.

That makes the next architectural point straightforward:

MCP servers are the governed capability layer
agents are MCP clients

An Agent Is Simpler Than Many Frameworks Make It Look

Before talking about architecture, it helps to simplify the mental model.

At its core, an enterprise agent usually needs only three things:

Instructions — the job description. What this agent is for, what success looks like, what boundaries it must respect, what tone to use, when to ask for clarification, when to stop, and when to escalate.
LLM — the reasoning and language engine. It interprets the request, decides what to do next, calls tools, reads results, and produces the answer.
Tools — the actions and data-access surfaces. In a serious enterprise design, these come from MCP servers rather than as private helper functions bundled inside a single agent.

That leads to a simple loop:

read instructions
read user request
decide whether to answer directly or call a tool
if needed, call a tool on an MCP server
read the tool result
repeat until the request is satisfied or the task fails safely

That is the basic ReAct loop. For many real enterprise cases, that is the right default model. The agent does not need a giant graph, a maze of private adapters, or a pile of framework-specific abstractions to be useful.

The claim that "an agent is simple" is easy to make and easy to doubt. So let us prove it in code.

The Simplicity Is Real: An Agent in Forty Lines

Here is a complete agent written in Rust with the PMCP SDK. Don't worry if you're not fluent in Rust — the comments walk through every important line, and the shape is what matters. Watch how directly the three parts above map onto three pieces of code.

// -- Dependencies --
// pmcp-agent: the deploy-anywhere agent decision loop for the PMCP SDK.
// An agent here is a loop over THREE swappable "seams":
//   CompletionSource  -> the LLM (produce the next model turn)
//   ToolInvoker       -> the tools (run a tool call, return its result)
//   ConversationStore -> the memory (persist and replay the conversation)
use pmcp_agent::{
    AgentEngine, ResolvedAgentConfig, InMemoryStore,
    CompletionSource, ToolInvoker, ToolCall, ToolCallResult, RunOutcome,
};
use async_trait::async_trait;
use serde_json::json;

// -- Part 1: INSTRUCTIONS --
// The "job description." Plain text a business person can own and edit.
// The last two numbers are guardrails, not intelligence: a token budget
// and a hard cap on how many tool-calling turns the loop may take before
// it stops safely. That cap is why a thin agent cannot run away.
fn instructions() -> ResolvedAgentConfig {
    ResolvedAgentConfig::new(
        "You are a concise support assistant. Look up orders when asked. \
         If a request is outside orders, say so and stop.",
        "claude-sonnet-4-5", // the model name; the LLM itself is the next seam
        /* max_tokens     */ 100_000,
        /* max_iterations */ 5,
    )
}

// -- Part 3: TOOLS --
// The tools seam. This is the ENTIRE surface where an agent touches the
// outside world. In production you would hand this seam a `ClientToolInvoker`
// that forwards each call to a governed MCP server (see the next section).
// Here we stub one tool inline so the example runs with no network:
struct OrdersTools;

#[async_trait]
impl ToolInvoker for OrdersTools {
    async fn invoke(&self, call: ToolCall) -> ToolCallResult {
        // `call.name` is the tool the LLM chose; `call.arguments` is its JSON input.
        // A real invoker dispatches to an MCP server and returns its result verbatim.
        ToolCallResult::ok(call.id, json!({ "status": "shipped", "eta": "2 days" }))
    }
}

#[tokio::main]
async fn main() {
    // -- Assemble the agent from its three parts + memory --
    // Part 2 (the LLM / CompletionSource) is passed in as `llm`. In a test, it is
    // a mock; in production it is one line — see "Swapping the Brain" below.
    let llm = /* a CompletionSource: mock, Anthropic, or any OpenAI-compatible endpoint */;

    let engine = AgentEngine::new(
        llm,                  // Part 2: the brain
        OrdersTools,          // Part 3: the tools (the MCP boundary)
        InMemoryStore::new(), // memory
        instructions(),       // Part 1: the job description
    );

    // -- Run it. This IS the ReAct loop. --
    // A "run" is a conversation identified by `run_id`; its first turn is the
    // user's request, seeded into the store. (In production the AgentServer
    // adapter seeds it for you from the incoming MCP tool call — see below.)
    // `run` then interprets that request, lets the model choose a tool, dispatches
    // it through the invoker, feeds the result back, and repeats until the model
    // ends the turn or a guardrail trips. You did not write that loop.
    let run_id = "order-status-4815";
    match engine.run(run_id).await {
        RunOutcome::Completed { result } => println!("done: {result:?}"),
        RunOutcome::LimitReached        => println!("hit the iteration guardrail — stopped safely"),
        other                           => println!("stopped: {other:?}"),
    }
}

Now look at what you did not write:

No ReAct loop. AgentEngine::run is the loop.
No tool-dispatch plumbing, no argument parsing, no result threading.
No retry, budget, or turn-limit bookkeeping — the config and the engine own that.
No framework-specific graph, node, or edge abstractions.

What remains is exactly the three-part model: a paragraph of instructions, a model name, and a tools seam. The agent is thin on purpose. All of its leverage lives on the other side of that ToolInvoker.

If you want to scaffold this rather than type it, the CLI generates the whole thing and runs it offline against a scripted model, so you can see the loop before wiring a real LLM or any MCP servers:

cargo pmcp agent new support-agent   # scaffold instructions + the three seams
cargo pmcp agent dev --source fixed   # run the loop offline, no API keys, no network

Where The Tools Come From

The single most important line in that example is the one that looks least important:

OrdersTools,          // Part 3: the tools (the MCP boundary)

The ToolInvoker seam is the entire boundary between the agent and the outside world. Whatever you plug in there decides where the agent's capabilities live. That is the whole architectural argument of this series, expressed as one type parameter. Once you understand that an AI agent is simply an MCP client, everything else falls into place, and all the investment in building MCP server connectors returns with huge dividends of security, simplicity, and scale.

Plug in a struct full of private helper functions, and you have rebuilt the Mad Max convoy: this agent's Salesforce logic, this agent's "North America," this agent's auth. Plug in an invoker that forwards each call to a governed MCP server, and every capability comes from the shared platform layer — designed, secured, tested, and observed once.

The PMCP SDK ships exactly that production invoker. Instead of OrdersTools, a real agent uses ClientToolInvoker, which dispatches each tool call to an MCP server and returns its result — including long-running work modeled as tasks:

// Production: the tools seam forwards to a governed MCP server.
// `connector` is an MCP client pointed at your orders/CRM/warehouse server.
// The agent gains no private connector code; it borrows the platform's.
let tools = ClientToolInvoker::new(connector, /* max_poll_secs */ 30);

let engine = AgentEngine::new(llm, tools, InMemoryStore::new(), instructions());

Nothing else in the agent changes. The instructions are the same, the loop is the same, the model is the same. The only thing that changed is where the capabilities come from — and that is the only thing that should decide whether your fleet of agents is governable.

This is the capability-first model as a code diff: swapping a private-tools invoker for an MCP-backed one is a one-line change, and it is the difference between a rogue demo and a platform citizen.

Swapping The Brain Is One Line

The other seam that proves how thin an agent should be is the LLM. Because the model sits behind CompletionSource, changing it never touches the loop, the tools, or the instructions:

use pmcp_agent::sources::{AnthropicSource, OpenAiCompatSource, SecretString};

let key = SecretString::new(std::env::var("LLM_API_KEY")?);

// A local model for development (any OpenAI-compatible endpoint, e.g. Ollama):
let llm = OpenAiCompatSource::new("http://localhost:11434/v1", "llama3.2", key)?;

// A frontier model for production — same engine, same tools, same instructions:
let llm = AnthropicSource::new("https://api.anthropic.com", "claude-sonnet-4-5", key)?;

Both satisfy the same seam, so both drop straight into AgentEngine::new. You can develop offline against a scripted or local model and promote to a hosted one for production without rewriting the agent. When the model is this easy to swap, you stop entangling business logic with a particular vendor — and you keep the reusable work where it belongs, on the MCP servers behind the tools seam.

The Capability-First Agent Model

Step back from the code and the architecture reads as one principle: capabilities are shared infrastructure; agents are thin clients.

Capabilities live on MCP servers. Tools, prompts, resources, and governed code-mode surfaces belong in the shared platform layer.
Agents stay thin. An agent is mostly instructions, an LLM, and a selected set of those capabilities — the forty lines above.
Specialized agents can themselves become capabilities. If a later design exposes a specialist agent through MCP, another agent can call it like any other tool. (The PMCP AgentServer adapter does exactly this — it exposes the same loop as a normal MCP server. That recursive step is the subject of the next article.)
Shared state stays explicit. Tasks, artifacts, and systems of record are better homes for shared state than hidden prompt state or private connector logic.

This article is mainly about the first two points. The next article extends the same model to teams.

The Mistake: Treating Tools As A By-Product Of The Agent

The common development pattern today looks like this:

choose an agent framework
define the agent's instructions
pick a model
write some tools for that agent
wire them together

That feels natural because the agent is the visible thing. But architecturally it is upside down. When tools are created as part of one agent, the problems appear quickly:

The same backend gets wrapped multiple times by different teams.
Business semantics drift between agents.
Authentication and authorization logic get duplicated.
Observability is fragmented across agent codebases.
Testing quality varies by team.
Security posture depends on whoever wrote the last connector.
No one knows which tool implementation is the canonical one.

This is how organizations end up with rogue agents: individually clever, collectively ungovernable. The deeper issue is that the tool layer is being treated as disposable application code when it should be treated as shared platform infrastructure. In the code, that mistake is a single choice — what you pass to the ToolInvoker seam.

Build The Roads Before The Cars

The cleanest analogy I know is transportation.

The agent-first model looks like a Mad Max convoy. Every team builds its own vehicle from scratch. The wheels are different. The fuel systems are different. The controls are different. Each vehicle may be brilliant in isolation, but the system as a whole is chaotic and expensive.

The MCP-first model looks like a city that built the roads first. The roads are paved, the lanes are marked, the traffic lights are standardized, the signs are shared, maintenance is organized, and safety rules are explicit. Once that infrastructure exists, many kinds of efficient vehicles that everybody can drive can use it.

That is what MCP servers are for enterprise AI. They are the roads. They give the organization shared interfaces to data systems, shared business definitions, shared authentication and authorization boundaries, shared observability, shared testing discipline, and shared governance. Agents are then the vehicles that move on top of that infrastructure — some simple, some specialized, some aggressively optimized, but all benefiting from the same road system.

The Enterprise Shift

The shift I want enterprise teams to make is this:

Stop asking "what tools should this agent have?" before you ask "what MCP servers should this organization have?"

Those are not the same question. The first is local and short-term. The second is architectural. An enterprise-first sequence looks more like this:

Identify the core data systems and business domains.
Build MCP servers for those systems using the best practices from the earlier articles.
Expose outcome-oriented tools, prompts, resources, and, where appropriate, code mode.
Apply security, testing, and governance at the server layer.
Only then define agents as instruction sets plus models plus selected MCP capabilities.

This order matters because the MCP server layer is where reuse happens. It is not a call for a big-bang platform rewrite — start with one business domain, publish one or two well-governed MCP servers, and build a small number of agents on top. The important shift is not "wait until the platform is perfect." It is "put reusable capability work in the platform layer from the beginning."

The agent-first model can look fine when there are three demos in one lab. It breaks when the organization starts talking about dozens, then hundreds, then thousands of agents. At that scale you are no longer solving a prompt problem; you are solving a platform problem:

Question	Agent-First Model	MCP-First Model
Where do tools live?	Inside each agent codebase	In shared MCP servers
Who defines business semantics?	Each team, repeatedly	Once per governed server surface
How is auth handled?	Per agent, inconsistently	At the server layer, consistently
How do agents share capabilities?	Copy code or rebuild connectors	Reuse the same server
How do you test the interface?	Ad hoc by team	Server-level test gates
How do you observe usage?	Fragmented logs across agents	Centralized capability-layer telemetry
How do you swap models or instructions?	Often entangled with tools	Independent from tools (one line)
How do you span systems?	Custom orchestration in every agent	Compose across MCP servers

This is why the current market emphasis on "building agents" can be misleading. It draws attention to the system's visible tip — the forty lines — and away from the part that determines whether the whole thing can scale.

Internal MCP Servers First, Vendor MCP Servers Second, Agents Third

Enterprise AI rarely lives in one system. A real company has internal databases, internal APIs, data warehouses, identity systems, document stores, ticketing systems, and SaaS platforms. At the same time, major vendors are all pushing their own agent stories — Salesforce, Azure, AWS, ServiceNow, Google — and each wants you to build agents inside its orbit.

That is understandable from the vendor's perspective, but it is not the right control point for the enterprise. The better model is:

Build MCP servers around your internal systems.
Adopt MCP servers from external SaaS providers when they exist and are good enough.
Build your agents as MCP clients that can call both.

A customer-success agent might need internal customer master data, a SaaS CRM, a support platform, and internal revenue and usage data. No single vendor should own that whole orchestration surface by default. The agent should sit above those systems — behind that one ToolInvoker seam — not be trapped inside one of them.

The Business Analyst Matters More, Not Less

This shift does not remove the role of the business analyst. It makes that role more important. If agents are lightweight clients on top of MCP servers, then the quality of those servers matters even more. Someone still has to decide:

Which business outcomes deserve first-class tools
Which workflows should be packaged as prompts
Which resources should be injected as governed context
Which definitions of quarter, region, customer status, and policy are canonical
Which actions should be impossible rather than merely discouraged

If every team embeds those choices inside its own agent prompt, the organization gets semantic drift. If those choices are encoded once in the MCP layer, the organization gets consistency. The security article made exactly this point for access control and aggregation semantics; the same idea applies here at the full agent-platform level.

The ReAct Loop Is Simple. The Platform Under It Is Not.

It is important not to confuse two statements:

The runtime loop of a good agent can be simple — you saw it fit in forty lines.
The infrastructure that makes simple agents safe and scalable is not simple.

That is why the earlier articles matter. The loop can stay small precisely because the MCP layer carries the real engineering burden:

tool design makes the actions understandable
prompts and resources package repeatable workflows and governed context
testing makes the capability surface stable
code mode covers the long tail without exploding tool count
security makes the whole thing enterprise-safe

If that foundation is missing, teams compensate by stuffing more policy, more special cases, and more connector logic into the agent itself — until the forty lines become four thousand. The result is the illusion of a smart agent built atop a weak platform.

The Core Message

The market talks constantly about building agents. Agents are visible. They demo well. Vendors can package them. Frameworks can make them feel like the main event. But for enterprise AI, the real leverage is lower in the stack, and the agent itself — as the code showed — is genuinely small.

Carry this model forward:

An agent is not magic. It is instructions, an LLM, and tools — three seams and a loop you don't have to write.
The right default loop is simple. Read instructions, read the request, call the right tool, inspect the result, repeat until done or failed safely.
Tools should not be private by-products of each agent. They belong on MCP servers, reached through the one seam that defines the agent's whole outside world.
MCP servers are the reusable foundation. They carry semantics, security, testing, observability, and governance.
The scalable model is cross-system. Internal MCP servers plus external SaaS MCP servers give agents a governed way to work across the real enterprise data estate.
The enterprise shift is architectural. Stop starting with "What tools should this agent have?" Start with "What MCP servers should this organization have?"

If an organization wants to run a handful of demos, it can build agent-first and survive the mess. If it wants to run AI seriously across the company, it should build MCP servers first, treat them as shared platform infrastructure, encode business semantics and governance there, and compose thin agents on top — incrementally. One good server in one domain plus one or two thin agents on top of it is already a better enterprise foundation than five siloed agents with five private connectors.

Build the roads before the cars.

DEV Community: AWS Heroes

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK - Part 1 Introduction to the series and services

Introduction to the series

Introduction to the services used throughout the series

Building AI Agents with Embabel, Spring AI and Amazon Bedrock AgentCore - Part 3 Develop Agents as web application with Embabel

Introduction

Implement AI agents as a web application with Embabel

Conclusion

EventBridge Scheduler DLQ'd My Successful Invokes

The Actual Problem

About That 30 Seconds

The Fix

A Few Things to Watch For

One More Thing About Retries

If You're Hitting This

# Blog #9: I Gave My Minecraft Mod a Serverless Brain — And It Decides How Many Allies to Summon Based on How Screwed You Are

I Gave My Minecraft Mod a Serverless Brain — And It Decides How Many Allies to Summon Based on How Screwed You Are

The Items That Make It Work

Tome of API Key

Cloud API Requester (Block)

Cloud-Enabled Weapons

The Architecture

The Async Pattern: Fire-and-Forget + Polling

The AI Agent: Strands + Ollama Cloud

The CDK Stack

Local Development: The Mock Server

Cloud Concepts Taught Through Gameplay

What I Learned

What's Next

I Built a Cloud Caravan Merchants Progression System as a cloud Marketplace

3 Merchants — 3 Cloud Layers

The Spawn Egg — Full Caravan Unit

The Donkey Chest Bug — A War Story

Elite Progression — The Loyalty Program

Elite Trades — Premium Vanilla Items

Natural Spawning — The Caravan Arrives

Never Despawn — Persistent Infrastructure

Custom Textures — Visual Identity

Rune Table Recipes — Mod Interop

Lessons Learned

Series Wrap-Up

Designing an Armor Progression That Mirrors Cloud Certification Levels

4 Armor Sets — 4 Cloud Maturity Levels

Set Bonuses — Cloud Capabilities

The Dodge System

Crafting Progression — The Certification Path

Platings (4 ingots each)

Accessories Integration — Optional Dependencies

Spell Book Slot — Cross-System Integration

Nether Star Duplication — Endgame Recipe

Lessons Learned

What's Next

I Built a Magic System Where Every Spell is a Cloud Operation

The Architecture: Spell Engine Integration

9 Tomes, 3 Tiers — Like Service Tiers

28 Spells — Cloud Operations as Game Mechanics

Rune System — API Credits

Upgrade Paths — Service Evolution

Empowered Tomes — Premium Tier

Right-Click Casting — The UX

Lessons Learned

What's Next

Monthly Amazon Location Service Updates - 2026.07

Monthly Amazon Location Service Updates - 2026.07

2026.07 Updates

Other Info

Related Articles

Monthly Amazon Location Service Updates - 2026.06

Send your events later

The surprisingly-simple-yet-not-obvious solution

Just a little bit of logic

Build your own platform

Building AI Agents with Embabel, Spring AI and Amazon Bedrock AgentCore - Part 2 Develop Agents with Embabel shell

Introduction

Implement AI agents with Embabel shell

Conclusion

Beyond The Single-Agent Ceiling: Scale Out With MCP Agent Teams

What Is MCP? (The 30-Second Version)

A Single Agent Is The Right Place To Start

The Single-Agent Ceiling

`team-mcp`: The Coordination Plane

`team-fs`: The Artifact Plane

`team-memory`: The Semantic Context Plane