The Question Left from Last Time
The previous article covered AgentTeams' design motivation: multiple agents collaborate in parallel, with a coordinator responsible for distributing tasks and collecting results.
But one detail wasn't explored: how does the coordinator know that a member agent has received and processed the task message it sent?
This question simply doesn't exist in a single-agent system. The primary agent calls a tool, the tool executes and returns the result — it's a round trip with no "message in transit" state.
But in a multi-agent system, the situation is completely different. After a message is sent, many things can happen:
- The member agent hasn't started yet, the message is sitting in the queue
- The member agent received it but is still processing
- The member agent finished processing but the result hasn't been read by the coordinator
- The member agent crashed, the message will never be processed
Without tracking these states, the coordinator doesn't know whether to wait or resend, and the entire collaboration falls apart.
This is the reason the message ACK (Acknowledgement) mechanism exists.
Conclusions First
AgentTeams' message protocol is designed around three core concepts:
| Concept | Role |
|---|---|
| SendMessage tool | The sole entry point for agents to send messages to each other |
| ACK three states | Track each message's lifecycle: pending → delivered → processed |
| TeamStatus tool | Query the current state of a team and its messages |
The three-state machine ensures the coordinator can always know "where is the message I sent right now."
1. Why Do We Need ACK?
First, a real-life analogy to explain what ACK is.
Imagine you're a project manager who assigned tasks to three developers via instant messaging. How do you know the tasks were accepted and work has started?
- If the other party merely saw your message, that's "read"
- If the other party replied to confirm, that's "acknowledged receipt" (delivered)
- If the other party completed the task and reported back, that's "processed" (processed)
If the other party neither read nor replied, you need to determine: did they not see it? Did they see it and not reply? Is their phone dead?
The ACK mechanism turns these ambiguous intermediate states into definite, queryable states.
In AgentTeams, every message has these three states:
pending
│ (message sent, waiting for member agent to receive)
▼
delivered
│ (member agent received, currently processing)
▼
processed
(member agent finished processing, result available for pickup)
2. SendMessage: The Sole Entry Point for Sending Messages
In AgentTeams, agents can only send messages to each other through the SendMessage tool.
Coordinator agent calls SendMessage:
- to: "member_agent_2" # recipient
- content: "Please refactor utils.py" # message content
- message_type: "task" # message type
SendMessage doesn't just pass a string. It does several things:
1. Writes the message to persistent storage
Messages aren't sent directly to the recipient agent's memory — they're first written to a persistent "message queue." This way, even if the member agent hasn't started yet, the message won't be lost.
2. Assigns a unique ID to the message and sets initial state
When a message is created, its state is pending, with a timestamp recording "when it was sent."
3. Returns the message ID to the coordinator
After receiving the message ID, the coordinator can use TeamStatus to query subsequent state changes for this message.
The benefit of this design: sending and processing messages are decoupled. The coordinator can move on after sending a message without waiting around.
3. The State Transition Logic of the ACK Three States
Now let's look at how each state is triggered.
pending → delivered
When the member agent's "inbox" is checked and the message is retrieved, the state changes to delivered.
After member agent starts:
1. Check its own "inbox" (retrieve unprocessed messages from the message queue)
2. Retrieve the message, immediately change state from pending to delivered
3. Begin processing the message content
The key point of this design: state change happens before "start processing", not after "processing complete."
Why? Because if we wait until processing is done to change state, the coordinator cannot distinguish between "the member hasn't received it yet" and "the member received it but is still processing" — both situations appear as "message is still pending" to the coordinator.
Moving the state change to the moment the message is "picked up" lets the coordinator know: pending = message not yet picked up; delivered = already being processed.
delivered → processed
When the member agent finishes processing the message and writes the result to the output area, the state changes to processed.
After member agent finishes processing:
1. Write processing result to the designated location (where coordinator can read it)
2. Change message state from delivered to processed
3. Notify the coordinator (or wait for coordinator to query actively)
processed state means: result is ready, coordinator can come get it.
4. How the Coordinator Knows a Message Was Processed
The coordinator has two ways to know a message was processed:
Method 1: Active polling
The coordinator periodically calls TeamStatus to query the current state of all messages:
Coordinator:
while True:
status = TeamStatus(team_id)
if all(msg.status == "processed" for msg in status.messages):
break # All messages processed, can now Collect
wait(a moment)
continue doing other things...
This is the simplest method, but has polling latency — from the last message changing to processed, to the coordinator discovering this change, there's a waiting gap.
Method 2: TeamCollect blocking wait
The TeamCollect tool can be configured with a wait strategy, keeping it waiting until all messages become processed, then returning all results at once.
Coordinator:
TeamFanout(tasks=[task1, task2, task3]) # distribute tasks
results = TeamCollect(timeout=300) # wait for all results, up to 5 minutes
This is the preferred approach — the coordinator doesn't need to write its own polling loop.
5. The Complete Lifecycle of a Message
Putting the above together, the complete journey of a message from sending to consumption looks like this:
Coordinator sends message
│
▼
[pending]
Message written to persistent storage
Coordinator continues doing other things...
│
▼
Member agent starts/activates, picks up message
[delivered]
Member agent begins executing task
│
▼
Member agent completes task, writes result
[processed]
Result waiting for coordinator to collect
│
▼
Coordinator calls TeamCollect, reads result
Message lifecycle ends
Every state transition has a clear trigger condition and a recorded timestamp, so when something goes wrong, the coordinator can use TeamStatus to precisely locate "where it got stuck."
6. Edge Cases: Lost Messages and Member Crashes
So far we've described the "everything goes well" scenario. Real systems have all kinds of edge cases — how does AgentTeams handle them?
Case 1: Member agent crashes after picking up the message
The message state stays at delivered, but the member is gone. The coordinator will see the message stuck at delivered state — after a timeout, it can choose to revert it to pending and wait for a retry.
Case 2: Message sent but member agent never starts
Message state stays at pending. If a timeout expires, the coordinator can determine this member is "unreachable" and choose to assign it to a different member, or simply return partial results.
Case 3: The coordinator itself crashes
Since message states are written to persistent storage, after the coordinator restarts it can re-read all message states and continue from where it left off. This is the core value of the persistence design.
Design Highlights
1. Messages Are Persistent, Not Memory-Dependent
Messages aren't passed via Python objects — they're written to persistent storage. This way, either party can crash and restart without losing messages.
2. Three States Instead of Two
Many simple messaging systems only have "sent/completed" states. AgentTeams adds the delivered intermediate state, letting the coordinator distinguish between "message not yet seen" and "message being processed" — greatly reducing the coordinator's need to guess.
3. State Machine Rather Than Signals
Instead of "event notification" (callbacks, signals), it uses "queryable state." The coordinator can actively query state at any time, not relying on the assumption "the other side will proactively notify me." In distributed systems, active pull is more reliable than passive push notification.
Summary
| Design Choice | Approach | Engineering Value |
|---|---|---|
| Message delivery | SendMessage tool, messages persisted | Sending and processing decoupled, supports crash recovery |
| State tracking | ACK three states (pending/delivered/processed) | Coordinator always knows where the message is |
| Result collection | TeamCollect blocking wait | Coordinator doesn't need to write its own polling loop |
| Failure handling | Timeout + state rollback | Task can be reassigned after member crash |
The next article covers TeamFanout and TeamCollect — how tasks are efficiently distributed and results are aggregated back.
About the Source Code for This Series
All analysis in this series is based on the open-source project MyCodeAgent.
AgentTeams' implementation has been removed from the stable release, but the design principles are fully documented in docs/research-archive.md and the removal plan document docs/plans/2026-07-12-lean-runtime/tasks/M2-03-remove-agent-teams.md.
git clone https://github.com/chendongqi/MyCodeAgent
cd MyCodeAgent
cp .env.example .env
uv sync
uv run python main.py
Visit PrimeSkills — a curated AI Agent and skills marketplace where every piece of content is validated against real enterprise workflows. No hype, only things that actually work.
For more practical insights and interesting products, visit my homepage
Top comments (0)