Two AI Agents Made the Right Decision. Together, They Corrupted the Same State.
The hidden concurrency problem behind multi-agent AI systems
Something is interesting about debugging intelligent systems: sometimes, every component appears to be doing exactly what it was designed to do, and the overall system still produces the wrong result. I have been thinking about this increasingly as AI agents move beyond answering questions and start performing tasks across business applications, databases, APIs and enterprise workflows. We have spent considerable time discussing whether an AI agent can reason correctly, whether it can remember previous interactions, whether it can reject outdated information and whether it can follow instructions.
But there is another problem that deserves just as much attention. What happens when two AI agents make perfectly reasonable decisions using the same information, at almost the same time? Neither agent necessarily makes a reasoning mistake. Neither misunderstands the user's request. Neither intentionally violates a business rule.
Yet, together, they can corrupt the same business state. I find this particularly interesting because it exposes a limitation in how we sometimes think about multi-agent architecture. We often focus on communication, orchestration and task delegation. We design agents to collaborate, exchange messages and divide responsibilities.
But collaboration does not automatically provide consistency. Two agents can communicate successfully, understand their individual responsibilities and still execute conflicting operations. And the problem becomes much more serious when those operations involve money, inventory, customer records, payments, manufacturing schedules or other business-critical resources.
Let's look at a simple example.
1. The problem begins when two agents see the same number
Imagine an e-commerce platform with an inventory of ten units of a particular product. A customer places an order. At the same time, another customer places an order for the same product. To handle the workload, the platform uses two AI agents.
- Agent A handles the first customer's order.
- Agent B handles the second customer's order.
Both agents connect to the same inventory service. At 10:00:01, Agent A requests the current inventory. The database responds:
Available inventory: 10
Almost simultaneously, Agent B requests the same information. The database also responds:
Available inventory: 10
Both agents now have a perfectly valid observation of the inventory at the time they read it. Agent A determines that reserving seven units is possible. Agent B determines that reserving six units is also possible. Individually, both decisions are reasonable. The problem appears when they execute.
| Event | Agent A | Agent B | Actual inventory |
|---|---|---|---|
| Initial state | Reads 10 | Reads 10 | 10 |
| Decision | Reserve 7 | Reserve 6 | 10 |
| Execution | Reserves 7 | Attempts 6 | Depends on implementation |
| Final state | Believes successful | Believes successful | Potentially inconsistent |
If the system performs independent read-and-write operations without proper concurrency protection, both agents may successfully reserve their requested quantities. The business has now promised thirteen units when it only had ten. This is not an AI reasoning failure. It is a concurrency-control failure. That distinction matters because improving the language model, adding more instructions or introducing another orchestration layer will not automatically fix it.vThe system needs a mechanism that protects the shared state.
2. Why can individually correct agents produce an incorrect outcome
One assumption I frequently question when looking at agentic architectures is that an agent's successful execution means the business operation was valid. It doesn't.
An AI agent typically works with a sequence of observations, decisions and actions. For example:
- Retrieve inventory.
- Evaluate whether sufficient stock exists.
- Decide whether to accept the reservation.
- Call the inventory API.
- Report the result.
This looks straightforward. However, there is a gap between steps two and four. The information used to make the decision may no longer represent the current state when the action is executed. I think this is one of the most important distinctions in designing reliable AI systems:
A decision can be valid when it is made and invalid when it is executed.
This is not unique to AI. Traditional distributed systems have dealt with concurrency for decades. What makes it particularly relevant to AI agents is that these systems introduce additional layers between observation and execution. An agent may:
- Retrieve information from multiple services.
- Spend time reasoning about the information.
- Ask another agent for confirmation.
- Wait for human approval.
- Generate a tool call.
- Retry an operation after a timeout.
- Execute the operation through an external API.
Every additional step creates an opportunity for the underlying state to change. The agent might still be following its original reasoning perfectly. But the environment has moved on.
3. The first hidden failure: stale reads
Let's examine the inventory example more closely. Suppose the database initially contains:
CREATE TABLE inventory (
product_id VARCHAR(50) PRIMARY KEY,
available_quantity INTEGER NOT NULL,
version INTEGER NOT NULL DEFAULT 1
);
INSERT INTO inventory
(product_id, available_quantity, version)
VALUES ('LAPTOP-001', 10, 1);
Agent A retrieves the inventory:
SELECT available_quantity
FROM inventory
WHERE product_id = 'LAPTOP-001';
Result:
available_quantity
------------------
10
Agent B executes the same query and receives the same result. Both agents now have an identical snapshot. The problem is what happens next. A naive implementation might look like this:
def reserve_inventory(product_id, quantity):
current = get_inventory(product_id)
if current >= quantity:
update_inventory(
product_id,
current - quantity
)
return "Reservation successful"
return "Insufficient inventory"
At first glance, this looks reasonable. But there is a serious problem. The read and write operations are separate. Agent A reads ten and calculates three remaining. Agent B reads ten and calculates four remaining. If Agent A writes three and Agent B subsequently writes four, the final database value becomes four.
Seven units have effectively disappeared from the accounting. This is a classic lost-update problem. The database contains a number that does not accurately represent the sequence of business operations. The code is not necessarily syntactically incorrect. The business rule is simply not protected against concurrent execution.
There are two separate issues here:
- Stale reads: agents make decisions using observations that may no longer be current.
- Lost updates: concurrent writes overwrite changes made by another operation.
They are related, but they are not identical. And neither can be reliably solved by asking agents to communicate more politely.
4. The dangerous assumption: orchestration equals coordination
Multi-agent frameworks make it increasingly easy to distribute work. One agent can retrieve customer information. Another can calculate pricing. A third can check inventory. A fourth can execute the order.
A central orchestrator can assign tasks and monitor their progress. This is useful, but it introduces a misconception. We sometimes treat an orchestrated workflow as if it were automatically a transactionally consistent workflow. Consider this architecture:
flowchart TD
A[Customer Order] --> B[AI Orchestrator]
B --> C[Inventory Agent]
B --> D[Pricing Agent]
B --> E[Payment Agent]
C --> F[(Inventory Database)]
D --> G[(Pricing Service)]
E --> H[(Payment Gateway)]
F --> I[Order Result]
G --> I
H --> I
The orchestrator knows which agent performs which task. It may know when each task starts and finishes. It may even maintain a record of every message exchanged. But does it know whether two agents attempted to modify the same inventory record? Does it know whether an inventory reservation was committed while a payment operation failed? Does it know whether a retry created a duplicate reservation?
Not necessarily. Task coordination and state coordination solve different problems. An orchestrator controls workflow progression. A database transaction or concurrency-control mechanism protects shared data. Both are important, but they should not be confused. In fact, the more independently agents operate, the more important it becomes to establish clear ownership and consistent boundaries.
5. Fixing the immediate problem: atomic database operations
The first improvement is to stop separating the inventory check from the inventory update. Instead of reading the available quantity, calculating a new value in application code and writing it back, we can make the database perform the check and update as one atomic operation. For example:
UPDATE inventory
SET available_quantity = available_quantity - 7
WHERE product_id = 'LAPTOP-001'
AND available_quantity >= 7;
The important part is the condition:
AND available_quantity >= 7
The database evaluates the condition as part of the update. If the update affects one row, the reservation succeeded. If it affects zero rows, there was insufficient inventory, or the product did not exist. We can apply the same logic to Agent B:
UPDATE inventory
SET available_quantity = available_quantity - 6
WHERE product_id = 'LAPTOP-001'
AND available_quantity >= 6;
Now consider the concurrent execution. Initially:
Available inventory = 10
Agent A attempts to reserve seven units. The database successfully updates the record:
Available inventory = 3
Agent B then attempts to reserve six units. The condition is no longer satisfied. The database updates zero rows. Agent B receives a failure rather than a false success. The result is:
| Operation | Requested | Result |
|---|---|---|
| Agent A | 7 | Successful |
| Agent B | 6 | Rejected |
| Remaining inventory | 3 |
The business invariant is preserved. This is a much stronger design because the database, rather than the AI agent, becomes the final authority on whether the reservation can happen. The agent can propose an operation. The database decides whether that operation is valid against the current state. That is an important architectural boundary.
A practical Python implementation
Using PostgreSQL and a parameterised query, the reservation logic could look like this:
def reserve_inventory(conn, product_id, quantity):
if quantity <= 0:
raise ValueError("Quantity must be positive")
with conn.transaction():
result = conn.execute(
"""
UPDATE inventory
SET available_quantity =
available_quantity - %s
WHERE product_id = %s
AND available_quantity >= %s
RETURNING available_quantity
"",
(quantity, product_id, quantity)
).fetchone()
if result is None:
return {
"success": False,
"reason": "Insufficient inventory"
}
return {
"success": True,
"remaining_quantity": result[0]
}
The database operation is atomic. The inventory cannot become negative through this operation, assuming all inventory modifications follow the same invariant. However, there is another detail worth considering. What happens if the inventory update succeeds but the reservation record fails to get created? We could have a quantity deducted without a corresponding reservation. That is why real reservation workflows need a transaction that covers all related database changes. For example:
BEGIN;
UPDATE inventory
SET available_quantity = available_quantity - 7
WHERE product_id = 'LAPTOP-001'
AND available_quantity >= 7;
-- Application checks that exactly one row was updated.
INSERT INTO reservations (
product_id,
quantity,
status
)
VALUES (
'LAPTOP-001',
7,
'CONFIRMED'
);
COMMIT;
If the reservation insert fails, the transaction should roll back, restoring the inventory quantity. This gives us a much stronger guarantee within a single transactional database. It does not, however, solve every problem in a distributed multi-agent architecture.
6. Optimistic concurrency: making agents prove that their information is still valid
Atomic updates are useful, but there are situations where agents need to perform more complicated calculations before modifying a record. For example, a manufacturing agent might retrieve:
- Current machine availability.
- Production capacity.
- Maintenance status.
- Raw material quantities.
- Existing production commitments.
It may take several seconds to evaluate the information. By the time it is ready to schedule production, another agent may have modified the same resource. In these situations, optimistic concurrency control becomes useful. The basic idea is straightforward:
An agent can prepare a decision using a particular version of the data, but the system must verify that the version has not changed before accepting the decision.
We can use the version column introduced earlier.
Suppose the initial record is:
product_id: LAPTOP-001
available_quantity: 10
version: 1
Agent A reads version 1. Agent B also reads version 1. Agent A attempts to reserve seven units:
UPDATE inventory
SET available_quantity = available_quantity - 7,
version = version + 1
WHERE product_id = 'LAPTOP-001'
AND version = 1
AND available_quantity >= 7;
The update succeeds. The database now contains:
available_quantity: 3
version: 2
Agent B still holds version 1. When Agent B attempts its update:
UPDATE inventory
SET available_quantity = available_quantity - 6,
version = version + 1
WHERE product_id = 'LAPTOP-001'
AND version = 1
AND available_quantity >= 6;
The update affects zero rows. Why? Because the version is now 2. Agent B's original observation is no longer current. It must retrieve the latest state and reconsider its decision. This is particularly useful for AI agents because it gives us a way to connect the agent's reasoning context to the actual data version used during execution.
Instead of simply saying: "Reserve six units." The agent's request effectively becomes: "Reserve six units, provided that the inventory is still at the version I evaluated." That is a much safer instruction.
The retry should not blindly repeat the original decision
This point is important. When an optimistic concurrency check fails, the agent should not automatically resend the same operation with the new version. It should first determine whether the original decision is still valid. For example:
def reserve_with_version(
conn,
product_id,
quantity,
expected_version
):
result = conn.execute(
"""
UPDATE inventory
SET available_quantity =
available_quantity - %s,
version = version + 1
WHERE product_id = %s
AND version = %s
AND available_quantity >= %s
RETURNING available_quantity, version
"",
(
quantity,
product_id,
expected_version,
quantity
)
).fetchone()
if result is None:
return {
"success": False,
"reason": "STATE_CHANGED",
"retry_required": True
}
return {
"success": True,
"remaining": result[0],
"new_version": result[1]
}
The next step should be a fresh read followed by a new evaluation. The important thing is that the original decision is not treated as permanently valid. This is a useful distinction between retrying an operation and reconsidering a decision. They are not the same thing.
7. Idempotency: what happens when an agent does not know whether its action succeeded?
Concurrency is only part of the problem. There is another scenario that can be equally dangerous. Imagine Agent A submits a reservation request. The database successfully processes the request. But before the confirmation reaches the agent, the network connection times out. Agent A now faces an uncertain situation.
Did the reservation succeed? Did the request fail? Did the database commit the operation but lose the response? The agent does not know.
A common reaction is to retry. But if the original operation already succeeded, retrying could reserve the inventory twice. This is where idempotency becomes essential. An idempotent operation produces the same effective result when the same request is processed repeatedly. For example, instead of submitting an operation with no unique identity:
{
"product_id": "LAPTOP-001",
"quantity": 7
}
The agent submits a unique request identifier:
{
"request_id": "reserve-order-78421",
"product_id": "LAPTOP-001",
"quantity": 7
}
The backend stores the request identifier and associates it with the resulting reservation. A simplified database structure could be:
CREATE TABLE reservation_requests (
request_id VARCHAR(100) PRIMARY KEY,
product_id VARCHAR(50) NOT NULL,
quantity INTEGER NOT NULL,
status VARCHAR(30) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
The backend should process the request identifier and inventory modification within the same database transaction.
The workflow becomes:
- Receive the reservation request.
- Check whether the request identifier already exists.
- If it exists, return the original outcome.
- If it does not exist, attempt the inventory reservation.
- Record the result.
- Commit the transaction.
A unique database constraint protects against two concurrent submissions using the same identifier. There is another subtle point here. The backend should also validate that a repeated request identifier has the same business parameters. If an agent first submits a request for seven units and later reuses the same identifier for nine units, the backend should reject the conflicting request rather than silently treating it as the original operation. Idempotency protects against duplicate execution. It does not automatically resolve conflicting business decisions. That distinction is important.
8. Distributed transactions: the problem becomes bigger than one database
So far, we have discussed inventory operations within a database. But real enterprise workflows rarely operate within one database. Consider an order-processing workflow involving:
- An inventory service.
- A payment gateway.
- A customer management service.
- A shipping provider.
- An AI orchestration platform.
The inventory reservation may succeed. The payment authorisation may succeed. But the shipping service may become unavailable. What should happen? Should the inventory remain reserved? Should the payment be reversed? Should the order remain pending? This is where distributed transaction design becomes important.
A traditional database transaction can provide atomicity across operations within its transactional boundary.
It cannot automatically guarantee atomicity across unrelated external services.
One common architectural approach is the Saga pattern.
A Saga divides a larger business operation into smaller transactions. Each transaction has a corresponding compensating action when the overall workflow cannot continue.
For example:
| Step | Operation | Possible compensation |
|---|---|---|
| 1 | Reserve inventory | Release inventory |
| 2 | Authorize payment | Void payment authorization |
| 3 | Create shipment | Cancel shipment |
| 4 | Confirm order | Mark order as failed or cancelled |
The workflow might look like this:
flowchart TD
A[Order Received] --> B[Reserve Inventory]
B --> C{Reservation Successful?}
C -->|No| X[Reject Order]
C -->|Yes| D[Authorize Payment]
D --> E{Payment Successful?}
E -->|No| F[Release Inventory]
E -->|Yes| G[Create Shipment]
G --> H{Shipment Created?}
H -->|No| I[Void Payment]
I --> F
H -->|Yes| J[Confirm Order]
An AI orchestrator can manage this workflow, but the compensating actions should be explicitly defined and implemented by the relevant services.
The agent should not simply be instructed to "undo everything" when something fails.
Some actions cannot be perfectly reversed.
A payment authorisation may be voided, but a captured payment may require a refund. A shipment may already be in transit. A manufacturing operation may have consumed physical material.
Compensation is a business process, not merely a reversal of database values.
This is especially important when agents operate across multiple services with different consistency guarantees.
9. The problem of two agents retrying at the same time
There is another failure mode that deserves attention.
Suppose both agents detect a conflict.
Both decide to retry.
Both retrieve the updated inventory.
Both independently determine that another reservation attempt is possible.
If the system does not use atomic updates or version checks, we have simply recreated the original problem.
Retries are not inherently safe.
In fact, poorly designed retries can increase contention and make the system less stable.
Consider the following retry logic:
while not a success:
success = reserve_inventory(
product_id,
quantity
)
This is dangerous because it does not distinguish between different failure conditions.
A request may fail because:
- Inventory is insufficient.
- The database is temporarily unavailable.
- The record version changed.
- The service timed out.
- The request was already processed.
These failures require different responses.
A better approach is to classify errors:
| Failure | Appropriate response |
|---|---|
| Insufficient inventory | Reject or request a revised quantity |
| Version conflict | Refresh state and reconsider |
| Temporary database failure | Retry with bounded backoff |
| Network timeout | Check the original request ID |
| Duplicate request | Return the recorded outcome |
| External service unavailable | Apply workflow recovery policy |
For transient failures, exponential backoff with jitter can reduce repeated collisions.
For example:
import random
import time
def retry_delay(attempt):
base_delay = min(
0.5 * (2 ** attempt),
10
)
return base_delay + random.uniform(0, 0.3)
The retry policy should also have a maximum number of attempts and an overall deadline.
An AI agent should not be allowed to retry an operation indefinitely simply because it believes the task remains incomplete.
A bounded retry policy is a reliability feature.
It is also a cost-control feature.
10. Shared memory is not the same as shared truth
This brings me to a broader architectural issue.
Many multi-agent systems use shared memory, message histories or common context stores.
These mechanisms help agents exchange information.
But shared memory is not necessarily a source of truth.
Imagine that Agent A writes:
{
"product_id": "LAPTOP-001",
"available_quantity": 10,
"observed_version": 1
}
Agent B reads this information from shared memory.
A moment later, another operation reduces the inventory to three.
The shared memory still says ten.
Both agents have access to the same information, but the information is outdated.
Sharing information does not make that information current.
This is why I prefer separating three concepts in agent architecture:
1. Working context
The information an agent uses to reason about a task.
2. Shared coordination state
The information agents use to communicate progress, ownership and workflow status.
3. Authoritative business state
The data that determines whether an operation is actually valid.
These layers can be connected, but they should not be treated as interchangeable.
A memory system can tell an agent what it previously observed.
A coordination service can tell it what another agent is doing.
Only the authoritative business service should determine whether the operation can be committed.
For an inventory system, that authority might be a transactional database.
For a payment system, it might be the payment provider's transaction record.
For a manufacturing system, it might be the production scheduling service.
The specific implementation varies.
The principle does not.
An agent should never treat a remembered state as proof that the current state is still valid.
11. A better architecture for multi-agent systems
If I were designing an agent-based order-processing system, I would avoid allowing every agent to independently modify shared business data.
Instead, I would establish a controlled execution boundary.
Agents could independently analyse information, prepare recommendations and request actions.
But state-changing operations would pass through a service responsible for enforcing business invariants.
Consider this architecture:
flowchart TD
A[Customer Request] --> B[AI Orchestrator]
B --> C[Inventory Agent]
B --> D[Pricing Agent]
B --> E[Payment Agent]
C --> F[Action Request]
D --> F
E --> F
F --> G[Execution Gateway]
G --> H[Authorization Check]
H --> I[Concurrency Control]
I --> J[Idempotency Check]
J --> K[Business Rules]
K --> L[(Authoritative Database)]
L --> M[Committed Result]
M --> N[Event / Audit Log]
N --> O[Agent Feedback]
The execution gateway is particularly important.
It should validate:
- Whether the agent is authorised to operate.
- Whether the request is associated with the correct workflow.
- Whether the operation has already been processed.
- Whether the expected state version is still current.
- Whether the business rules permit the operation.
- Whether the database can safely commit the change.
This architecture also creates a useful separation of responsibilities.
The AI agent is responsible for interpreting context and proposing actions.
The execution layer is responsible for validating and enforcing those actions.
The database is responsible for preserving transactional consistency within its boundaries.
The audit layer records what actually happened.
These responsibilities should not be collapsed into a single agent prompt.
12. What about locking? Should agents wait for one another?
Another approach is pessimistic concurrency control.
Instead of allowing multiple agents to read and independently attempt modifications, the system can lock a resource while an operation is being processed.
In PostgreSQL, for example:
BEGIN;
SELECT available_quantity
FROM inventory
WHERE product_id = 'LAPTOP-001'
FOR UPDATE;
-- Validate the requested quantity.
UPDATE inventory
SET available_quantity = available_quantity - 7
WHERE product_id = 'LAPTOP-001';
COMMIT;
The row-level lock prevents another transaction from modifying the same row concurrently until the transaction completes.
This can be useful when an operation requires several dependent database steps.
However, locking introduces its own challenges.
Long-running transactions can reduce throughput.
Deadlocks can occur when transactions acquire locks in conflicting orders.
And holding a database lock while an AI agent reasons, calls an external API or waits for human approval is generally a bad idea.
Imagine locking inventory while an agent spends eight seconds deciding whether a discount should be approved.
That is not a good use of a database transaction.
A better pattern is:
- Let the agent perform its reasoning outside the transaction.
- Prepare the proposed operation.
- Open a short transaction.
- Validate the latest state.
- Apply the update.
- Commit immediately.
- Return the result to the agent.
The transaction should protect the critical operation, not the entire AI reasoning process.
Optimistic versus pessimistic concurrency
| Consideration | Optimistic concurrency | Pessimistic concurrency |
|---|---|---|
| Main mechanism | Version checks | Locks |
| Conflict handling | Detect and retry | Wait or block |
| Best suited for | Relatively low contention | Higher contention |
| Throughput | Often better with fewer conflicts | Can decrease under heavy contention |
| Implementation | Version columns or conditional updates | Row-level or other locks |
| Main risk | Repeated conflicts | Lock contention and deadlocks |
Neither approach is universally better.
The choice depends on how frequently conflicts occur, how expensive retries are and how long the critical operations take.
For many AI-driven workflows, optimistic concurrency is a reasonable starting point because agent reasoning happens outside the database transaction.
13. The human approval problem: approval does not freeze the world
There is one more scenario worth considering.
Suppose an AI agent recommends a production schedule.
A human manager reviews the proposal and approves it.
The agent then executes the schedule.
But between approval and execution, another system changes the machine availability.
Should the original approval still be valid?
Not necessarily.
The manager approved a particular decision based on a particular state.
If that state changes materially, the approval may no longer authorise the new operation.
This creates an important connection between concurrency control and human oversight.
An approval should be bound to the specific operation and the relevant state that was reviewed.
For example:
{
"approval_id": "approval-9082",
"operation": "schedule_production",
"machine_id": "MACHINE-14",
"production_units": 500,
"expected_machine_version": 12,
"approved_by": "manager-104",
"status": "APPROVED"
}
At execution time, the system checks whether the machine is still at version 12.
If the version has changed, the operation should be reassessed according to the approval policy.
Depending on the change, this may require a new approval.
This is more meaningful than treating an approval as a general permission to execute whatever action the agent eventually decides to perform.
It also illustrates why human oversight and technical consistency cannot be designed independently.
The approval process must be connected to the state being approved.
14. Observability: how do we know two agents are interfering?
Concurrency bugs can be difficult to reproduce.
They may occur only under specific timing conditions.
A workflow might run thousands of times successfully before two agents happen to read and modify the same resource within a narrow time window.
Traditional application logs are often insufficient.
A useful multi-agent observability strategy should capture more than the agent's messages.
I would want to see:
| Field | Why it matters |
|---|---|
| Agent ID | Identifies which agent initiated the operation |
| Workflow ID | Connects related agent activities |
| Request ID | Supports idempotency and duplicate detection |
| Resource ID | Identifies the shared business object |
| Observed version | Records the state used for reasoning |
| Execution version | Records the version at execution |
| Operation type | Shows what modification was attempted |
| Conflict status | Identifies concurrency failures |
| Retry count | Reveals repeated contention |
| Commit status | Distinguishes proposed actions from committed actions |
For example, a structured event might look like this:
{
"workflow_id": "order-78421",
"agent_id": "inventory-agent-A",
"request_id": "reserve-order-78421",
"resource_id": "LAPTOP-001",
"observed_version": 1,
"execution_version": 2,
"operation": "reserve_inventory",
"requested_quantity": 7,
"status": "VERSION_CONFLICT",
"retry_count": 1
}
This gives engineers something much more useful than a log entry saying:
"Inventory reservation failed."
It tells us which state the agent observed, which state existed when the operation was attempted and why the system rejected it.
I would also monitor:
- Conflict rates by resource.
- Average retries per operation.
- Duplicate request frequency.
- Transaction failure rates.
- Time between state observation and execution.
- Unresolved workflow states.
- Compensation and recovery failures.
One metric I find particularly useful is the observation-to-execution gap.
The longer an agent spends between reading a resource and attempting to modify it, the greater the opportunity for that observation to become outdated.
It is not a direct measure of correctness, but it helps identify workflows that are more exposed to concurrency problems.
15. Testing concurrency is different from testing agent intelligence
A multi-agent system can pass every individual agent test and still fail when the agents operate together.
This is why I would separate agent-level testing from system-level concurrency testing.
An individual agent test might verify:
- It correctly reads inventory.
- It identifies insufficient stock.
- It generates the expected reservation request.
- It handles an error response.
These are useful tests.
But they do not establish that two agents can safely reserve the same inventory simultaneously.
For that, we need concurrency tests.
Consider a simple scenario:
from concurrent.futures import ThreadPoolExecutor
def run_reservation(quantity):
return reserve_inventory(
connection,
"LAPTOP-001",
quantity
)
with ThreadPoolExecutor(max_workers=2) as executor:
futures = [
executor.submit(run_reservation, 7),
executor.submit(run_reservation, 6)
]
results = [
future.result()
for future in futures
]
print(results)
The test should establish that:
- No more than ten units can be reserved.
- The inventory never becomes negative.
- Only valid reservations are committed.
- A failed operation does not create a reservation record.
- Duplicate requests do not create duplicate reservations.
- The final inventory matches the committed reservation records.
For the ten-unit scenario, the valid outcome is that seven units are reserved and six are rejected, or vice versa.
The system must not accept both.
The test should also run repeatedly under different timing conditions.
A more comprehensive test suite should include:
Simultaneous requests: Multiple agents attempt to reserve the same resource.
Duplicate submissions: An agent retries after an uncertain network response.
State changes during reasoning: The underlying record changes after the agent retrieves it.
Partial workflow failure: Inventory succeeds, but payment or shipping fails.
Service interruption: The orchestration service restarts after a database commit.
Conflicting approvals: Two workflows attempt to modify the same approved resource.
For production systems, I would also consider property-based testing and fault injection.
The goal is not merely to verify that the expected example works.
It is to establish that important business invariants remain true across many possible execution orders.
For inventory, the most basic invariant is:
available_quantity >= 0
Another useful invariant is:
Initial inventory
=
Available inventory
+ Active reservations
+ Other accounted inventory movements
The exact accounting equation depends on the business model, but the principle is that every committed change should have an explainable place in the system's state.
16. A practical checklist for building reliable multi-agent systems
After looking at these scenarios, I would use the following checklist when reviewing an architecture that allows multiple agents to interact with shared resources.
| Area | Engineering question |
|---|---|
| State ownership | Which service owns the authoritative data? |
| Concurrency | Can two agents modify the same resource simultaneously? |
| Atomicity | Are critical business updates performed atomically? |
| Versioning | Does the system detect stale decisions? |
| Idempotency | Can a retry accidentally repeat a committed action? |
| Authorisation | Is every state-changing operation independently validated? |
| Coordination | Can agents detect conflicting workflow ownership? |
| Recovery | What happens when a later step fails? |
| Observability | Can we reconstruct which agent changed what and when? |
| Testing | Have we tested simultaneous execution and partial failures? |
I would also introduce a simple design rule:
Agents may reason independently, but the shared business state must have an explicit consistency strategy.
That strategy might use atomic updates, optimistic locking, pessimistic locking, serializable transactions, unique constraints or distributed workflow patterns.
The right combination depends on the application.
What matters is that the consistency mechanism exists outside the assumption that agents will always make compatible decisions.
17. The bigger lesson: intelligence does not eliminate distributed systems problems
There is a tendency to discuss agentic AI as if adding intelligence to software changes the fundamental rules of system engineering. It doesn't. An AI agent can interpret an instruction, evaluate alternatives, understand business context and choose an appropriate action.
But it cannot make two concurrent database transactions automatically compatible. It cannot guarantee that an external API processed a request exactly once.
It cannot assume that another agent has not changed a resource. And it cannot turn a distributed workflow into an atomic transaction simply by coordinating messages. These are engineering problems that require engineering solutions.
In fact, as agents become more capable of acting independently, the consequences of weak concurrency control become more significant. A traditional application might have a small number of predictable processes modifying a resource. An agentic platform may have dozens of autonomous workflows interacting with the same business objects. One agent may update a customer account. Another may initiate a refund.A third may recalculate a credit limit. A fourth may trigger a compliance review. Each operation may be i ndividually valid. But the overall result depends on how the operations interact. That is where system-level correctness becomes more important than individual agent performance. I also think this changes how we should evaluate multi-agent systems. We should not only ask:
- Can the agents communicate?
- Can they complete their assigned tasks?
- Can they reason through complex instructions?
- Can they recover from errors?
We should also ask:
- Can they preserve business invariants under concurrent execution?
- Can they distinguish a proposed action from a committed action?
- Can they identify when the state they relied on has changed?
- Can they recover without duplicating or corrupting previous operations?
These questions are less glamorous than demonstrations of autonomous collaboration. But they are much closer to what organisations need before trusting agents with critical operations.
Conclusion: Two correct decisions do not guarantee one correct outcome
I started with a simple inventory example. Two agents saw ten available units. One wanted seven. The other wanted six.
Both made reasonable decisions based on the information they received. The failure was not necessarily in their reasoning. It was in the system that allowed two independent decisions to modify the shared state without sufficient coordination.
That distinction has stayed with me because it applies far beyond e-commerce. It applies to financial transactions, manufacturing schedules, cloud infrastructure, customer account management, healthcare workflows and enterprise automation.
The next generation of AI systems will increasingly operate in environments where multiple agents share resources and execute actions concurrently. We cannot solve the resulting problems by making agents more intelligent alone. We need better state management, transactional boundaries, concurrency control, idempotency, observability and recovery mechanisms.
Most importantly, we need to recognise that an agent's confidence is not a consistency guarantee. An agent can be correct about what it observed. It can be correct about what it decided. It can even be correct about the action it attempted. And the overall system can still be wrong.
The real challenge in multi-agent AI is not just teaching agents to work together. It is engineering the environment so that their independent actions cannot silently undermine one another. That is the difference between a collection of intelligent agents and a reliable multi-agent system.
What has been your experience with concurrency in multi-agent systems? Have you encountered situations where individually successful agents produced an inconsistent business outcome? I'd be particularly interested in how you handle shared state, retries and conflicting operations in production architectures.
Top comments (0)