If you have seen the story about an AI-controlled vending machine stocking underwear and live fish, you already know Andon Labs is not trying to make AI look polished.
That is the point.
Their work is built around a question that matters to anyone evaluating agents: what does reliability look like once an AI leaves the demo environment and starts making decisions that affect real operations? Andon’s answer is to combine repeatable simulations with real-world testbeds, because both are needed to understand whether an agent can do more than complete a single task.
A useful checklist for thinking about agent reliability
A good way to read Andon’s experiments is as a checklist rather than a spectacle.
- Can the agent complete one task?
- Can it repeat that task under changing conditions?
- Can it handle exceptions the engineers did not model?
- Can it operate when mistakes have consequences?
- Can it stay useful inside a workflow, not just in a sandbox?
That list matters because agent evaluation often stops too early. A system can look competent in a narrow test and still be unreliable when it is embedded in a business process. Andon’s work pushes on that gap.
Why simulations are necessary, but not sufficient
In simulation, you get control. You can reset the environment, reproduce conditions, and compare runs. That makes simulations ideal for iteration and debugging.
But there is a limitation: simulations only test what you thought to include.
Real-world environments are messier in ways that are hard to model ahead of time. People act unpredictably. Systems fail in odd ways. Inputs arrive late, incomplete, or contradictory. The world creates edge cases that are invisible in a fully scripted setup.
Andon’s team reasoned that moving into the physical world would expose the agents to situations and consequences that engineers would never fully anticipate in advance. That is why the company does not treat simulation as the whole evaluation strategy. It treats simulation as one layer, and real-world testbeds as the other.
For builders, that combination is the important pattern:
- Use simulation to move quickly.
- Use the real world to find what your simulation missed.
- Expect the second step to be harsher than the first.
Why the weird failures are actually the useful part
The vending machine example got attention because it was unusual and easy to describe. But the value of those failures is not novelty. It is visibility.
Memorable failures show a difference that is easy to miss in polished AI demos:
- finishing one action is not the same as maintaining a process
- a narrow success is not the same as reliable operation
- a good result in a constrained setup is not proof of business readiness
That distinction is central to Andon’s approach. The company is not just asking whether an agent can place an order, make a decision, or respond to a prompt. It is asking whether the agent can keep functioning when the outcome affects inventory, money, customer experience, or other real constraints.
This is where many AI evaluations become too optimistic. A task benchmark can tell you what the model did in one moment. A business test can tell you what happens when those moments accumulate.
What the real-world setup is meant to reveal
Andon’s move into the physical world comes with a trade-off.
On one side, real environments are harder to control. That makes experiments noisier and less tidy than simulation. You lose some repeatability. You also have to accept that results may be harder to interpret cleanly.
On the other side, the mess is the point.
Real-world tests are especially good at surfacing failure modes that would never appear in a scripted sandbox. That is why Kapoor frames these experiments as a way to discover how agents fail, not just whether they can succeed. This is a useful distinction for anyone building with AI agents.
If the goal is only to prove capability, then a demo may be enough.
If the goal is to understand reliability, then you need to know:
- where the agent breaks
- what kinds of inputs confuse it
- how often it recovers
- what kinds of intervention a human needs to provide
- whether the workflow stays stable when the agent is wrong
That is a much more practical standard than asking whether the agent looked impressive in a controlled environment.
The Andon Café detail is revealing
One of the most interesting details is the display inside Andon Café. Visitors can see Mona’s bank balance and recent activity. Nearby, a handset and tablet let someone speak to the system.
That setup matters because it shows that the experiment is not just about observation. It is also about interaction.
A display helps you monitor the state of the system. A handset or tablet gives you a way to intervene, question, or redirect it while it is still responsible for something in the world. In other words, the interface is part of the test.
For developers, this is a practical design lesson. If you are building or evaluating an AI agent, telemetry alone is not enough. You also need to think about the operator experience:
- How does a human inspect what the agent is doing?
- How quickly can someone step in?
- What does intervention look like in practice?
- Can the system be audited after the fact?
Those are not cosmetic questions. They determine whether a live AI workflow is manageable.
The real lesson for builders
Andon Labs is treating real businesses as stress tests, not as marketing props.
That approach is valuable because it keeps evaluation grounded in consequences. If an agent is going to participate in operations, then it should be tested in conditions that expose operational risk, not just in conditions that make the model look clever.
The practical takeaway is simple:
- Simulations help you iterate.
- Real-world testbeds help you find failure modes.
- Completing a task is not the same as running a business process.
- Reliability only becomes visible when the system has to handle consequences over time.
If you are building AI agents today, that checklist is worth keeping nearby. It is easy to overestimate a model that works in a clean benchmark. It is much harder, and much more useful, to learn how it behaves when the environment pushes back.
And that is why Andon Labs puts AI agents in charge of real businesses: not because the systems are ready, but because the only way to learn what they can really do is to let them face reality.
Top comments (0)