Here is a Hinglish call to a restaurant voice agent:
caller: Do paneer wrap... actually ek hi karo. Bina pyaaz. Aur ek mango lassi.
agent: 2 Paneer Wrap add kar diya. Paneer Wrap 1 kar diya. Paneer Wrap No onion kar diya.
1 Mango Lassi add kar diya. Aur kuch?
caller: Bas itna hi, thanks.
agent: Aapka order: 1 Paneer Wrap, No onion; 1 Mango Lassi. Order place ho gaya, shukriya!
The agent read back one wrap without onion. This is what it committed:
| Field | Expected | Actual | Result |
|---|---|---|---|
| Submitted orders | 1 | 1 | Pass |
| Paneer Wrap quantity | 1 | 2 | Fail |
| Paneer Wrap · Onion | No onion | With onion | Fail |
| Mango Lassi quantity | 1 | 1 | Pass |
A transcript-based test, or an LLM judge reading the conversation, passes this call. The customer gets two wraps with onion.
What I built
IndicOrderBench is an open-source benchmark and CI gate for voice ordering agents. It compares the committed backend state against each scenario's acceptable end states, in English and Hinglish, instead of grading the words the agent said.
The single-file HTML report from the fixed reference agent. The failing run of the same scenario is the one that catches the wrong order.
The hard failures it targets aren't accent or latency. They're mid-sentence corrections ("two... actually one"), modifiers ("no onion"), cancellations and duplicate submissions, especially in code-mixed speech where number words and negations don't look like English.
pip install indicorderbench
iob demo --out demo-out # buggy vs fixed reference agent, no network, no keys
iob run starter --agent builtin:correct --tag smoke --trials 3
A run writes results.json (every transcript, tool call and backend snapshot), junit.xml for any CI system, and a single-file report.html that opens from file://. The exit code is non-zero when an agent change starts placing wrong orders, so the build fails before a customer sees it.
You can connect your agent in-process from Python, or over an HTTP adapter. The sandbox backend hands you JSON-schema tool definitions to give your LLM.
Scope
It's small and deterministic on purpose: reproducible runs, no API keys for the demo. It's an early benchmark, not a leaderboard of production systems.
Repo (Apache-2.0): https://github.com/Lingikaushikreddy/indicorderbench
PyPI: https://pypi.org/project/indicorderbench/
If you've seen a voice agent fail in a way this doesn't cover, open an issue with the transcript. Scenarios are easy to add.
Top comments (1)
One failure case I’d add: the order commits, but the acknowledgement is lost, so the agent retries. I’d check both the final order and the events sent to the kitchen or payment system; one order row can still hide duplicate side effects. Does IndicOrderBench inspect those events, or only the sandbox’s final state?