In PlaceBench, our test of six AI assistants on 150 everyday questions about real places, 12 answers sent the user to a place that a public source showed had closed, moved or changed its name.
That's 12 out of 900 first answers, so it isn't most of the errors. But it's the error you only find out about when you're standing at the door.
How we counted a place as closed
We only counted a place as closed, moved or renamed when a public source showed it: a closure or move notice, the operator's own list, a licence suspension, a dated press report, or Google's "permanently closed". A listing that still shows the place as open didn't outweigh that evidence.
A closed place lowered the grade. When it was the main recommendation, the answer usually counted as wrong. When it was a backup or one option among several, it counted as partly right or a small error.
Why assistants recommend closed places
PlaceBench doesn't measure the cause. One likely reason: assistants answer from listings and pages on the web, and those often aren't updated when a business closes.
We measured this separately before. In a check of 400 places, about 1 in 4 closed businesses still showed as open on Google Maps, or had no listing at all: Is Google Maps accurate for closed businesses?
The scores
| Assistant | Score | Wrong answers, of 150 |
|---|---|---|
| Dots | 92 | 0 |
| Muse | 83 | 6 |
| Claude Duet | 79 | 6 |
| Instinct | 76 | 12 |
| Grok Bot | 73 | 12 |
| Ask Maps | 49 | 30 |
Checking a place before you send someone there
If you're building an assistant or agent that recommends places, check the place's status before you recommend it. The VOYGR Validate API takes a business name and address and returns open, closed, unknown or not_exists, with a confidence score:
voygr validate "Blue Bottle Coffee" "66 Mint St, San Francisco, CA 94103" --country US
{"operational_status": {"status": "closed", "closure_confidence": 0.939, "closure_reason": "permanent"}}
That café still scores 0.9954 on existence. It existed, and it's closed. Go by the status, not the existence score. Install with pip install voygr; a free key comes with credits: api.voygr.tech/checkout.
Data
PlaceBench questions, answer keys, sources and grades are on GitHub under CC BY 4.0: github.com/voygr-tech/placebench. Full write-up: voygr.tech/placebench.
Top comments (0)