DEV Community

VOYGR
VOYGR

Posted on

Do AI assistants recommend closed places? 12 answers in PlaceBench did

In PlaceBench, our test of six AI assistants on 150 everyday questions about real places, 12 answers sent the user to a place that a public source showed had closed, moved or changed its name.

That's 12 out of 900 first answers, so it isn't most of the errors. But it's the error you only find out about when you're standing at the door.

How we counted a place as closed

We only counted a place as closed, moved or renamed when a public source showed it: a closure or move notice, the operator's own list, a licence suspension, a dated press report, or Google's "permanently closed". A listing that still shows the place as open didn't outweigh that evidence.

A closed place lowered the grade. When it was the main recommendation, the answer usually counted as wrong. When it was a backup or one option among several, it counted as partly right or a small error.

Why assistants recommend closed places

PlaceBench doesn't measure the cause. One likely reason: assistants answer from listings and pages on the web, and those often aren't updated when a business closes.

We measured this separately before. In a check of 400 places, about 1 in 4 closed businesses still showed as open on Google Maps, or had no listing at all: Is Google Maps accurate for closed businesses?

The scores

Assistant Score Wrong answers, of 150
Dots 92 0
Muse 83 6
Claude Duet 79 6
Instinct 76 12
Grok Bot 73 12
Ask Maps 49 30

Checking a place before you send someone there

If you're building an assistant or agent that recommends places, check the place's status before you recommend it. The VOYGR Validate API takes a business name and address and returns open, closed, unknown or not_exists, with a confidence score:

voygr validate "Blue Bottle Coffee" "66 Mint St, San Francisco, CA 94103" --country US
Enter fullscreen mode Exit fullscreen mode
{"operational_status": {"status": "closed", "closure_confidence": 0.939, "closure_reason": "permanent"}}
Enter fullscreen mode Exit fullscreen mode

That café still scores 0.9954 on existence. It existed, and it's closed. Go by the status, not the existence score. Install with pip install voygr; a free key comes with credits: api.voygr.tech/checkout.

Data

PlaceBench questions, answer keys, sources and grades are on GitHub under CC BY 4.0: github.com/voygr-tech/placebench. Full write-up: voygr.tech/placebench.

Top comments (0)