DEV Community

Cover image for Asked for real places, an LLM got 7 of 17 wrong. My agent isn't allowed to name one.
Edy Cu
Edy Cu

Posted on

Asked for real places, an LLM got 7 of 17 wrong. My agent isn't allowed to name one.

A therapist treating depression often knows the next step: schedule a few things the patient enjoys, then have them rate how it went. The catch is that a depressed patient often can't name one thing they would enjoy this week. So I built Rekindle, a planning agent for clinicians that starts from what the patient used to love (an artist, a show, a food) and turns it into a week of real, open, nearby activities, built on Qloo's taste API, for the Qloo Agentic Hackathon on Devpost.

The first design question was the obvious one: why not just ask an LLM for the week? I measured it. Then I made one rule that shaped the rest of the code: the model chooses where to look; Qloo and deterministic code decide what is true. The model never names a venue.

Live app (no login, fictional cases only): https://rekindle.edycu.dev · code (MIT): https://github.com/edycutjong/rekindle

The number, measured on October 7

I gave the same model (qwen3.8-flash) the same 20 fictional patient profiles in two arms. Arm A: asked for named places for the week. Arm B: Rekindle. A random 20 outing venues per arm were then checked one by one against public listings (venue sites, city listings, review and delivery listings) for three things: does it exist at that address, is it open at the planned hour, and, for a patient in recovery, does it serve alcohol.

Measured on October 7: the LLM alone got 7 of 17 places wrong (41.2%): 4 not found, 2 closed at the planned hour, 1 serving alcohol to a patient in recovery. Rekindle got 0 of 14. The other rows were inconclusive (3 of the LLM's 20, 6 of Rekindle's 20) and are left out of the denominator rather than guessed.

Before you quote that, here is what it is not:

  • It was not blind. The hand check of that run's venues (finished the next day) was done by my build agent, Claude, an AI. It ran the bench, so it could have inferred the arm from the address style. No external human auditor has returned a result yet.
  • It is small. 17 and 14 checked venues.
  • It predates later work. The October 7 run came before 30 later changes to the plan path, all listed in the repo's DEMO.md. A re-run on the final deploy is planned for late October.

There is also an automated checker over all 20 profiles. It flags 58 of the LLM's 79 venues, but I don't lead with it: it grades with Qloo, which flatters Rekindle by construction, and it over-flags. In the audit sample, 4 of the 10 venues it couldn't match were real and open (it matches by street address and misses large venues such as parks). So the smaller, hand-checked number goes first.

The rule, in code

The tools the model can call carry no constraints at all. Here is the outing tool's input schema from lib/agent.ts:

findOutings: tool({
  description: "Fill a solo slot with a verified place: open at the hour and checked by the server against the case row (the model cannot see or change those checks).",
  inputSchema: z.object({ day: dayEnum, hour: z.number().int().min(8).max(21).optional(), narrowMeters: z.number().int().min(300).max(5000).optional() }),
  // ...
}),
Enter fullscreen mode Exit fullscreen mode

The model can pick a day, an hour and a tighter radius. Distance limit, budget, "in recovery", "quiet" and "wheelchair entrance" are read from the stored case on every request; narrowMeters can only tighten. Qloo returns candidates, and a pure TypeScript verifier with no I/O (lib/verify.ts) decides each one: open at the slot, near, within budget, an activity rather than a shop, alcohol-free in recovery, quiet or accessible when asked, and not rated low before. Refused candidates stay on the clinician's sheet, struck through, with the rule that refused them.

The model's only free text is one coaching line per card, and that goes through a boundary too:

for (const { entity_id, line } of submitted) {
  const s = byId.get(entity_id);
  if (!s) {
    dropped.push({ rule: "R-ID", entity_id, detail: "coaching line references an ID no tool returned this run" });
    continue;
  }
  // ...
  const fault = lineFault(s.card!, text, names);
  if (fault) {
    dropped.push({ ...fault, entity_id });
    continue;
  }
Enter fullscreen mode Exit fullscreen mode

A line keyed to an ID no tool returned is dropped and logged. So is a line over 25 words, a line that names another place from the run, a line that states a fact the card owns (a distance or a price), and a line that mentions drinking, on every case, because the model never sees the recovery flag.

The wall: "exclude alcohol" is not one filter

The demo moment is a toggle. The clinician clicks In recovery, and places that carry an alcohol tag or keyword are struck in place and their slots refilled live. In a live run on October 8 that was "3 outings struck in place and 3 slots refilled live · 2.2 s". Getting there took three wrong turns.

Wrong turn 1: trusting the exclusion filter. Qloo's insights endpoint takes filter.exclude.tags. I sent Maya's Thursday place query (Maya is a fictional case, Williamsburg, Brooklyn) three times in a live probe on October 7 and counted what came back:

filter.exclude.tags Places returned Still alcohol-tagged
none 25 19
4 bar tags 25 17
14 alcohol tags 25 2 (plus 2 ambiguous)

Excluding the four obvious bar tags removed 2 of 19. The alcohol vocabulary is spread over at least six tag namespaces (category:place, amenity:qloo, beverage_offering:qloo and others), so excluding a concept means listing every tag that expresses it. Even with 14 tags excluded, a coffee shop tagged beverage_offering:qloo:specialty_alcoholic came back.

Wrong turn 2: a substring screen. The fix seemed to be "refuse any tag containing pub, bar, wine…". A probe showed it would refuse espresso_bar and independent_publishing. The screen now matches exact deny-listed tag IDs first, then whole tokens of each tag's last segment, with an allow list:

const tokens = seg.split("_");
const isBar = seg === "bar" || BAR_SERVICE.has(seg) || (seg.endsWith("_bar") && !BAR_ALLOW.has(seg));
// `bar_<x>` segments describe furniture or ordering (bar_stools, bar_area), and `happy_hour` alone is a price deal:
// a signal for the clinician, not proof of alcohol
const furniture = (seg.startsWith("bar_") && !BAR_SERVICE.has(seg)) || seg === "happy_hour";
const tokenHit = tokens.some((tok) => ALC_TOKENS.has(tok)) || isBar || furniture;
Enter fullscreen mode Exit fullscreen mode

The furniture case is why there is a third verdict. "Bar stool seating" doesn't prove alcohol, so it goes to the clinician in a dashed REVIEW box instead of being silently passed or refused.

Wrong turn 3: a beer hall got through. The bench placed "TeKu Tavern + Café" for a profile not in recovery, because its cafe category passed the activity check. Drinking-establishment categories are now refused for every case.

Does the screen earn its place? The October 7 bench ran an ablation on the 10 in-recovery profiles: with the screen on, 0 of 30 outing cards carried an alcohol tag or keyword. Screen off with the 14-tag exclusion kept: 3 of 28. Screen off with only the 4 bar tags excluded: 21 of 28. It counts tags and keywords, not menus, which is its limit (below).

Opening hours are day-level

filter.hours=thursday keeps places open on Thursday, not at 18:00. In the same October 7 probe, a Thursday query returned 25 places, 12 of them closed at 18:00 by their own listed hours, and some places have no hours at all. So the verifier checks the slot itself and returns OPEN, CLOSED or UNKNOWN. UNKNOWN is refused, never assumed open. A card closed at 18:00 but open earlier that day moves to an open hour and says so on the card ("moved from 18:00 (closed then)").

Week two learns, and says how sure it is

The patient rates pleasure and mastery 0–10. A rating of 7 or more becomes a Qloo signal for week 2, 3 or less an exclusion, and the top-rated activity comes back in its slot after a live re-check. Qloo's explainability weights say which input led each new card; Rekindle draws an "ember thread" from a rated activity only when it leads the patient's best original love by at least 0.02, and labels narrower leads as ties. It is Qloo's attribution, not causation, and the app says so on the screen. On live re-plans most leads are still ties.

In the bench (re-plan check, same October 7 run), 16 of 16 eligible profiles passed: week 2 had at least one new card whose top input was a highly rated activity, and none of the low-rated ones came back. Four profiles were excluded because week 1 had too few verified outings to rate.

What it costs

Grounding is slower. In the October 7 bench, a Rekindle plan took p50 17.43 s and p95 20.61 s on the hosted app, with about 9.4 Qloo calls per plan. The LLM alone answered in p50 7.2 s. It meets the 30 s budget I set but misses the 15 s I wanted; the coaching step dominates.

Honest limits

  • It is clinician support, not a clinical tool: no diagnosis, no efficacy claims, no patient logins. Every case in the app, Maya included, is fictional, and no clinician has endorsed it.
  • The alcohol screen is a deny list. A venue that serves alcohol and carries none of the tags or keywords would pass.
  • Opening hours come from Qloo; if they are wrong upstream, the check passes a closed door.
  • In thin areas, outing slots stay empty with a reason. It doesn't guess.
  • The headline audit was done by an AI build agent, not blind, on 17 and 14 venues.

Try the click path (plan Maya's week, toggle In recovery, approve, rate three cards, re-plan): https://rekindle.edycu.dev · every live Qloo request behind the latest plans: https://rekindle.edycu.dev/integrations/qloo · code and bench files: https://github.com/edycutjong/rekindle · Devpost entry: https://devpost.com/software/rekindle-xw53nk

Top comments (0)