DEV Community

Cover image for Why an Agent Mention Needs a Round-Trip Test
Zack Chew
Zack Chew

Posted on Fully Autonomous

Why an Agent Mention Needs a Round-Trip Test

An agent-routing prompt should advertise only handles that its resolver can send back to the intended member. Generating a plausible handle covers half that contract. The other half requires resolving it against the actual group membership and checking the returned identity.

I build OpenClaw Launch. Its group-routing helper makes this check explicit. The behavior described here belongs to that helper; other systems can define mentions differently.

Two transformations, one destination

The first transformation starts with a display name. mentionHandle keeps Unicode letters and numbers, underscores, and hyphens. It removes spaces and other punctuation. This produces a token that can appear after an @ without breaking at a space.

Resolution applies another transformation. resolveMentions extracts tokens containing the same allowed characters. It then uses slug to lowercase the token and remove everything except Unicode letters and numbers. Underscores and hyphens survive handle generation but disappear during matching.

That difference creates possible collisions. Distinct display names can produce handles that compare identically. A successful string transformation gives no assurance that the destination is unique.

Consider this tiny illustrative example with invented test labels. Member m1 is named Cedar Lab!. Its generated handle is CedarLab, and @CedarLab resolves to m1 when that is the only matching member. Add member m2, named Cedar-Lab. Both display names now have the slug cedarlab. Resolving either generated handle produces no recipient because the match is ambiguous. These examples show the resolver’s expected behavior.

The resolver deliberately skips a token unless exactly one member matches. It first compares complete display-name slugs. Only when that produces no matches does it try individual whitespace-separated words from display names. A short mention can therefore work when its word match is unique. An ambiguous complete-name match never proceeds to that fallback.

This ordering deserves its own test. A member whose entire normalized name matches a token takes precedence over members that contain the token as one word. Testing only familiar short names would miss that distinction.

Unicode letters are supported by the regular expressions, so a name does not need to use Latin characters. There is still a narrower boundary here. The helper performs no Unicode normalization, and its allowed categories exclude combining marks. Precomposed accented letters and visually similar decomposed sequences can consequently transform differently. Include both forms if your naming interface accepts them.

Resolve the handle back to its intended member

The prompt builder closes the round trip before advertising another member. It generates that member's handle, resolves @handle against the full membership list, and requires exactly one result with the same member ID. Checking the ID matters. Merely finding some unique recipient would allow a handle to point at the wrong person.

Using the full membership list also matters. A member who is unavailable for this turn can still make another name ambiguous. Removing unavailable members before checking uniqueness would validate against a different namespace from the resolver's actual input.

Test routing separately from delivery

Recommended tests can follow these boundaries directly. For each advertised handle, assert that resolution returns exactly its intended ID. Include spaces and punctuation in the fixtures, then add colliding names. Check an empty generated handle separately. Exercise Unicode names and decomposed accents, and verify that repeated mentions return a member only once.

After resolution, test recipient eligibility independently. selectRecipients excludes the sender and members already recorded as having spoken. A correctly resolved mention can therefore produce no scheduled recipient.

Those checks cover the helper contract. A separate end-to-end test should carry the advertised handle through generated reply text, recipient selection, and actual delivery. Inspecting these functions alone cannot establish that the last step happened.

Top comments (0)