Disclosure: I’m Soham, founder of Builderr. We’re running Signalpost, a challenge for agents that research Norwegian companies. I want to share the design question behind it, because it’s easy to miss when a result looks polished.
A company-research agent can return a real fact, cite a real page, and still be wrong if that page belongs to a parent company or a similarly named business. A useful profile needs to get the entity right before it gathers facts. It also needs to show where each fact came from, when it was checked, and what it could not verify.
In Signalpost, the agent receives a Norwegian company number and must return a complete profile from public information. The challenge scores coverage, fact accuracy and evidence, synthesis, and ease of checking. A material wrong-company match blocks qualification. Every entry is tested against the same checked reference collection, so the comparison is about the submitted agent, not who picked the easiest companies.
The hard parts are practical: match the exact entity, keep the source and retrieval date with every claim, preserve useful history when a profile is refreshed, and return a clear result even when information is missing. I’m especially interested in failure cases that could make a well-written profile misleading.
If you build research or browsing agents, what would you test first: parent/subsidiary matching, stale facts, coverage of smaller companies, or something else?
The challenge is open until October 21, with $2,500 in final rewards. The brief, scoring rules, and starter are here: https://builderr.ai/challenges/signalpost
Keep building,
Soham
Top comments (1)
Parent and subsidiary matching, and not because it is the hardest. The other three degrade gracefully and that one does not. A stale fact is wrong on one line. A wrong entity is wrong on every line while looking complete.
UK payments has the narrow version of this as a live product. Confirmation of Payee checks the name a payer typed against the name on the account before money moves, and the thing it got right after some painful iterations is that the answer has three states rather than two: match, close match, no match. The close-match state exists because legitimate variants are extremely common. Registered name against trading name, a missing Ltd, a personal name on a sole trader's account. A binary answer forces the system to be wrong about a large and entirely innocent population.
For your scoring that suggests a specific test rather than a general one. Take a company whose registered name differs from its trading name, where a parent or a sibling in the group shares that trading name. The agent that returns a confident single profile there is the one to penalise. The agent that returns the right entity plus an explicit note that two candidates matched the name is the one to reward, even though it reads as less polished.
The other thing I would test is whether the retrieval date sits on every claim or on the profile. A profile dated today holding a fact checked in March is a dated figure that is not actually dated, and that is worse than an undated one, because it reads as fresh.