What should a paper-trading benchmark log?

Soham Sinha — Tue, 13 Oct 2026 00:47:17 +0000

Disclosure: This post was generated autonomously with AI.

For a simulated market order, a volume total alone does not show what happened between submission and settlement. Timestamps, partial fills, fees, and remaining exposure may all matter—but which details are essential to reproduce an execution?

What should a minimum event log contain to distinguish a credible simulation from overly generous fills? How should reporting separate execution assumptions from a strategy’s predictive performance?

What does “verified” mean for a company-research agent?

Soham Sinha — Mon, 12 Oct 2026 23:07:41 +0000

Disclosure: I’m Soham, founder of Builderr. We’re running Signalpost, a challenge for agents that research Norwegian companies. I want to share the design question behind it, because it’s easy to miss when a result looks polished.

A company-research agent can return a real fact, cite a real page, and still be wrong if that page belongs to a parent company or a similarly named business. A useful profile needs to get the entity right before it gathers facts. It also needs to show where each fact came from, when it was checked, and what it could not verify.

In Signalpost, the agent receives a Norwegian company number and must return a complete profile from public information. The challenge scores coverage, fact accuracy and evidence, synthesis, and ease of checking. A material wrong-company match blocks qualification. Every entry is tested against the same checked reference collection, so the comparison is about the submitted agent, not who picked the easiest companies.

The hard parts are practical: match the exact entity, keep the source and retrieval date with every claim, preserve useful history when a profile is refreshed, and return a clear result even when information is missing. I’m especially interested in failure cases that could make a well-written profile misleading.

If you build research or browsing agents, what would you test first: parent/subsidiary matching, stale facts, coverage of smaller companies, or something else?

The challenge is open until October 21, with $2,500 in final rewards. The brief, scoring rules, and starter are here: https://builderr.ai/challenges/signalpost

Keep building,
Soham

Trying to turn everyday problems into measurable outcomes

Soham Sinha — Sat, 12 Sep 2026 17:06:57 +0000

I’m building Builderr around a simple hypothesis: if I can turn a business problem into a clear, measurable outcome, I can invite builders from around the world to come up with solutions.

Companies pay for verified outcomes. Builders know what they’re trying to achieve and how their work will be assessed.

I’ve been working through what this takes in practice. Here are two examples.

A friend of a friend is building a prediction market. We got talking about what he needed: more trading activity and liquidity on the platform.

To turn that into a challenge, I first needed data builders could test with.

I had to figure out what market data and API access were available, and how builders could use them.

Researching APIs gave me a way forward. Polymarket—a separate platform from his—publishes APIs and SDKs, for example. I still needed to turn data access into something builders could test against.

For the proposed challenge, I settled on a small interface: give the bot current market data, its account state and the limits, then ask it to buy, exit or wait. The evaluator would handle the connection to the venue.

I needed a baseline before I could set a bar

I spent a couple of days researching approaches and building a simple bot. Could a basic version do something useful? What would it cost?

That gave me a starting point for the qualification threshold: what could a basic implementation achieve, and what would be useful to him?

I expect builders to improve on it. A simple bot is fine if it delivers the result.

There's still a calibration step here. A short run wouldn't prove that a bot can sustain the proposed 12-hour threshold. The baseline needs to run under the same conditions entrants will face before that bar is final.

I didn't want entrants to fund the experiment

The next question was how to evaluate submissions in a real market without making every developer open and fund an account.

The proposed process has two stages. First, builders develop against a test API using real-time market data, and we run their submissions there. The bot has to complete its trades, respect the limits and keep working without manual help.

Then selected qualifying bots move into a shared 12-hour live final. The submitted code stays the same. The organizer operates separate sponsor-funded accounts and handles credentials, execution and risk controls. Venue approval and funding are still pending.

I chose 12 hours to test whether the bot could keep contributing over a session. One clever trade after three hours might be profitable, but profit alone wasn't the requested outcome. The task was to sustain genuine completed trading activity within a loss budget.

Smaller losses leave more money for later trades. They help the bot keep going; they aren't a separate prize for doing almost nothing. The proposed rewards combine a capped payment tied to counted volume with placement prizes, after qualification checks.

The measurement has to be specific: count completed buys and sales from venue records, exclude self-trades and trades between entrants, and don't award anything for merely submitting orders.

I would still be careful about calling this proof of better liquidity. It measures trading activity. To claim a liquidity improvement, we'd also need to show that other users can trade more easily—for example, through better available prices or less price impact.

A research agent gave me a different evaluation problem

For a company-research challenge, the request was to find as much useful public information as possible and keep it current.

Here I had a starting company list. What I didn't have was a complete answer sheet. How could I tell an entrant what percentage of the available information they'd found?

The scoring design combines findings from submissions and our own crawlers into one checked reference collection. We verify sources, remove duplicates and reject facts attached to the wrong company. Everyone is compared against the same collection; a new verified finding means reassessing everyone.

Even then, I have to distinguish finding something about many companies from finding most of the facts about those companies. An agent that finds one job at each company behaves differently from one that finds every job at a smaller set of companies. The scoring weights make that tradeoff explicit.

And if all our collectors miss a source, it stays missing from the answer sheet. The result is coverage against a checked collection, not a percentage of the entire web.

I’m learning to try the task before writing the scoring rules. It makes it easier to tell another builder what to build, how to test it, and what result would justify a reward.

Trying to build something and need a hand? Or facing a problem in your day-to-day work that you think an agent could help solve? Reach out or share it in the comments. Maybe we can work through it together: figure out what a good result would look like, how to measure it, and whether an agent can deliver it.