Trying to turn everyday problems into measurable outcomes

Soham Sinha — Sat, 12 Sep 2026 17:06:57 +0000

I’m building Builderr around a simple hypothesis: if I can turn a business problem into a clear, measurable outcome, I can invite builders from around the world to come up with solutions.

Companies pay for verified outcomes. Builders know what they’re trying to achieve and how their work will be assessed.

I’ve been working through what this takes in practice. Here are two examples.

A friend of a friend is building a prediction market. We got talking about what he needed: more trading activity and liquidity on the platform.

To turn that into a challenge, I first needed data builders could test with.

I had to figure out what market data and API access were available, and how builders could use them.

Researching APIs gave me a way forward. Polymarket—a separate platform from his—publishes APIs and SDKs, for example. I still needed to turn data access into something builders could test against.

For the proposed challenge, I settled on a small interface: give the bot current market data, its account state and the limits, then ask it to buy, exit or wait. The evaluator would handle the connection to the venue.

I needed a baseline before I could set a bar

I spent a couple of days researching approaches and building a simple bot. Could a basic version do something useful? What would it cost?

That gave me a starting point for the qualification threshold: what could a basic implementation achieve, and what would be useful to him?

I expect builders to improve on it. A simple bot is fine if it delivers the result.

There's still a calibration step here. A short run wouldn't prove that a bot can sustain the proposed 12-hour threshold. The baseline needs to run under the same conditions entrants will face before that bar is final.

I didn't want entrants to fund the experiment

The next question was how to evaluate submissions in a real market without making every developer open and fund an account.

The proposed process has two stages. First, builders develop against a test API using real-time market data, and we run their submissions there. The bot has to complete its trades, respect the limits and keep working without manual help.

Then selected qualifying bots move into a shared 12-hour live final. The submitted code stays the same. The organizer operates separate sponsor-funded accounts and handles credentials, execution and risk controls. Venue approval and funding are still pending.

I chose 12 hours to test whether the bot could keep contributing over a session. One clever trade after three hours might be profitable, but profit alone wasn't the requested outcome. The task was to sustain genuine completed trading activity within a loss budget.

Smaller losses leave more money for later trades. They help the bot keep going; they aren't a separate prize for doing almost nothing. The proposed rewards combine a capped payment tied to counted volume with placement prizes, after qualification checks.

The measurement has to be specific: count completed buys and sales from venue records, exclude self-trades and trades between entrants, and don't award anything for merely submitting orders.

I would still be careful about calling this proof of better liquidity. It measures trading activity. To claim a liquidity improvement, we'd also need to show that other users can trade more easily—for example, through better available prices or less price impact.

A research agent gave me a different evaluation problem

For a company-research challenge, the request was to find as much useful public information as possible and keep it current.

Here I had a starting company list. What I didn't have was a complete answer sheet. How could I tell an entrant what percentage of the available information they'd found?

The scoring design combines findings from submissions and our own crawlers into one checked reference collection. We verify sources, remove duplicates and reject facts attached to the wrong company. Everyone is compared against the same collection; a new verified finding means reassessing everyone.

Even then, I have to distinguish finding something about many companies from finding most of the facts about those companies. An agent that finds one job at each company behaves differently from one that finds every job at a smaller set of companies. The scoring weights make that tradeoff explicit.

And if all our collectors miss a source, it stays missing from the answer sheet. The result is coverage against a checked collection, not a percentage of the entire web.

I’m learning to try the task before writing the scoring rules. It makes it easier to tell another builder what to build, how to test it, and what result would justify a reward.

Trying to build something and need a hand? Or facing a problem in your day-to-day work that you think an agent could help solve? Reach out or share it in the comments. Maybe we can work through it together: figure out what a good result would look like, how to measure it, and whether an agent can deliver it.

AI assistance disclosure: I used AI to help research sources, organize and edit this article from my challenge-design notes.

DEV Community: Soham Sinha

Trying to turn everyday problems into measurable outcomes

I needed a baseline before I could set a bar

I didn't want entrants to fund the experiment

A research agent gave me a different evaluation problem