Every week there is a new model launch and yet another benchmark released in the wild. But they do not help product builders evaluate how well their products can be discovered and used by these agents and models, or talk about actual tasks their users would perform. Most benchmarks today exist in curated environments and do not translate well to the real world.
We built Oqoqo to bridge this gap. Oqoqo makes it super simple to build realistic evals and custom benchmarks for tasks users actually care about.
With Oqoqo, you can define tasks as simple as a prompt your user might give to an agent e.g. “integrate supabase to my webapp to store user sign ups”, provide what you want to test for e.g. Supabase SDK, API, CLI etc. and define what success looks like e.g. “must set up RLS”.
We handle the rest. Our infrastructure spins up isolated sandboxes, executes the tasks against agents of your choice, catalogs every single step the agents take including tool calls, retries, discovery loops etc, and documents token consumption, cost, along with evaluating success/failure based on your success criteria.
With Oqoqo you can:
- Reliably measure how agent-friendly your product surfaces are against Codex, Claude Code, OpenClaw, Hermes, Pi, Opencode, Cursor, GitHub Copilot
- Regression test MCP, CLI, skills, SDK, and any agent-facing interface (we are continuously using Oqoqo to dogfood and improve our own MCP/CLI)
- Create and share custom benchmarks for how agents discover and use your product
- Compare models and harnesses for domain-specific tasks
- See whether new versions improve agent experience
We built Oqoqo for teams building products that agents want to use, and for teams putting agents into day-to-day work.
And the best thing? Your agent can handle the setup for you ✨, try it out for free today: https://oqoqo.ai/
We would love to learn what kind of experiments you would like to run and what questions you have about agent interactions and agent experience.
Top comments (1)
The "integrate Supabase into my web app" task paired with "must set up RLS" shows why generic coding benchmarks miss the product question: can an agent discover the right surface and use it safely? Capturing tool calls, retries, discovery loops, token use, and cost inside isolated sandboxes should make failures much more actionable than a single pass/fail score. I'd also split each benchmark into discovery, execution, and verification stages, because a broken CLI help path and a model reasoning error need different fixes even when they produce the same failed task.