Hi, I’m Yu. Earlier this month I released Tura and wrote about why agent harnesses need long-horizon benchmarks, not just one-off demos.
Over the past two weeks, I’ve seen more people publish tests of token-saving tools such as RTK and Ponytail. That’s great, but the results are hard to compare when everyone uses different tasks, logs, and output formats.
So I open-sourced the benchmark framework I use:
https://github.com/Tura-AI/benchmark/tree/main
It can run the benchmark workflow, collect logs and artifacts into one result schema, and let CI index any result added to the repository. The website then turns those results into comparable graphs and detailed run pages.
If you have a tool, harness, or test case you want included, tell me. You can also run the benchmark locally, reproduce the evaluation, and submit the result folder as a pull request. CI handles the indexing automatically.
I’d especially like feedback on what should be tested next and where the framework may introduce bias.
Disclosure: I maintain Tura and this benchmark framework.
Top comments (1)
I appreciate how you've open-sourced the benchmark framework, making it easier for others to compare the effectiveness of token-saving tools like RTK and Ponytail. The ability to collect logs and artifacts into a single result schema and generate comparable graphs is particularly useful. I'm interested in exploring how this framework can be applied to other areas, such as evaluating the performance of different agent harnesses. Have you considered adding support for multiple benchmarking workflows or task types to further increase the framework's flexibility?