The latest articles on DEV Community by alethios000 (@alethios000). https://dev.to/alethios000 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3863971%2Ffa8724f6-9d35-4ceb-b844-3480d12a2c62.jpeg https://dev.to/alethios000 en alethios000 Mon, 06 Apr 2026 13:15:48 +0000 https://dev.to/alethios000/i-built-an-open-source-benchmark-that-scores-ai-agents-not-models-36aa https://dev.to/alethios000/i-built-an-open-source-benchmark-that-scores-ai-agents-not-models-36aa <p><a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F3wxpxqqbttb36cb4xura.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F3wxpxqqbttb36cb4xura.png" alt=" " width="800" height="421"></a><br> Two agents built on the same GPT-4o can have wildly different reliability. But every benchmark only evaluates the model.</p> <p>So I built Legit — an open-source platform that scores the agent as a whole.</p> <h2> How it works </h2> <p><code>pip install getlegit<br> legit init --agent "MyBot" --endpoint "http://localhost:8000/run"<br> legit run v1 --local</code></p> <p>36 tasks across 6 categories (Research, Extract, Analyze, Code, Write, Operate). Two scoring layers:</p> <ul> <li>Layer 1: deterministic checks, runs locally, free</li> <li>Layer 2: 3 AI judges (Claude, GPT-4o, Gemini), median score</li> </ul> <p>Agents get an Elo rating and tier (Platinum/Gold/Silver/Bronze).</p> <p>Free, Apache 2.0.</p> <p>GitHub: <a href="https://github.com/getlegitdev/legit" rel="noopener noreferrer">https://github.com/getlegitdev/legit</a></p> <p>Would love feedback on the scoring methodology!</p> ai opensource python agents