The latest articles on DEV Community by Mucahit Akın (@mucahitakin). https://dev.to/mucahitakin https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4103908%2Ff3fa7ee3-6b77-4246-b3b2-463c13ef6ba3.jpg https://dev.to/mucahitakin en Mucahit Akın Tue, 01 Sep 2026 08:04:15 +0000 https://dev.to/mucahitakin/open-source-llm-observability-5181 https://dev.to/mucahitakin/open-source-llm-observability-5181 <p>Hello, I am sharing the content I’ve released as open source. Anyone who wishes to evaluate it can download it for free—it’s completely open source—and integrate it into their systems. This is also to avoid paying for observability during my LLM development process.</p> <p>Local, open-source LLM observability and testing. Squirrel is a desktop app (macOS / Windows) with a headless server mode: trace your AI app's runs, turns, tool calls, token usage and feedback scores — on your own machine, with your data never leaving it.</p> <div class="ltag-github-readme-tag"> <div class="readme-overview"> <h2> <img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"> <a href="https://github.com/Mucahitakin" rel="noopener noreferrer"> Mucahitakin </a> / <a href="https://github.com/Mucahitakin/squirrel" rel="noopener noreferrer"> squirrel </a> </h2> <h3> Local, open-source LLM observability and testing — desktop app + tracing SDK </h3> </div> <div class="ltag-github-body"> <div id="readme" class="md"><div class="markdown-heading"> <h1 class="heading-element">Squirrel 🐿️</h1> </div> <p><strong>Local, open-source LLM observability and testing.</strong> Squirrel is a desktop app (macOS / Windows) with a headless server mode: trace your AI app's runs, turns tool calls, token usage and feedback scores — on your own machine, with your data never leaving it.</p> <ul> <li> <strong>Runs &amp; Traces</strong> — every turn with its tool-call chain (<code>llm → tool → llm</code>) measured durations, errors and events on a timeline.</li> <li> <strong>Live Test</strong> — run a dataset against <em>any</em> chat endpoint straight from the app: pluggable auth (API key header, Bearer, session cookie, Basic, custom header, or a refresh-token flow that renews on 401), custom headers/body.</li> <li> <strong>Monitoring</strong> — success rate, token usage and latency charts per run project or conversation thread.</li> <li> <strong>Threads</strong> — read any conversation end-to-end as chat bubbles.</li> <li> <strong>Datasets</strong> — build test sets from CSV/JSON/JSONL or from scratch; compare two runs turn-by-turn.</li> <li> <strong>Projects &amp; API keys</strong> — HMAC-signed per-project…</li> </ul></div> </div> <div class="gh-btn-container"><a class="gh-btn" href="https://github.com/Mucahitakin/squirrel" rel="noopener noreferrer">View on GitHub</a></div> </div> ai webdev programming productivity