DEV Community: Marcell Havlik

Claude Code for non-coders and non-coding tasks: what works, what doesn't, and what to run for the rest

Marcell Havlik — Tue, 22 Sep 2026 09:59:21 +0000

The Reddit threads asking "anyone using Claude Code for non-coding tasks?" are long because the answer is yes, with caveats nobody puts in the title. We use Claude Code every day. This is a straight account of what it does well outside the repo, where it stops, and what we run next to it. Facts checked against Anthropic's docs on 2026-09-21.

Disclosure: TODO for AI competes with Claude Code for part of the same budget. Where it beats us is stated. Our own case is confined to the last section.

What works

Claude Code is an agent with a shell, a file system and MCP, reachable from a terminal, an IDE, a desktop app and claude.ai/code. Anything that is a file, a command or an MCP server is fair game.

Email triage and drafts. With the Gmail MCP server wired, it reads the inbox, labels, drafts replies. For a founder inbox this is the single most useful non-code job it does.

Calendar prep. Google Calendar plus Slack plus Notion MCP: a "prep me for tomorrow" command that gathers context per meeting while you do something else.

Reports and documents. Anything that starts from files or an API. Weekly numbers from a CSV, a changelog from git log, a summary from meeting transcripts. This is its home turf and it is very good here.

Data chores. Renaming, reformatting, merging spreadsheets, deduping a contact list. Faster than writing the script yourself, because it writes the script.

The common thread: the data is reachable from a shell on the machine Claude Code is running on, or from an MCP server it has been given.

Where it breaks

Browser work needs you at the machine. Anthropic ships the Claude in Chrome extension, so Claude Code can open tabs and drive your logged-in session for the CMS admin, the ad account or the supplier portal. It works, and it pauses for you on logins and CAPTCHAs. But it needs a direct Anthropic plan and Chrome running next to the session, on your own machine. A cloud session runs on Anthropic's infrastructure and has no path to your local Chrome, so the browser half of your work is the half you have to be present for.

Unattended work is repo-anchored. This is the caveat that moved. Cloud sessions and routines do run on Anthropic's infrastructure while your machine is off, triggered by a schedule, an API call or a GitHub event, and they can call your MCP connectors, writes included. But a routine is configured around GitHub repositories and a cloud environment whose network is allow-listed by default, and it pushes to claude/-prefixed branches. Nightly PR review: excellent. Nightly "work the inbox and update the ad budgets": possible through connectors if every tool you need has one, impossible for anything that only exists behind a login.

MCP tax. Every SaaS tool without a CLI is a server to install, authenticate and keep working. Gmail, Calendar, Slack, Notion: four setups before the "chief of staff" demo works. Fine once; painful across a team.

One vendor. Claude only. When Claude is the wrong model for a job (cheap bulk classification, a second opinion on its own output), there is no switch.

Still developer-shaped. The desktop app and claude.ai/code removed the terminal requirement, but the surface is still repos, branches and diffs. The person on your team who would benefit most from email triage is not going to connect a GitHub repository to get it.

What we run next to it

TODO for AI is built around the gaps above and not around the strengths: Claude Code's terminal workflow is more mature than ours and its code output is excellent, and the numbers, including where we score higher and why that is not a like-for-like comparison, are on the comparison page. The setup that works for us: Claude Code stays open for the repo. Everything that touches an inbox, a browser or a schedule goes on the TODO list, where it runs on a persistent cloud VM or a paired machine, on whichever model fits, from the web or desktop app. Browser items still need Chrome open on your side; what that reaches is in the extension post.

Short version

Task	Claude Code	TODO for AI
Fix the bug, open the PR	Excellent, most mature workflow	Yes, on your PC or its VM
Email triage via MCP	Yes, after setup	Yes, via your browser or MCP
Post to the CMS, run ads, update the CRM	With Claude in Chrome or an MCP connector, while you are there	With our Chrome extension or a CLI, while you are there
Unattended overnight	Yes: repo-anchored cloud session plus connectors	Yes, on a persistent VM; browser items wait for Chrome
Switch models per task	No	Yes
Non-developer can use it	Not really: repos, branches, diffs	Yes

If your non-coding tasks are files, APIs and the occasional browser job while you sit there, Claude Code already does them. If the unattended half of your work is not a repository, or you want a second vendor's model reading the first one's output, that is what TODO for AI is for.

How an AI agent works in your logged-in browser: extension vs CDP relay vs cloud browser

Marcell Havlik — Tue, 22 Sep 2026 09:54:15 +0000

Plenty of the work a business does in a browser has no usable API: the ad manager's bulk editor, the CMS admin, the supplier portal, the bank. If an agent is going to do that work, it needs a browser that is logged in as you. There are three ways to get one. They are not equivalent.

1. Cloud browser

A headless Chrome on someone else's server. Browserbase, Firecrawl, Manus and most "computer use" demos run this way.

Reach: anything public. For your accounts it starts logged out, so you either paste credentials into a vendor's machine, sync cookies to it, or run a login flow with 2FA each time. Some vendors persist the session after the first login.

Leaks: your session lives on their infrastructure. Whatever their retention policy says, that is where the cookie is.

Stops at: anything that fingerprints a datacenter IP, anything that wants your device (passkeys, hardware 2FA), and any site whose terms forbid automation from an unknown machine.

Good for scraping, research and parallel jobs with no personal login. For "reply to the three support tickets in my Zendesk" it works only once your Zendesk session lives on their server.

2. CDP relay to your local Chrome

Chrome DevTools Protocol is the wire Chrome exposes for debugging. Start Chrome with --remote-debugging-port, forward that socket to a server, and a remote agent drives that Chrome. This is how Playwright, Puppeteer and most "connect to my browser" tools work.

One catch people hit: since Chrome 136 the debugging flags are ignored for the default profile directory, so --remote-debugging-port alone gives you a fresh profile with none of your logins. Getting your real sessions means copying the profile to a --user-data-dir you pass explicitly, or going through an extension instead.

Reach: with your profile attached, everything you can reach. Same cookies, same extensions, same device for passkeys.

Leaks: the relay sees every page, every DOM, every keystroke it sends. Whoever holds the relay endpoint holds your browser. The token in that URL is a full-session credential and has to be treated like one.

Stops at: Chrome not running. Raw CDP on the debugging port is also all-or-nothing: the protocol has no "this tab only" or "read but do not type", and Network.getAllCookies returns the whole jar, so any scoping has to be built on top of it.

Powerful, and the right answer when the agent already runs on your machine. Risky when the agent runs on a vendor's machine and the relay goes across the internet.

3. Browser extension

Code that runs inside your Chrome with the permissions the extension manifest declares. The agent sends intents; the extension executes them locally.

Reach: whatever the extension's code lets through, on the tabs it is allowed to touch, and whatever its manifest permissions allow. An extension that requests cookies can read cookies; one that requests debugger gets the same CDP surface as a relay, scoped to the tabs it attaches to. It runs as you, on your device, so logins and device-bound 2FA are available if the extension exposes them.

Leaks: page content goes wherever the extension sends it. A well-built one sends the accessibility tree or a screenshot of the working tab, not your whole browser. The scope is inspectable: it is in the manifest and in the Chrome permissions dialog.

Stops at: the same place a CDP relay does. Chrome has to be running, so nothing happens while your laptop is closed. Chrome also blocks extensions on browser-internal pages and the Web Store.

Narrower than raw CDP if it is built that way. That is the feature, and it is the extension author's choice, not the transport's.

How TODO for AI does it

The Chrome extension is a hybrid of 2 and 3, and it is worth being precise about which parts are which.

CDP-shaped commands, through the extension. The agent speaks a CDP subset over a WebSocket to our API, and the extension executes it. Two modes, and the difference matters: the default shim mode serves a fixed list of page-level methods (DOM, Accessibility, Input, Runtime.evaluate, screenshots) by injecting an engine into the page, with no debugger banner and no browser-level domains; debugger mode forwards commands to Chrome's debugger API for full fidelity, banner included, and then the reachable surface is CDP itself.
Attached tabs only. You click "attach" on a tab in the extension popup. The agent can control those tabs and any new tab it opens itself. A command aimed at an unattached tab is refused in both modes. Debugger mode plus the explicit allow-unattached setting lifts that, and then an agent that names a tab id can reach any tab. Chrome internal pages and the Web Store cannot be attached at all.
Cookies: no extension cookie API, but do not read that as a boundary. The extension does not request the cookies permission. In shim mode the agent reads what a script on the attached page could read, document.cookie included. In debugger mode CDP goes to Chrome as-is, and the Network domain is broader than one tab: Network.getAllCookies is a whole-browser read. The boundary is the mode you pick and the permission you give the agent, not the missing cookies permission.
Page content crosses the wire. Snapshots, DOM and screenshots of attached tabs go to the API so the agent can see them. That is the leak surface. The attach list bounds it, unless you turn on allow-unattached in debugger mode, which lets an agent name any tab.
Agent-level permissions and a stop button. Browser access is a capability you set per agent to allow, ask-first or block, and the default for a new agent is allow. Ask-first gates each tool call, not each outcome, and the shell is a separate tool that can also reach a browser, so gate both if that matters. Any run can be halted mid-task; the tab stays where it was.
Cloud browser for the rest. Research that needs no login runs in a per-account cloud browser, so your Chrome is free and your laptop can be closed.

What it does not do: run while Chrome is closed. There is no per-site block list and no read-only mode; the attach list, the mode, and the per-agent permission are the scope controls. If a site must never be reachable, do not leave it attached.

Which one, in one table

The last column is our extension, not extensions in general; another extension can make different choices.

	Cloud browser	CDP relay	TODO for AI extension
Uses your existing logins	Only after you hand them over	Yes, with your profile attached	Yes
Passkeys, hardware 2FA	No	Yes, you are at the device	Yes, you are at the device
What crosses the wire	The whole session	Everything in the browser	Page content of attached tabs
Scope control	n/a	All or nothing	Attached tabs, shim vs debugger mode, per-agent permission
Runs while laptop closed	Yes	No	No, Chrome has to be running
Datacenter IP blocks	Often	No	No

If the task needs you, use the extension. If it needs nobody, use the cloud browser. A relay is for when the agent is already on your machine and you trust every hop. What this means for a to-do list that actually closes items is in the to-do post.

Our task categorizer routed 46% of tasks. A $0.04/MTok decision model routes 97%.

Marcell Havlik — Tue, 22 Sep 2026 09:54:08 +0000

Cosine similarity between a task title and a group description is not a classifier.

Bench, cases, raw verdicts: backend/bench/categorization in the TODOforAI repo. Total
cost ~$0.50.

Setup

TODOforAI boards have groups (SEO, Paid, Frontend…); new tasks get a suggested
group. V1: embed title and group name + description, nearest cosine wins. Embedder
Qwen/Qwen3-Embedding-4B via DeepInfra, 512 dims, unit-normalized, query instruction
"Given a task, retrieve the workstream whose purpose best matches the task". Abstain if
best < 0.50 or margin to runner-up < 0.06.

100% precision on 18 hand cases. Most of my real board in Unsorted.

Judges

Same 138 titles, same 8 groups, same description text, no example tasks — exactly what
production sees. Titles written like real todos: short, half Hungarian, 15 nonsense
(call mom, asdf), 5 ambiguous.

embedding — production, unchanged
jev — TypeSafe Jev, non-autoregressive decision model on Vercel AI Gateway. One choice question, 8 groups + none. $0.04/MTok in, output free.
sonnet — Claude Sonnet 4.6, temp 0, one slug or none
opus — Claude Opus 5, same prompt. The truth. My labels are one more opinion.

Trap: Opus 5 reasons before answering; max_tokens: 20 looked like abstaining on 88/138.
Give it 400.

Numbers

Judge	agrees with Opus	wrong group	missed	138 tasks
embedding	77/138	1	60/112	8 s
jev	130/138	5	3/112	24 s
sonnet 4.6	131/138	7	0/112	57 s

Opus routed 112, called none on 26 (15 nonsense + 11 vague). Embedding: wrong once,
missed 54%. Jev: 97% routed, 5 wrong. Sonnet's 7 wrong are coin flips with Opus.
On my real board Jev and Opus agree 21/26.

Why the embedding abstains

Not the margin:

                         agree   wrong   missed
margin 0.06  floor 0.50    76      2       60     ← production
margin 0     floor 0.50    79     41       18
margin 0     floor 0       76     62        0

Drop the gate and Unsorted becomes wrong group. Top-1 is wrong on ~45% of tasks.

bun test flaky on CI          frontend 0.540   development 0.537
call mom                      email 0.540      plg 0.525
asdf                          development 0.663  frontend 0.659

Scores sit in 0.42–0.72; asdf outscores a real task, so no floor finds nonsense.
Frontend/Development, Paid/SEO, Enterprise/Paid overlap, so real tasks sit inside any
margin. Jev and the LLMs know outcomes: a CI flake is Development because of what fixing
it achieves. Similarity can't say that.

What shipped

Jev routes; embedding is fallback when the gateway is unset or down. Batches of 25,
none explicit, accept probability ≥ 0.5. ~0.3 s per task.

Not Sonnet: 57 s vs 24 s and ~100× the price, for a suggestion 400 ms after you stop typing.

Router	per 1,000 tasks	per 1M tasks
embedding (~15 tok, $0.01/MTok)	$0.0002	$0.15
jev (~260 tok catalog per question, $0.04/MTok in, out free)	$0.01	$10
sonnet 4.6 (~350 in / 10 out)	$1.20	$1,200
opus 5 (~350 in / ~200 out with reasoning)	$7	$7,000

List prices, 8 groups. Jev costs ~70× the embedding and routes twice as much.

Jev's 5 wrong are channel-vs-outcome (sponsor a newsletter, measure signups → email,
not paid). Confidence is lower when wrong (0.70 vs 0.94), but floor 0.6 trades 2 wrong
for 7 Unsorted. We keep the 5.

Takeaways

Precision alone lies. Print recall against something you trust.
Score against a strong model, not your own labels.
Check the reference's output budget. Truncated looks cautious.

138 cases is small. It's enough to see a 54% recall gap; that's the only claim.

94.7% on LoCoMo — and why most of the gap between published memory numbers isn't the memory

Marcell Havlik — Tue, 22 Sep 2026 09:46:31 +0000

94.7% on LoCoMo at 5.0K context tokens — 95.0% when re-graded by Mem0's own judge. The
more useful finding is what we measured on the way there.

The number, with its configuration attached

94.7% on LoCoMo, all 1540 questions, at 5.0K context tokens per question. It is
the highest LoCoMo number we are aware of, published or otherwise, and we can hand you the
code and the per-question outputs for it
(github.com/todoforai/livemem).

The nearest published row is Mem0's April 2026 algorithm at
92.5, delivered in 6,956 mean context tokens;
then MemMachine's 91.7% for v0.2. So: +2.2 points on about 30% less context.

Per category: open-domain 97.0%, single-hop 93.6%, temporal 93.5%, multi-hop 81.2%.
That last one is where the remaining work is, and we'd rather print it than average it
away.

How much of that is the grader? This is the first question we would ask, so we
answered it: we re-graded the identical 1540 answers with gpt-4o-mini running Mem0's own
ACCURACY_PROMPT — the judge model and prompt behind their published LoCoMo numbers — and
got 95.0% (+18 flips to correct, −13 the other way). The two judges agree within 0.3
points on these answers. The lead is not a grading artefact.

How much of it is the ensemble? The answers come from two passes over the same
retrieved memory: gemini-flash and claude-haiku-4-5 answer the identical 5.0K block
independently, then a rule hierarchy (agree > commit > abstain) settles the 82
disagreements. One pass alone scores 92.4%. So +2.3 points are answer-side, bought with
a second model call and ~6 seconds per question instead of one — not with more context.

That distinction turns out to be the theme of this entire post.

And before anyone else says it: LoCoMo at 94.7% is a nearly saturated benchmark. Mem0
say so themselves
— "older benchmarks like LoCoMo are less informative today," because they don't stress
conflicting updates, identity ambiguity or long-running workflows. We agree, and we still
publish the row, because a saturated benchmark is exactly the regime where the unreported
variables — answerer, judge, context budget — decide the ranking. Two points of headroom
and a seven-point answerer effect is not a leaderboard; it's a measurement problem. The
rest of this post is that measurement problem.

The setup

We build TODOforAI, an agent that runs long-lived tasks on your
machine. Long-lived means it has to remember: what you decided three weeks ago, what your
deploy procedure is, which of two conflicting instructions is the newer one.

So we built a memory system, and then we did the thing you're supposed to do: benchmarked
it against published numbers instead of trusting our own demos.

Two public datasets:

LongMemEval_S — 500 questions over long, multi-session chat histories. Single-session recall, multi-session aggregation, temporal reasoning, knowledge updates.
LoCoMo — 1540 questions, heavier on temporal and multi-hop reasoning.

Both run through AMB, an
independent harness that fixes the dataset, the answering model and the judge for every
memory provider it runs.

LoCoMo went well from early on. LongMemEval was the humbling one, and still is: our
first honest full-dataset number was 85.4%, against Mem0's published 94.4%. We are
at 87.8% today — better, still behind, and behind at 4.2K context tokens against their
6.8K, which is the only part of that comparison we're happy with.

That 9-point hole is what sent us looking for bugs, and the bugs turned out to be more
interesting than the number. It is also why the rest of this post spends more time on
LongMemEval than on the benchmark we lead.

Chasing it

The obvious hypothesis is that their architecture is better. So we went looking for what
we were doing wrong, and we found real things:

Deduplication was deleting evidence. A question like "how many plants did I buy last
month?" needs every instance, not the top-k most similar ones. Our dedup merged "bought
2 basil plants" and "bought 3 basil plants" — near-identical sentences — silently
destroying an instance the count depended on. Making dedup number-aware (two facts with
different digits are never duplicates, however similar the prose) was worth +4.3 points
on a 94-question validation subset (78.7% → 83.0%).

Extraction was too tasteful. Our prompt skipped things "an LLM already knows." Then a
question asked "what was that Borges quote you gave me?" — public knowledge, yes, but
the specific thing we said in that conversation is not in any model's weights. The
lesson generalizes: extraction should be greedy, because a fact never extracted can never
be retrieved. Selectivity belongs to the selector, not the extractor. This fixed three
questions on our subset that had been unanswerable.

Some answers don't live in facts at all. They live in a conversation turn that no
summary preserved. Adding a retrieval configuration that mixes entity cards with 3-turn
conversation-window excerpts alongside the distilled facts took the full 500-question run
from 85.4% to 87.8%.

Real improvements, real points. Still not 94.4%.

The variable we hadn't controlled

Here's what we should have checked on day one. We had been answering with the cheapest
model we could — gemini-flash-lite — deliberately, for the reason above.

Mem0's published harness config uses gpt-5 as the answerer and gpt-5 as the judge.

So we ran the control: same memory system, same extracted facts, same retrieval, same
rendered context. Swap only the answering model.

Config (94-question subset)	Accuracy
flash-lite answerer, flash-lite judge	85.1%
gpt-5 answerer, flash-lite judge	92.6% (+7.4)
gpt-5 answerer, gpt-5 judge, same answers re-judged	90.4% (−2.2)

+7.4 points from the answerer alone. And the judge is a free variable too — though
note the stronger judge was stricter, not more generous: it accepted two hedged answers
we'd been denied and rejected four we'd been given.

Be careful what this does and doesn't show. It does not prove Mem0's 94.4% is "really"
lower, or that the gap between two systems is entirely answerer choice — we never ran
their system on our harness. What it shows is narrower and still useful: on the same
memory, configuration differences of the size that separate published rows are worth
about as many points as the memory architecture itself. Our own comparable-ish number
is 90.4%, still short of 94.4%. We just no longer believe the raw distance between two
differently-configured rows is a measurement of anything.

What we actually learned

1. Cross-harness memory numbers are not a ranking. The clearest evidence isn't even
ours: in the published literature Mem0 appears at 94.4% (self-reported) and at
67.6% (measured in a third-party paper). Same system, 27 points apart, because the
harness, answerer, judge prompt and budget differ. Any table mixing sources — including
the one in our repo — should be read as a
collection of separately-configured experiments, not a leaderboard.

2. Judges are worth points, not decimals — and not only via the prompt. Our
single-pass answers scored 92.4% under claude-haiku-4-5 and 95.6% under gpt-4o-mini,
on near-identical judge prompts. Nothing about the memory changed between those two
numbers; only which model read the same grading instruction. (On the ensemble answers the
same two judges land within 0.3 points of each other — the size of the judge effect is
itself configuration-dependent, which is the point.)

3. Context size is a hidden axis. We run at ~4–5k tokens per question, measured after
rendering. Mem0's new algorithm reports its 92.5 at 7.0K and its 94.4 at 6.8K; the
Hindsight run on the AMB leaderboard reports 43.6k context tokens per question — about 9×
ours. Accuracy per token is a different ranking than accuracy, and only one of the two is
usually reported. Credit where due: Mem0 do report mean tokens next to every score, which
is more than most leaderboard rows offer.

4. Our own most embarrassing finding was in our token accounting. Our "5000-token"
budget was actually delivering 5742 real tokens. Date prefixes (- [2026-08-14]) are
15 characters but 10 tokens — dates are token-dense — and across a full block that
came to roughly 1.6k tokens we never charged for. After enforcing the real budget, delivered
context dropped to 4182 tokens and the score on our subset dropped from 83.0% to 77.7%. A
rerun with the budget raised so delivered context matched the old 5742 recovered to
81.9% — so the drop was the missing context, not lost capability. The uncounted tokens
had been buying real answers. If you don't measure delivered tokens, your budget is
decoration.

5. The dominant remaining error family is aggregation. "How many X in total", "what
percentage of Y". The instances are usually in the retrieved context — the answerer just
doesn't count them reliably. That's not fixed by retrieving harder. It's also a direct
cost of our weak-answerer choice: flash-lite stresses retrieval honestly, but it fails
arithmetic that a stronger model wouldn't. There's no free methodology.

Where LoCoMo stops being enough

Mem0's benchmark post is worth reading precisely because it argues against its own top
row, and its list of failure modes matches ours: paraphrase overfitting (systems pass
when the query echoes the stored wording), temporal revision (the user changed their
mind and both facts are still in the store), identity ambiguity, mixed-granularity
scoping (per-user vs per-project vs per-device), and scale — benchmarks run over
hundreds of events, production over millions.

Their own numbers show the drop: 92.5 on LoCoMo and 94.4 on LongMemEval_S, but 64.1 on
BEAM 1M and 48.6 on BEAM 10M — same system, same year, thirty to forty-five points lower
as soon as the benchmark contains interference, updates and long-horizon chains. That is a
much more honest picture of where memory actually is than any 9x.x% row, including ours.

We have not run BEAM yet. It's next, and we'll publish it the same way — with the answerer,
the judge, the delivered token count and the per-question outputs attached — whatever the
number turns out to be. Our expectation, stated in advance so it's falsifiable: our
multi-hop score of 81.2% is the honest predictor of how we do there, not our 94.7%.

The metrics they suggest reading instead of raw accuracy — recall@K, end-to-end task
success, context inflation, cross-session continuity, stability under paraphrase — are
the right list. We already publish one of them (delivered tokens per question, item 4 below
is what happens when you don't) and we're missing the other four. So is nearly everyone.

The code

We published the reference implementation: github.com/todoforai/livemem

The core is about 200 lines of TypeScript, because the architecture genuinely is small:

conversations ──extract──▶ dated facts + embeddings ──pack──▶ ≤ N-token block
                (1 LLM call,                          (cosine + greedy knapsack,
                 offline)                              no LLM on this path)

The bet the design makes: extract everything once, select per question, never call an
LLM to retrieve. Selection is cosine similarity plus a greedy knapsack under a token
budget — tens of milliseconds — so you can afford to re-select for every single question
instead of maintaining one static "user profile" that's stale by definition.

Our hosted API adds the tuned extraction prompting and the retrieval refinements described
above — number-aware dedup, entity cards, conversation-window units — and the benchmark
rows above were run in that configuration; the repo is the same architecture without that
tuning. The benchmark harness in bench/
runs against either one through the same provider, so you can measure the difference
yourself rather than taking our word for it.

The per-question outputs behind our principal rows — answers, judge verdicts,
context-token counts — are in that repo, each labelled with the answerer, judge and budget
it used. Some intermediate numbers in this post come from validation subsets we ran during
development and did not publish as full artifacts; they're marked as such. Numbers from
other systems are linked to their published source and were not re-run by us. If we got
something wrong, it's falsifiable, which is the only claim worth making about a benchmark.

We know where the next points are: multi-hop at 81.2%, aggregation, LongMemEval_S
where we still trail Mem0 by 6.6 points, BEAM — which we haven't run and expect to hurt —
and the hosted live path still behind our offline one. We'll publish the next number the
same way — measured against this one, with the artifacts attached.

If you want the memory without running it yourself, it's what powers agent memory in
TODOforAI.

voiceloop: the fastest voice agent loop in the browser is now open source

Marcell Havlik — Tue, 22 Sep 2026 09:46:15 +0000

Everybody can now build the best voice agent into their own product. The #1 loop is open
source, and the benchmark that says so is public.

Demo, no keys: todoforai.github.io/voiceloop
Library: github.com/todoforai/voiceloop — npm i @todoforai/voiceloop
Bench: github.com/todoforai/voice-agent-bench

Why we built it

We wanted the fluid JARVIS feel in the browser for TODOforAI — you talk, it answers within a
second, you interrupt it mid-sentence and it just stops. We could not find a stack that did
this properly. Closed APIs were close but not ours; the open frameworks talked over the user,
or worse, heard their own voice through the speakers and cut themselves off.

It is 2026. This should be a solved problem. So we solved it and published the whole thing:
the loop, the numbers, and the rig that produced the numbers.

What voiceloop is

A zero-dependency JavaScript library that runs the full loop in the browser:
VAD → STT → LLM → TTS, with the hard parts already handled.

Real barge-in — triggers on transcribed novel words, not mic energy, so the agent's own voice leaking into the mic never cuts it off.
Self-echo filtering — a word-match filter compares what the mic hears against what the agent is currently saying. 0 self-interruptions in 30 echo-coupled turns with echo cancellation fully off.
First audio under a second — TTS speaks sentence 1 while the LLM is still writing sentence 2, and the LLM call starts speculatively during your end-of-turn pause.
Local-first — Silero VAD and Piper TTS run as WASM in the tab. Free, no cloud round-trip.
Serialized turns — rapid-fire turns, tool results, holds and replays can never talk over each other. Locked in by 178 tests.

Everything is pluggable: any OpenAI-compatible LLM, four STT providers (Web Speech,
ElevenLabs Scribe, Deepgram Flux, Speechmatics), swappable TTS (Piper local, ElevenLabs
cloud, or your own).

import { VoiceAgent, unlockAudio } from '@todoforai/voiceloop';

const agent = new VoiceAgent({
  llmUrl: '/api/chat/completions',   // any OpenAI-compatible endpoint, behind your proxy
  model: 'claude-haiku-4-5',
  persona: 'You are a friendly cooking assistant.',
  onEvent: (e) => { if (e.type === 'assistant') render(e.text); },
});

button.onclick = async () => { unlockAudio(); await agent.start(); };

That is the entire integration.

The numbers

Latency claims in voice AI are usually self-reported and unreproducible. We did not want to
add another one, so we built voice-agent-bench:
a black-box rig. A scripted "person" (byte-identical pre-generated speech) talks into a
virtual mic, the agent's speaker output is recorded, and every score is derived from the
audio alone. No integration needed — any agent that makes sound can be measured, including
closed ones.

Every system gets the same scripted conversations and, where the system allows it, the same
fixed mock LLM (300ms TTFT), so the comparison isolates the voice loop from the model. 5
conversations × 6 turns pooled, n=30, median and p95 — single runs jitter by ±300ms and are
not worth printing.

Smalltalk — same mock LLM, 5×6 turns pooled

configuration	voice→voice	p95	barge-in stop	stalls
OpenAI Realtime (speech-to-speech, own LLM) *	866ms	1644	429ms	20
voiceloop · deepgram + ElevenLabs flash	862ms	1067	944ms	16
voiceloop · deepgram + Piper (free, local TTS)	974ms	1287	1463ms	19
Pipecat 1.8.1 · deepgram + EL flash	1046ms	3573	542ms	14
ElevenLabs ConvAI	1454ms	1632	1042ms	8
voiceloop · EL Scribe + EL flash	1562ms	1855	1566ms	12
voiceloop · Speechmatics + EL flash	1706ms	2069	1046ms	17
voiceloop · webspeech + Piper (zero-key)	2113ms	2607	1257ms	30

Across scenarios

system	clean	hesitation	talked through user	echo	cut itself
OpenAI Realtime *	870	1290	0 (yields 130ms)	790	17/30
voiceloop · deepgram + EL flash	860	1400	0 (420ms)	930	0
voiceloop · deepgram + Piper	970	1400	0	—	—
Pipecat	1050	1290	2 (200ms)	1320	20/30
ElevenLabs ConvAI	1450	1810	0 (490ms)	1410	0

* Realtime is speech-to-speech and can't use the fixed mock LLM, so its row isn't fully
apples-to-apples.

How to read it

Clean audio: voiceloop with Deepgram Flux + ElevenLabs flash is the fastest configuration
we measured, at 862ms median — and its p95 (1067ms) is the tightest in the table by a wide
margin. Pipecat's p95 of 3573ms on the same providers means one turn in twenty takes over
three seconds. The free, fully local Piper path lands at 974ms with no cloud TTS at all.

Echo is the failure that separates the stacks. Feed each system its own voice back
through the mic (−15dB, 30ms delay, no AEC — what a laptop with the speakers on actually
does) and the other fast stacks hear themselves as the user and cut their own replies:
Pipecat on 20 of 30 turns, OpenAI Realtime on 17. voiceloop cut itself zero times and
ran echo-coupled turns at 930ms — parity with clean. Word-level echo filtering costs no
latency once it classifies correctly.

Hesitation: a user who pauses mid-sentence should not be talked over. Every stack except
Pipecat backs off; voiceloop enters 2 of 30 hesitation turns and yields within 420ms.

The zero-key default is honest about its cost. Browser Web Speech + Piper needs no
account anywhere and runs the demo, but it is ~1.2s slower to close a turn than cloud STT
(2113ms). Pick a pipeline STT provider for the numbers above.

Full per-scenario tables, methodology and reproduction steps:
results/RESULTS.md.

Hundreds of configurations, so you don't have to

The rows above are the survivors. Behind them are hundreds of runs across STT providers, TTS
engines, VAD thresholds, end-of-turn debounces, barge-in minimum lengths, prefetch stability
windows and echo-match thresholds. Every knob that mattered is exposed in
src/tuning.js with the
default set to what won on the bench. The edge cases you would otherwise discover one
production bug at a time — the agent interrupting itself, tool calls firing on a sentence
the user was still amending, a hung tool stalling the next turn — are already handled and
regression-tested.

Try it, use it, beat it

Try: todoforai.github.io/voiceloop — 20 seconds, no keys, runs in your tab.
Use: npm i @todoforai/voiceloop — MIT, zero dependencies. Point it at any OpenAI-compatible endpoint and wire in your tools.
Beat it: the bench is black-box and public. If your stack does better, add it — ADDING_A_SUT.md is the contract. We will print the row.

This is the voice loop inside TODOforAI's JARVIS; the integration overhead between the
library and the product is nil, which is exactly the point. Everybody should have the best
voice loop. Star it, share it, contribute — let's keep the best one open source.