DEV Community: Favur

What it takes to run an agent unattended for two days

Charles Solar — Mon, 20 Jul 2026 16:22:28 +0000

Favur ran unattended for two days on Meta Muse Spark 1.1, working from a Statement of Work for a 2048 game. It produced 131 files with tests across six sessions, with no human intervention at any point.

Unattended runs past a few hours fail in a small number of ways. A better model raises the ceiling on output quality but removes none of them, because they are properties of running a model in a loop for a long time rather than properties of the model. This covers what the harness does about each, including one it does not address.

False completion

A step reports success without having done the work. In a supervised run this costs a minute. In an unattended run downstream steps build on the result and the error surfaces hours later, past any clean rollback point.

Completion is gated on artifacts rather than on reports. Each step type has exactly one tool that can advance it, and validators run before the step closes.

The validators are the substantive part. Verification must cite at least two context documents. A commit is rejected without a commit hash and a non-empty list of committed files. Acceptance criteria are returned as structured fields and machine-checked rather than as prose asserting success. A claim is not accepted anywhere an artifact is available, because a model can assert anything but cannot produce a hash for a commit it did not make.

Work and judgment are separated. The agent running the development workflow writes no code; it sequences sub-agents and enforces the gates. Test creation is a distinct delegated step ahead of implementation, and the implementing step exists to make those tests pass. Failed verification routes back to implementation rather than forward.

Non-convergence

The failure that ends long runs is not a crash, which is detectable, but a loop, which presents as normal operation. The process is live, requests are being issued, tokens are being spent, and output is being produced.

The common form is not unbounded retry but oscillation, where an agent alternates between two candidate solutions that are both wrong and never terminates.

This is bounded at three levels. Each step carries a retry budget; exhausting it marks the step failed rather than retrying, and the terminal state cannot return to pending, so a failed step cannot re-arm and restart the cycle. A failed step routes to a declared failure path rather than terminating the run. Where a step can route back to an earlier planning stage, re-entry is counted and reaching the cap escalates. Each agent additionally carries circuit breakers counting cumulative failures across the whole run, which separates repeated failure of a single step from the same error recurring across different steps.

Context exhaustion

Running out of context partway through a unit of work leaves a half-implemented feature that the next session cannot safely resume, and the usual outcome is discarding work that was nearly complete.

Conversation history is summarized or dropped in tiers before each request as it approaches the model's context limit, escalating with fill level. Near the ceiling a request is hard-blocked rather than issued in a state where it cannot complete.

Task drift

Task drift is the one failure here the harness does not address. There is no semantic check on whether output remains relevant to the original task, and nothing evaluates the agent's work for topical relevance.

The constraints on drift are structural. A step cannot advance until its validators pass. The advancing tool is fixed per step type. Step requirements are injected on entry, before output is produced, so the evidence a step owes is known in advance rather than assessed afterwards. Strategy knowledge from completed planning steps persists for the remainder of the run, which removes re-derivation of settled decisions as a drift source.

Review quality

The practical failure in review is not that an automated reviewer approves bad code. It is that human review of a large generated diff degrades to skimming, so the verification burden moves rather than disappearing when code generation gets cheap.

That improves only if the automated review is trustworthy enough to rely on, which requires the reviewer to carry a cost for being wrong in either direction.

The reviewer must read the files before it can produce findings. Every finding carries a severity. A false-positive circuit breaker trips after three findings that are not real issues, which prices over-flagging rather than leaving it as a free hedge.

Concealed capability gaps

Capability-justified abandonment carries no scoring penalty. An agent that halts and reports a missing capability is not marked down for halting.

The reasoning is the incentive structure rather than the mechanism. Penalizing honest abandonment causes agents to conceal gaps, and a concealed gap is filled with fabricated work.

State across sessions

A context reset begins a session with no memory of prior state. Over a two-day run this means half-implemented work that is not identified as such, and a new agent resuming from a state it cannot interpret.

This run spanned six sessions. Checkpoints are written periodically and immediately on any agent completion or failure, and that state persists across session boundaries, so each session resumed with the set of passed and failed steps intact. Checkpointing does not block the run.

Output

The SOW specifies a creative variant rather than a clone of 2048, so the theme was model-selected. Tiles run from cool to unstable.

The run launched the built game and played it, producing moves, merges and achievement events, rather than reporting the test suite green. On an unattended run that is a stronger completion signal than a passing suite.

The repo is open source at https://github.com/awesoftsolutions/idea_meta-muse-2048-game

The full run recording is at https://www.youtube.com/watch?v=o0fQ7WQRTjo

The replay can be driven and stepped through at https://favur.dev/go/devto/the2048

We sent 67.9M tokens to Muse Spark 1.1. It cached all but 4 per request.

Charles Solar — Fri, 17 Jul 2026 13:00:00 +0000

Meta's Muse Spark 1.1 showed up on OpenRouter on July 16. We handed it a Statement of Work the same day and let it build something end to end — not a benchmark prompt, a whole software project: spec, plan, pseudocode, code, review, tests, build, sprint review. One model in every agent role.

It burned through 67,923,486 input tokens getting there.

67,918,878 of them were served from cache.

That's a 99.9932% cache hit rate.

The part I can't explain

Here's where it gets strange. The uncached remainder isn't noise. It's 4,608 tokens across 1,152 requests — exactly 4 tokens per request.

Not "about 4." Our run telemetry breaks every request down by agent role and workflow step, which gives 115 separate buckets. I checked all of them:

Role / step	Input tokens	Cached	Delta	Requests	Delta ÷ req
orchestrator / determining_status	32,540	32,536	4	1	4.0
orchestrator / architecture	453,221	453,189	32	8	4.0
orchestrator / sprint_review_gate	3,023,667	3,023,579	88	22	4.0
health-check / code_implementation	34,984	34,972	12	3	4.0
develop / completion	915,234	915,158	76	19	4.0
sprint-plan / consolidation_review	1,966,978	1,966,862	116	29	4.0

All 115 buckets. Zero deviation. A one-request bucket and a 29-request bucket both land on exactly 4.0.

That is not what I'd expect a cache to look like from the outside. Each turn in an agent loop appends a different amount of new content — a tool result might be 40 tokens or 4,000. If the cache were matching a prefix and paying full price for the genuinely new suffix, the delta should wobble all over the place. It doesn't wobble at all.

So there are two readings, and from outside the API I genuinely can't distinguish them:

The cache is that granular. It matches to within 4 tokens of the end of every prompt, and the 4-token tail is some constant that always differs — a role marker, a turn boundary, something structural.
cached_tokens is derived, not measured. The provider reports prompt_tokens - 4 and the real physical behavior is something else entirely.

If someone from Meta or OpenRouter reads this and can say which, I'd love to know — genuinely, no gotcha. I'll update the post.

Here's what makes it more than trivia, though: the bill is computed from this number. Our cost reconciles to the cent against it — uncached tokens at the list input rate, the other 67.9M at the cache read rate, output on top. Whatever cached_tokens physically represents, it's what we're charged on. The economics are real even if the mechanism is opaque.

99.99% cached is not 99.99% cheaper

This is the part I'd want to flag for anyone budgeting an agent workload, because the intuition is a trap.

A cache read isn't free. On OpenRouter's posted rates, a Muse Spark cache read costs about 12% of a list input token. So a fully cached input bill doesn't go to zero — it goes to roughly an eighth.

And output never caches at all. Every token the model writes is billed at the full output rate, forever. In our run, output was under 1% of total tokens but a meaningful slice of the bill, because output pricing is several times the input rate.

Put together, caching absorbed about 85% of what this run would have cost with no caching at all. That's a huge number. It is also not 99.99%, and the gap between those two figures is entirely made of "reads cost something" and "output doesn't cache."

For reference, OpenRouter's own docs describe prompt caching as typically landing 60–80% cheaper than list price. 85% is at the top of that band, not off the chart.

Why an agent pipeline gets to be this cache-shaped

We didn't do anything clever to earn this. We did one boring thing consistently.

Every LLM request our orchestrator makes is split in two: a stable prefix and a volatile tail. The prefix — identity, directives, scoring philosophy, the anti-pattern catalog, the completion signal — is byte-identical on every single cycle. The tail — the fleet snapshot, per-agent activity, the inline rubric, prior feedback — is rebuilt every time.

The invariant is enforced by the function signature rather than by discipline: the prefix builder takes no per-cycle arguments at all. It can't accidentally interpolate a timestamp or an agent ID, because it isn't given any. Byte-identical by construction, not by code review.

That design predates this model. It was written because caching is the difference between an agent loop being viable and being a science project — a long-running loop re-sends its entire accumulated context on every turn, so the prefix gets paid for hundreds of times. Making it cacheable is the whole ballgame.

There's a compounding effect too: in a multi-turn loop, the volatile tail from turn N becomes part of the frozen history in turn N+1. Today's new content is tomorrow's cacheable prefix. Over 1,152 requests, almost everything ends up on the cached side of the line eventually.

A model that caches this aggressively turns that architectural decision into money. It's the closest thing to a free lunch I've seen in this stack.

What caching didn't do

It didn't make Muse Spark a better engineer, and I'd rather say that plainly than let a headline number carry an impression it hasn't earned.

The run scored 77.2/100 composite. It was genuinely strong where the model's own launch material claims strength — orchestration came in at 9.0/10, and it delegated 9 tasks with a 100% first-dispatch success rate and no re-delegations. Sprint review, also 9.0. Code review, 8.7.

It was rough elsewhere. Scout scored 4.5/10. Pseudocode, 5.8. Sprint planning, 6.9. Those are real weaknesses on a real scorecard, and no amount of cache efficiency touches them.

Cheap tokens make a mediocre plan cheaper. They don't make it a better plan.

The disclosure part

I work on Favur, so take the enthusiasm with the appropriate salt. Favur is an autonomous multi-agent software team — you hand it a Statement of Work and it plans, architects, codes, reviews, tests, and ships a repository. It's closed-source and currently invite-only; the repos it produces are open.

The scores above come from Favur Evals, our public benchmark: the same Statements of Work run through different models inside the same agentic pipeline, scored by a deterministic engine with no LLM judge. It's self-funded and not sponsored by any model vendor, including Meta. Which is worth saying out loud on a post that's this complimentary about a Meta model.

The full Muse Spark scorecard, including the parts that aren't flattering:
https://evals.favur.dev/runs/circles__meta-muse-spark-1.1-7.16.2026?utm_source=devto&utm_medium=social&utm_campaign=muse-cache

More about Favur itself: https://favur.dev/go/devto/muse-cache

And if you want the run-by-run version of this as we go, we post them at https://x.com/favurdev

If you've seen the 4-token thing on another model, or you know what it is, tell me. That's the part I actually want an answer to.

Which AI model actually builds the best software? We built a public benchmark to find out

Charles Solar — Thu, 16 Jul 2026 21:39:46 +0000

Today we're opening up Favur Evals — a public, live leaderboard for a question we kept needing an answer to ourselves: which model should I actually put on a coding job?

Here's how it works. We take a fixed statement of work and hand it to a full team of AI agents — planner, architect, tester, coder, reviewer, builder. Then we run that same job again and again, swapping only the model underneath the team:
all-Qwen, all-OpenAI, all-DeepSeek, all-Gemini, and so on. Same task, same
scaffolding, different brain. Every run gets scored across the whole lifecycle — the code, the tests, the cost, the discipline — not just the final diff.

What you'll find on the board

A leaderboard that refuses to crown one winner.
As I write this, the top run isn't even a single model — it's a two-model mix, Gemini Flash 3.5 on the orchestration-heavy seats with Gemini Flash Lite on eight specialist roles.
Meanwhile the strongest test suites came from an all-OpenAI run, the best
value-per-dollar from all-DeepSeek, and the leanest zero-failure run from
all-Qwen. Different models really are good at different parts of the job — the board makes it visible which part, so you can pick by the seat you're hiring for.

Subject breakdowns. Every composite splits into eight engineering subjects — code quality, test quality, cost efficiency, velocity, tool discipline, effort efficiency, process discipline, deliverables — so you can rank by the thing you care about instead of our blend.

Behavior fingerprints. How each model actually behaves in each seat: cache utilization, reasoning depth, tool cadence, throughput.

Where the money goes. Cost share by agent role across the runs — spoiler: code review is the biggest line item, whichever model is underneath.

One thing worth knowing about the scoring

Nothing on the board is graded by an LLM. Every number recomputes from artifacts the run already produced — lint, complexity metrics, the run's own test results, tool telemetry. Same run output, same score, every time. The full rubric is public, and any score on the board expands into its formula when you click it.

These are relative rankings inside our pipeline, under a versioned rubric — the board reshuffles as the pipeline improves, and that's by design. Treat any single number as a point estimate, not a verdict.

Who's behind it

We build Favur, an autonomous multi-agent software team; these runs execute in Favur's own pipeline. The benchmark is independent and self-funded — no vendor sponsorship, credits, or grants — and every model on the board is doing something genuinely hard.

Go poke around: evals.favur.dev.
New runs land all the time — we post them as they do at @favurdev.