DEV Community: Don Karter

Benchmarking AI Coding Agents on Real Pull Requests

Don Karter — Sat, 01 Aug 2026 17:25:42 +0000

No synthetic puzzles, no contaminated suites: 25 tasks harvested from PRs merged in 2026 across five languages, graded by each project's own held-out tests. Four agents at stock settings. octomind with an open model solved 24/25 - ahead of Claude Code with Opus - while the same model in another harness solved 19 at double the cost. Here's how we built the benchmark and what it taught us.

We wanted one number we could actually trust: if you hand a coding agent the kind of task a maintainer faces on a normal Tuesday - a real bug, a real feature request, in a real codebase - how often does it deliver a fix the project's own test suite accepts?

None of the public benchmarks could give us that number, so we built octobench. This post is the story of how, what broke along the way, and what the scoreboard says.

Why not an existing benchmark

Two problems kept biting us with the popular suites.

Contamination. The well-known benchmarks are old enough that frontier models have seen the fixes - sometimes literally, commit by commit - in training data. At one point we caught a model running git show on a commit hash it had no business knowing about. Whatever that measures, it isn't problem-solving.

Taste-grading. Many benchmark tasks are graded by tests that assert an implementer's arbitrary choices: an internal variable name, the exact wording of an error message. An agent can write a maintainer-grade fix and fail because it phrased an error differently than the original author. That measures mimicry, not engineering.

So the design goals were: real tasks, fresh tasks, fair grading.

25 tasks, 5 languages, mostly newer than the models

We harvested every task from a real, recently-merged pull request in a respected open-source project: werkzeug, click, anyio, pydantic, twig, carbon, symfony, guzzle, commonmark, uuid, rayon, chrono, bytes, serde-json, fmt, yaml-cpp, catch2, spdlog, eslint, fastify, undici, pino, pino-pretty.

That list is deliberate on two axes. Languages: five of them - python, php, rust, c++, js - because an agent that's great at python and lost in a CMake build is not a general coding agent. Freshness: every fix was merged in 2026, mostly after current model training cutoffs - several within days of harvesting. One case, chrono's reversed date iterators, was merged the same morning we picked it. A benchmark you can re-harvest as cutoffs advance is a benchmark that can't go stale.

The scenario mix runs from one-line crash fixes to multi-file features, and each case reconstructs the moment before the fix existed:

setup.sh checks out the repository at the commit before the fix - as a single-commit shallow clone, so the answer isn't hiding in the git object store - prepares the toolchain, and removes the git remote so the agent can't just fetch the upstream fix.
The agent gets a task prompt and works in the repo like a hired contractor.
validate.sh runs after the agent finishes: it fetches the merged fix's test files - which the agent has never seen - overwrites whatever the agent may have done to the test suite, and runs exactly those tests. The project's own tests, written by the project's own maintainers, decide pass or fail.
An LLM judge separately grades work quality from the diff and logs.

Before any case entered the benchmark it had to prove itself fail-to-pass: held-out tests must fail on the pre-fix code and pass with the real merged fix applied. No proof, no case.

Writing prompts that don't cheat - in either direction

For 20 tasks we ran a dedicated agent (octomind's developer:reverse-spec) over each merged commit to reconstruct the request that plausibly produced it, then curated by hand with one rule - the derivability rule: everything the hidden tests assert must be derivable from the prompt. Bug fixes got a short informal prompt, because any correct fix passes. Features whose tests pin public API names or exact output formats got a spec-tight version, because a real requester would state a wire format.

For the other 5 - one per language - we used the actual GitHub issue text, verbatim, trimmed only of fix-leaking sections (one author had helpfully written "I have a PR ready that does X"). That tests something different: turning a user-shaped bug report, sometimes with a screenshot instead of expected output, into a maintainer-grade fix.

A few favorites, by the skill they isolate:

anyio's cancel-scope spin - a 4-line fix that requires understanding why an asyncio event loop pins a CPU core. Tiny diff, deep async reasoning.
guzzle's cookie prefixes - 27 lines with three independent traps; the naive startsWith("__Secure-") && !secure fix fails half the tests.
commonmark's fenced-code tabs - a user reported "code blocks sometimes lose their first character" with a screenshot. The held-out fixtures include cases the issue never mentions - only a true root-cause fix passes, a symptom-patch fails.
pino-pretty's control characters - two hidden tests exist purely to punish over-broad sanitizing: they pass on the broken code and fail any fix that doesn't respect the trust boundary.

Eating our own dogfood

Before the final run, three reviewers read every held-out assertion against every prompt. The audit caught real problems - in our benchmark, not the agents. One case's tests asserted the exact prose of an error message the issue never quoted; two frontier agents produced perfect fixes (one even picked the same error-code number as the maintainer) and both failed on wording. We replaced the case. Another case's hidden tests contradicted its own issue text. A third had its held-out tests inside a source file, so restoring gold tests wholesale would silently revert a correct fix.

The meta-lesson: benchmark infrastructure fails in ways that look exactly like model failures. Every anomaly we chased - a judge scoring 0 on a passing run, a "solved" case with an empty diff - deserved a real root-cause, and about half were ours.

The scoreboard

Four agents, each at its stock, out-of-the-box single-agent invocation. No tuning, no custom prompts.

Agent	Solved	Judge Σ / 2500	Cost	Wall Time
octomind + glm-5.2	24/25	2264	$63.43	3.6h
claude code + claude-opus-5	23/25	2262	$81.79	6.7h
codex + gpt-5.6-sol	21/25	2127	$14.86	1.0h
opencode + glm-5.2	19/25	2093	$129.54	3.3h

Four things the table says:

The harness matters as much as the model. octomind and opencode ran the same model on the same endpoint at the same prices - a pure harness A/B. octomind solved 24, opencode 19, at half the cost. The difference is context discipline (opencode pushed 30M input tokens through one task where octomind needed 8M) and supervision: the cases opencode dropped are exactly the deep-root-cause, multi-trap ones where an unsupervised agent declares victory too early.
An open model beat Opus while paying full price for every token. glm-5.2 ran via Ollama cloud, which has no prompt caching - every agent turn re-bought its full context at list price. Opus, meanwhile, billed ~97% of its context re-reads at 1/10 cache rates. glm's dollar figure is its worst case, and it still came out ahead on solves, cost, and time. On any cache-enabled endpoint the cost gap becomes a chasm.
Speed has a thoroughness tax. Codex is remarkable - 2-4 minutes per case, $0.59 median - but all four of its failures are cases that punish not checking one more caller, one more surface, one more trap.
No task was impossible. Every case was solved by at least one agent, and the failure classes were clean: the trust-boundary case caught opus, codex, and opencode; the deep root-cause parser case caught everyone except octomind; and the one case both glm harnesses failed identically is a genuine model blind spot, not a harness artifact.

Where the harness edge comes from

A big share of "context discipline" is simply how the agent finds code. Stock claude code, codex, and opencode navigate a repository the same way: grep, read the file, grep again, read more - and every re-read is context the model pays for on every subsequent turn. octomind's developer:general ships with octocode out of the box: structural search that finds the exact symbol, signature, or code pattern directly, so the agent jumps to the three functions that matter instead of paging through files that don't. Enable semantic indexing and it also searches the codebase by meaning - "where is cookie validation handled" as a query, not a grep pattern. Fewer wrong files read is fewer tokens re-bought, and that compounds into the 8M-vs-30M input-token gap the A/B exposed.

The other share is supervision. During dry runs an agent once "finished" a task by announcing "Now let me create a plan and implement the fix" - and stopping, with a zero diff. In a non-interactive run a text-only turn ends the session, so octomind added a deterministic guard: if a turn ends with no action while the agent's self-reported status is still in progress, it gets sent back to work. In the final run that failure mode was gone - and the cases opencode dropped are precisely the ones where nothing questioned an early declaration of victory.

Reproduce it

Everything is public and pinned: the full per-case table and reproduction guide at the exact commit of this run, per-case agent traces and judge verdicts in the run artifacts, and the case-harvesting pipeline - the reverse-spec agent, the derivability rules, the fail-to-pass verifier - documented so the set keeps growing as training cutoffs advance.

The result we care most about isn't the ranking. It's the A/B: same model, same endpoint, five more tasks solved at half the cost. Model quality is table stakes now - the harness is where the leverage is. That's the thesis octomind is built on, and now we have the number.

Workflows Come to the Cloud: Running Multi-Agent Pipelines from the Browser

Don Karter — Wed, 29 Jul 2026 08:22:38 +0000

Workflows Come to the Cloud: Running Multi-Agent Pipelines from the Browser

I've been building AI tools for three years now. Here's what I keep running into: most real work isn't a conversation – it's a contract.

You have the input (an article, a diff, a pile of CSV data). You know what the output should be (social drafts, a review verdict, a readable report). What you want in between isn't chat. It's a pipeline: specialists doing their parts, handing off to each other, with an auditor gating the result before it reaches you.

That's the core idea behind workflows in Octomind – our session-first, multi-provider AI agent runtime. And as of this week, those workflows now run in the cloud panel, not just the CLI.

Same pipelines. Same tap library. New surface.

If you've been running octomind workflow from your terminal, nothing changed – you just got a second way to do the same thing. If you haven't tried workflows yet, here's what I built and why I built it this way.

A workflow is a team with a contract

Every workflow in Octomind is a declared pipeline of steps. Each step is a full agent run with its own role. Steps can be sequential, parallel, conditional – or loops, which is where it gets interesting.

Take the promote workflow. You feed it an article. It turns that article into platform-native social drafts. Here's what actually happens:

A researcher grounds the article in its source material
A writer drafts per platform (Twitter, LinkedIn, whatever)
An auditor checks each draft against quality gates
If the audit fails, the draft loops back for re-editing
The pipeline halts rather than shipping something the auditor rejected

Or review – it takes your unstaged diff and produces an independently verified verdict. Or report – it turns raw data into a decision-ready document where every number is computed, never invented.

There are fourteen of these in the public tap today. They're the same whether you run them from a terminal or a browser.

# CLI still works exactly the same
cat article.md | octomind workflow promote

The full input goes in at the start – the whole brief, up to 256 KB of it – and the finished result comes out at the end. No steering mid-flight, no drip-feeding context. That's not a limitation; it's the point. You're not chatting with a team, you're commissioning one.

From library to running pipeline

In the cloud panel, the Library tab shows the catalog. Pick a workflow and the new-run screen asks for exactly two things: which machine, and your input.

The Dry run button is the part I'd point at first. It validates the workflow on the machine itself and renders the resolved pipeline – every step, every role, the loop caps – without spawning anything or reading your input.

It's free. It takes a couple of seconds. And it means a broken pipeline fails before a single token burns.

Starting a real run performs the same validation again on its own, so the button is a preview, not a chore. But I use it anyway. There's something useful about seeing the full pipeline shape before you commit to running it.

Then the run page becomes the thing worth watching: the whole chain renders immediately from the validated plan, and steps light up as the stream comes in. A checkmark when a step completes. A spinner on the one running. A pass counter on loops as the auditor sends drafts back.

You can close the tab and come back. The run belongs to the machine, not to your browser.

Artifacts: when output is a file, not text

Under the result there's a detail we're calling artifacts. Workflow deliverables are often files – promote writes drafts, report renders a document – and the final message names the paths.

The panel picks those mentions out and turns each one into a download chip, with inline previews for images and text. Nothing is fetched until you click – the mention is just a string until you ask for the file.

This is also the security posture: text can name a path, but only you can fetch one. Sessions got the same mechanic: when an agent in a chat says it saved something, the file is right there to grab.

Everything the run printed is kept, verbatim. The collapsible full log holds the validated plan, the machine-readable event stream, and the human progress view – downloadable, nothing summarized away. Step outputs get their own friendly view too, so you can read what each specialist handed to the next.

Honest one-shots: why there's no retry button

Here's a design decision I want to be explicit about: a workflow run has five states – queued, running, done, failed, canceled – and the last three are final.

There is no resume. No retry-in-place. No "continue from step 3".

The CLI has no such mode, and we didn't fake one in the UI. Partial re-execution of a pipeline whose earlier steps fed later ones is a good way to get confidently wrong output. If step 2 consumed step 1's output and step 5 consumed step 4's output, you can't just rerun step 3 without invalidating everything downstream.

So a failed run tells you plainly what went wrong – the actual stderr tail, not a shrug – keeps every completed step's output visible (that spend was real, and the work is often still useful), and offers exactly one recovery: Run again, which pre-fills a fresh run with the same workflow and input.

Your runs list is history you can trust: what ran, what it cost, what came out.

I'd rather give you a clean slate than a half-broken retry mechanism that produces garbage output.

What it costs: nothing new

There's no workflow fee and no new meter. A run's model calls flow through your account's hub key exactly like a session's – plan caps and credits decide, per published prices – and the machine bills per second while it works, like always.

The run page shows the aggregated cost the pipeline reported, so you can see what a promote actually costs you end to end. On the open models, it's typically cents.

Dry runs are free. And workflows are on every plan, free tier included – the free tier's one-at-a-time model calls just serialize the steps, which makes runs slower, not smaller.

Same contract as the CLI

If you already run workflows from the terminal, nothing changed and everything got a second surface. The panel drives the same octomind workflow binary on your machine, with the same tap resolution and the same machine-readable output contract the CLI exposes via --format jsonl:

# Machine-readable output
cat article.md | octomind workflow promote --format jsonl

One event per completed step. One aggregated cost at the end. A pipeline you tested locally behaves identically in the cloud, because it is identical.

That's also the direction this is headed. Today the library is the public tap. The plan is for every account to get its own tap – personal workflows, edited in the panel with the same dry-run validation, distributed to your machines through the exact mechanism tap content already uses.

One distribution system, not two.

Why pipelines beat chat for production work

I want to zoom out for a second. Why build this at all?

Chat is great for exploratory work. You don't know what you're looking for. You want to poke around, ask follow-ups, change direction mid-conversation. A session – a conversation where you steer turn by turn – is exactly right for that.

But production work is different. You have a contract. Input in, output out. You want repeatability. You want auditability. You want to know that if you run the same thing tomorrow, you get the same result.

Workflows give you that. They're not better than sessions – they're different. And for a whole class of problems, they're the right tool.

Here's what I mean:

Chat: "Help me figure out what's wrong with this code"
Workflow: "Review this diff and tell me if it's safe to merge"
Chat: "What should I post about this article?"
Workflow: "Turn this article into Twitter, LinkedIn, and Mastodon drafts"
Chat: "Help me analyze this data"
Workflow: "Turn this CSV into a decision-ready report with computed metrics"

The chat version is open-ended. The workflow version has a contract. Both are useful. I use both. But they're not the same thing.

The technical bit: how the pipeline actually runs

Under the hood, a workflow run is a state machine. Each step is an agent invocation with its own context, role, and constraints. The pipeline orchestrates these invocations, passing outputs between steps, handling loops, and aggregating costs.

The event stream is the key. Every step completion emits an event. The panel subscribes to that stream and renders it live. That's how you see steps light up, how loop counters increment, how the final cost gets computed.

Here's what a simplified event stream looks like:

{"event": "step_start", "step": "research", "role": "researcher"}
{"event": "step_complete", "step": "research", "output": "...", "cost": 0.002}
{"event": "step_start", "step": "write", "role": "writer"}
{"event": "step_complete", "step": "write", "output": "...", "cost": 0.004}
{"event": "step_start", "step": "audit", "role": "auditor"}
{"event": "audit_failed", "step": "audit", "reason": "..."}
{"event": "loop_iteration", "step": "write", "iteration": 2}
{"event": "step_complete", "step": "audit", "passed": true, "cost": 0.001}
{"event": "workflow_complete", "total_cost": 0.015, "artifacts": ["drafts/twitter.md", "drafts/linkedin.md"]}

The CLI and the panel both consume this same stream. That's why they behave identically – they're two different UIs over the same underlying contract.

What's next

If you haven't tried the cloud yet, it's open to everyone – no card, no invite. Create a machine, open Workflows, feed promote a blog post you like, and watch a team you didn't have five minutes ago argue itself into shippable drafts.

And if you're already running workflows from the CLI: nothing broke. Your taps still work. Your machines still work. You just got a browser option now.

The repo is at github.com/muvon/octomind. Apache-2.0, like everything we build. Pull it, run it, tell me what breaks.

This is the original article: Workflows Come to the Cloud

We Built an Invite System. We Deleted It Three Days Later.

Don Karter — Sun, 26 Jul 2026 15:18:29 +0000

Three days after launch, we killed the invite gate on Octomind Cloud machines.

Not paused. Not "temporarily opened." Deleted.

Here's what happened, why it was wrong, and what changed.

The Confession

At launch, machines were invite-only. We're bootstrapped – no investors – and hardware gets bought with revenue, not term sheets. A gate felt responsible. Don't let everyone in if you can't handle them.

Three days of production taught us the gate was solving a problem we didn't have yet, and creating one we definitely did.

Fleet had headroom. Bottleneck wasn't capacity – it was new users hitting a locked door and leaving before seeing what a machine actually is.

So we removed it. The invite system is gone from the product. Fewer moving parts, one honest answer to "can I try it?": yes.

What Octomind Cloud Is (Briefly)

Octomind Cloud is an agent runtime. Persistent Linux machines from $0.05/hr, real Docker inside, a hub of 21 models behind one key at published per-token prices, one wallet with caps.

Free tier: $0, $0.15/day usage, 1 Tiny machine, no card required.

The agent runs on these cloud machines, not your laptop. Sessions survive you closing the tab – keep running, full transcript when you come back.

Whole stack is open source Apache-2.0: octomind, octocode, octobrain, octohub.

What Three Days Taught Us

Day one: invites felt prudent. Don't oversubscribe a fleet you're funding yourself.

Day three: the prudent thing was creating friction for people who just wanted to see what this does.

We watched the funnel. Signups were fine. Activation dropped at the invite wall. People weren't hitting capacity limits – they were hitting a locked door and bouncing.

The invite system wasn't protecting us from overload. It was protecting us from users.

That's backwards.

What Changed

As of today, every Octomind account – free tier included – can create a machine and have it running in under a minute. No codes, no waitlist, no earning your way in.

What this means:

New users sign up, create a machine – real Linux box, whole agent toolchain, web terminal, Docker inside – in under a minute
Free tier, no card
Unredeemed codes are moot
Referral program still alive (both earn storage and credit bonuses)
Paid plans still buy bigger everything (caps, sizes, slots) – just no longer buy a key to a door that's now open

We're still bootstrapped. Fleet grows as subscriptions fund hardware. If demand outruns capacity, machine creation says so plainly rather than silently degrading.

Honesty over optics.

Try It

Open the panel, sign up free (no card), create a machine. Pick a name, pick a size, done – boots in seconds. Open a session and put the agent to work.

GitHub: https://github.com/muvon/octomind

Full post: https://octomind.run/blog/cloud-invites-removed

We Taught Our Code Search to Think — Here's What Broke (and What Fixed It)

Don Karter — Sun, 26 Jul 2026 08:14:36 +0000

Octocode is our open source semantic code search engine built in Rust (Apache-2.0). It's the code-search layer behind Octomind, and you can grab it at https://github.com/muvon/octocode.

Last month we added an LLM reasoning step to the retrieval pipeline. The first version made things worse.

Hit@10 dropped 7 points. Recall@10 dropped 7.5 points. The model was pruning away true positives because it thought they weren't relevant.

Here's what we learned, what we shipped, and why "add contextual retrieval" is not the universal win everyone claims.

The Problem: Similarity Is Not Relevance

Octocode uses hybrid retrieval — vector search plus keyword overlap. Works well. But there's a gap: the snippet that shares the most tokens with your query is often not the code that answers it.

Ask "where do we decide a request is retryable" and the similarity ranker hands you:

The retry config struct
The retry constant
The test that names the word four times

The actual function that makes the call? Rank seven.

Vector search ranks by similarity. Hybrid adds keyword matching. But neither reads the code. Neither knows what the code does.

First Attempt: Pure LLM Reranking

We added an LLM reasoning step after hybrid retrieval. The model reads the candidate code bodies and re-ranks by whether they actually answer the query. No new index, no reindexing — just a reranker sitting between retrieval and results.

Tested on 127 queries. Here's what happened:

Metric	Before	After	Change
MRR	0.595	0.752	+0.157
NDCG@10	0.658	0.758	+0.100
Hit@10	0.913	0.843	−0.071
Recall@10	0.886	0.811	−0.075

MRR and NDCG went up. The ranking quality improved. But Hit@10 and Recall@10 dropped.

The LLM was being too aggressive. It returned only what it judged relevant and quietly discarded true positives at ranks 6–10. Better ranking head, worse recall floor.

The Fix: Fuse, Don't Replace

Don't let the LLM replace the ranking — fuse it with the hybrid ranking via Reciprocal Rank Fusion (RRF).

score = 1/(k + hybrid_rank) + reasoning_weight * 1/(k + reasoning_rank)

Hybrid rank always contributes — that's your recall floor. Reasoning rank drives the head.

Final fused results on the same 127 queries:

Metric	Before	After	Change
MRR	0.595	0.809	+36%
NDCG@10	0.658	0.833	+27%
Hit@5	0.827	0.953	+0.126
Hit@10	0.913	0.969	+0.056
Recall@5	0.777	0.924	+0.147
Recall@10	0.886	0.944	+0.058

Every metric up. Hit@5 at 0.953 means nineteen times out of twenty the answer is in the top five.

What We Tuned (and What Didn't Matter)

Reasoning weight: Best at 2.0. Weight 5 buys nothing and costs recall.

Candidates: 25 is the sweet spot. We tried 40 — worse across the board.

Context level: Full code bodies win by a lot. Signatures-only was the worst performer.

LLM temperature: 1.0 beat 0.3 and 0.0. Counterintuitive, but the model reasons better with normal sampling. Stiff decoding made it dumber.

What Did NOT Work: Contextual Retrieval

Anthropic's contextual retrieval approach — adding descriptions at index time — gets recommended everywhere. We tested it.

Hit@5: −0.008

Recall@10: −0.019

It trades recall for ranking. On code, that's a net loss. "Add contextual retrieval" is cargo-culted advice — not a universal win. We didn't ship it.

How to Turn It On

The feature is in Octocode's master branch now (not yet released — landing soon). Off by default, one config flag:

[search.reasoning]
enabled = true
model = "deepseek:deepseek-v4-flash"
max_candidates = 25
context_level = "full"
reasoning_weight = 2.0
final_top_k = 10

Or swap in any other provider:model:

[search.reasoning]
enabled = true
model = "openai:gpt-4o-mini"
max_candidates = 25
context_level = "full"
reasoning_weight = 2.0
final_top_k = 10

Any provider:model works. One LLM call per semantic search. structural_search stays pure grep. No reindex needed.

The Takeaway

LLM reranking works – but only if you keep the hybrid ranking as a floor. Fusion beats replacement. And don't cargo-cult "best practices" without testing them on your actual workload.

Octocode is open source under Apache-2.0. The benchmark suite, ground truth dataset, and raw results are all in the repo: https://github.com/muvon/octocode

This feature is merged to master now, not yet released – landing soon.

Full writeup with all the numbers and tuning details: https://muvon.io/blog/reasoning-retrieval-code-search

The Tap Registry Grew Beyond Code: 108 Agents, 27 Domains, and Workflows That Verify Their Own Work

Don Karter — Fri, 24 Jul 2026 10:30:00 +0000

This post originally appeared on octomind.run/blog/octomind-tap-registry-expansion. Cross-posted to dev.to.

The cloud launch got the attention this week. But something quieter happened alongside it — the tap registry went through its biggest expansion since taps were first introduced, and it's worth a closer look if you're building with agents.

The default registry now ships 108 agents across 27 domains, 71 capabilities, and 18 workflows. All available with one command. No downloads, no marketplace accounts, no setup.

Here's what changed and why it matters.

What a tap actually is

If you haven't used taps before, the mental model is simple: a tap is a git-hosted registry of TOML manifests. Each manifest defines one agent — its system prompt, model, tools, MCP servers. Octomind fetches the manifest at runtime and merges it into your config.

Nothing gets installed. There's no global state to rot.

octomind run developer:rust      # a Rust specialist
octomind run lawyer:immigration  # an immigration-law explainer
octomind workflow harden         # a multi-step security audit pipeline

The tag format is domain:spec. The official muvon/octomind-tap ships as the default registry. You can browse it on the site, and adding your own registry is a single octomind tap away.

The point: agents should be distributed like code — reviewable in a diff, not installed like apps.

The registry isn't just for developers anymore

This is the biggest shift. The tap registry started as a developer toolbox, but most problems people bring to an agent aren't code problems.

This week added 17 new agents across domains you wouldn't expect in a dev tool:

Family — family:parenting, family:eldercare, family:bereavement (an agent for the paperwork and decisions that arrive at the worst possible time, written with the gravity that moment deserves)
Finance — finance:tax, finance:retirement, finance:insurance
Health & wellbeing — doctor:sleep, coach:wellbeing, coach:negotiation
Life — lawyer:immigration, home:realty, vet:general, vet:behavior, vehicle:mechanic, tutor:coding, content:fiction

These aren't replacements for licensed professionals. They're structured, well-prompted assistants that help you prepare, organize, and understand. The value is that the preparation — which is honestly 80% of the work — no longer starts from a blank page.

There's also assistant:concierge, which routes across the whole ecosystem. A bare octomind run discovers matching specialists by intent.

Four new workflows that close the loop

Workflows are multi-step TOML pipelines. Each step gets its own agent, model, and toolset, with validate scripts acting as deterministic quality gates between steps.

The shared pattern across all four new workflows: don't stop at the first answer — loop until the check passes.

harden

An OWASP-lens security audit of your current repo, then a fix ⇄ re-audit loop that runs until findings are resolved and your project's own checks stay green.

octomind workflow harden

upgrade

Dependency bumps — named packages, or safe patch/minor by default — running in a bump ⇄ verify loop until your build is green.

scout

This one's interesting. It's the phase before anyone has an idea: take a field or market, mine it for real evidenced pain through three blind parallel sweeps, and come back with vetted opportunities instead of vibes.

learn

Bootstraps a project knowledge base. Reads the codebase, extracts durable facts, promotes them into .box/ where they're indexed and readable by every agent that works on the project afterward.

All four run the same way as everything else — octomind workflow — reading from stdin, writing to stdout, scriptable by construction. That brings the registry to 18 workflows total.

Agents that remember the project

A new octobrain-backed knowledge capability gives agents durable project memory. assistant:knowledge curates it, the learn workflow seeds it from the codebase.

Facts get extracted once, stored in the repo's .box/ directory, and become available to every specialist that touches the project afterward. You run learn once. Every agent after that starts with context instead of zero.

The unglamorous fixes that actually matter

Two things that don't make headlines but fix real pain:

Dependency installs now work in containers and CI. Tap agents bootstrap their own tooling (Node for MCP servers), and those scripts assumed sudo exists. Root containers — Docker, CI images — don't have it. Every dep script now escalates through a helper that runs bare as root, uses sudo when present, and tells you exactly what's missing otherwise.

Web search needs no API key. The websearch capability defaults to DuckDuckGo now. One less credential between a fresh install and a working researcher agent.

Use it anywhere

Everything works on a local Octomind install (one curl command) and comes preloaded on cloud machines. Models route through the hub with octohub:auto routing.

What to do next

Browse the full registry at octomind.run/tap — see what's available before you build something from scratch
Read the manifests at github.com/Muvon/octomind-tap — they're TOML files, fully reviewable, and contributions are open
Try a workflow — octomind workflow harden on a repo you care about and see the loop-until-green pattern in action

If there's a specialist you keep wishing existed, it's one manifest away. And if you build one, we'd genuinely like to see it.

Your First Five Minutes with Octomind: One Install, One Login, Zero API Keys

Don Karter — Fri, 24 Jul 2026 04:43:08 +0000

This post originally appeared on octomind.run/blog/octomind-hub-quickstart. Cross-posted to dev.to.

Every AI coding tool starts the same way: sign up, grab an API key, paste it into a config file, forget it's there until it leaks. Octomind doesn't do that. One curl command, one browser click, and you're talking to an agent that reads your code, finds bugs, and fixes them — then proves the fix works.

Total cost for the whole thing: under two cents.

Here's the walkthrough. Everything below ran in a clean Ubuntu 24.04 container. No dotfiles, no cached credentials, no Node, no Python pre-installed. Just the agent and a real codebase.

What you're actually getting

Octomind's Hub is an OpenAI-compatible gateway sitting in front of a curated model roster. Not a 400-model dropdown where half the entries are deprecated — 21 models that earn their place.

Open-weights models (DeepSeek, Qwen, Kimi, GLM, MiniMax, Gemma) ship with every paid plan. Premium models (Claude, GPT, Gemini) bill per token from prepaid credits at published prices. There's a free model with a daily allowance, so a brand-new account can try everything without entering a card.

The key part: you never handle an API key. Not once.

octomind login runs a device flow — same pattern as gh auth login. The CLI shows a short code, you confirm in your browser, and the Hub mints a key scoped to that specific machine. No password in your terminal. No secret in your shell history.

Every machine gets its own revocable key. Log in from your laptop and your server, and you'll see two keys in the panel named after each device. Revoke one, the other keeps working. Fair use is per account, not per key — past the parallelism limit, requests queue briefly instead of failing.

And the meter is visible while you work. Every response prints what it cost. No end-of-month archaeology.

Under the hood, it's all octohub — open source, Rust, one static binary.

Step 1: Install (one command)

curl -fsSL https://octomind.run/install.sh | bash

Or on macOS: brew install muvon/tap/octomind

Here's what actually happened in the container:

[INFO] Installing octomind...
[INFO] Detected platform: x86_64-unknown-linux-musl
[INFO] Latest version: 0.38.2
[INFO] Downloading octomind 0.38.2 for x86_64-unknown-linux-musl...
[INFO] Extracting binary...
[INFO] Installing to /root/.local/bin...
[SUCCESS] octomind installed successfully!
[SUCCESS] Installation verified!

Static binary drops into ~/.local/bin. No runtime dependencies, no package manager churn, works identically on macOS and Linux.

Step 2: Log in (one click)

octomind login

Output:

╭ login · octomind account
│   code  6PVJ-NHW9
│   url   https://octomind.run/app/login/cli
│
│ Confirm the code in your browser to finish signing in.
│ waiting…
│   account  don@muvon.io
│   key      octomind-cli-35f0c0f849cd
│   stored   /root/.local/share/octomind/config/.env
╰ ✓ login · signed in

The CLI opened the panel URL, I confirmed the code, and it claimed a per-device Hub key. If you don't have an account yet, the same page signs you up first — free plan needs no card.

The key shows up in the panel under Keys. Revoking it takes one click. This shipped in 0.38.0 because pasting long-lived secrets into terminals is exactly how keys end up in shell history and screen recordings.

Step 3: Ask your first question

I dropped a Python file with a deliberate bug:

# stats.py
def median(values):
    ordered = sorted(values)
    mid = len(ordered) // 2
    if len(ordered) % 2 == 0:
        return ordered[mid]
    return ordered[mid]


if __name__ == "__main__":
    print(median([3, 1, 4, 1, 5]))
    print(median([3, 1, 4, 1, 5, 9]))

The bug: both branches return ordered[mid]. The even-length case should average the two middle values.

If your directory isn't a git repo, run git init first — the code-search index works repo-per-repo. First run bootstraps its own tooling (it installed Node for MCP servers unprompted), so give it a minute.

Then:

octomind run

This starts the concierge orchestrator — it works out what you need, routes to the right specialist, and remembers across sessions. The header shows the agent, model setting (octohub:auto — the Hub routes each request to a suitable model), working directory, and session ID.

Or skip routing entirely: octomind run developer:general for direct specialist access.

I asked: "Read stats.py and explain in a few sentences what it does. Is the median calculation correct?"

The agent:

Read the file with a tool call (not hallucinating from context)
Found the bug: duplicated branch, even-length case wrong
Demonstrated on concrete input: median([3,1,4,1,5,9]) → sorted [1,1,3,4,5,9], mid=3, returns 4. Correct answer is (3+4)/2 = 3.5
Suggested the fix: return (ordered[mid-1] + ordered[mid]) / 2

Cost: $0.00403 · 27K in · 518 out

Step 4: Make it prove the fix

Here's where it gets interesting. I asked: "Fix the median bug, then run python3 stats.py to prove the fix works."

The agent:

Edited exactly the broken line with str_replace
Ran python3 stats.py
Output: 3 and 3.5 — fix verified. Even-length now yields 3.5 instead of 4.

Cost: $0.00960 · 67K in · 525 out

Total for review plus verified fix: under a cent and a half.

Every run is a persistent session. octomind run --resume-recent picks up where you left off. -n mywork gives the session a name. You can pipe it for CI and scripting:

echo "Fix the median bug, then run python3 stats.py to prove the fix." \
  | octomind run developer:general --format plain

How to ask questions that actually work

A few things I learned from the transcripts:

One task per message, with the file named. "Read stats.py and check the median calculation" beats "look around and tell me what's wrong." Specificity is free speed.

Ask for proof, not confidence. The agent has a shell. End your request with "run it and show the output." A claim that compiles is worth ten that don't.

Bound the answer. "In a few sentences" saved tokens on every single transcript in the walkthrough.

Escalate when the task outgrows the shelf. -m on the command line or /model mid-session switches to any model. Premium ones bill per token from credits. Escalation is a flag, not a migration.

Does it hold up?

The SWE-bench-Live numbers: 70% solved on the open-shelf model (GLM) at roughly $3.30 per solved task. The harness and baseline live in the repo under bench/ with committed history.

Honest framing: it's their own harness, a pilot-sized suite (15 instances, two runs each), efficiency-first metric. Not a leaderboard submission. The claim is straightforward — an agent on open-weights pricing, resolving real issues, with a cost-per-solved you can actually reason about.

What's next

Everything above ran on a local machine. That's the point — start where you are.

Machines (persistent Linux boxes with the full toolchain, real Docker inside, sessions that survive closing your laptop) are invite-gated while the fleet scales. Subscribing unlocks access instantly. Free accounts earn invites through weekly usage. Every subscriber gets invites to hand out.

It's bootstrapped — hardware bought with revenue, the gate widening at the pace they can genuinely serve.

Get started

Free tier, no card required:

curl -fsSL https://octomind.run/install.sh | bash
octomind login
octomind run

Five minutes, two cents, no API keys. Ask it something concrete, make it prove its answer, then see how it goes.

Octomind Cloud Is Live: Real Machines for Your Agent, Every Model Through One Key

Don Karter — Wed, 22 Jul 2026 16:42:12 +0000

This post originally appeared on octomind.run/blog/octomind-cloud-launch. Cross-posted to dev.to.

You've built an agent. It's running a 40-minute refactor. You close your laptop.

Gone. Session dead. Tokens burned. Context lost.

Agents are long-running processes. Laptops aren't long-running machines. The math doesn't work.

That's the first problem we solved with Octomind Cloud. There are two more — and they're equally annoying if you've ever tried to run AI workflows in production.

The Three Problems

Problem 1: Your agent dies with your laptop lid.

You start a long task. Close the laptop. Come back. Everything's gone. You're starting over.

Problem 2: API keys in five dashboards.

OpenRouter. Anthropic. OpenAI. Voyage. That .env file you're scared to look at. Every provider has its own billing, its own rate limits, its own way of telling you you're out of credits.

Problem 3: Your AI bill is a surprise.

Opaque seat pricing. Or worse — usage-based, but you can't see the meter until the invoice hits. Runaway agents burning money overnight while you sleep.

Octomind Cloud is the answer to all three. It's cloud compute for AI agents — one subscription, one wallet, one key.

What You Actually Get

Two things sharing one account:

The Hub — an OpenAI-compatible gateway with 21 curated models behind a single API key
Machines — persistent Linux containers with the full agent toolchain, ready in under a minute

The whole stack is open source (Apache 2.0): octomind, octocode, octobrain, octohub. Self-host if you want. All doors open, docs assume you might.

The Hub: One Key, 21 Models

The Hub is an OpenAI-compatible gateway. You get one API key, and it works across every model on the shelf.

Open shelf (included in every paid plan):

DeepSeek V4 Flash, DeepSeek V4 Pro
Qwen3.7 Plus, Qwen3.7 Max
Kimi 2.6, Kimi 2.7 Code
MiniMax M3
GLM-5.2
Gemma 4

Premium shelf (bills per token from credits):

Claude Fable 5, Opus 4.8, Sonnet 5
GPT-5.6 family: Sol, Terra, Luna
GPT-5.5, GPT-5.5 Pro, GPT-5.4 mini, GPT-5.4 nano
Gemini 3.5 Flash, Gemini 3.1 Pro

There's also a free model with a daily allowance. When everything else is exhausted, this one keeps working.

Prices are published. Metering matches the table exactly. No surprises.

Fair use per account:

Free: 1 parallel call
Pro: 5 parallel calls
Max: 10 parallel calls
Team: 20 pooled parallel calls

Under the hood, the Hub runs on octohub — open source, written in Rust, ships as one static binary.

Here's what a call looks like:

curl https://hub.octomind.run/v1/chat/completions \
  -H "Authorization: Bearer $OCTOMIND_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-luna",
    "messages": [{"role": "user", "content": "Explain Rust ownership in one paragraph"}]
  }'

Same endpoint, same format. Switch models by changing the model field. That's it.

Machines: Persistent Linux Containers

Machines are persistent Linux containers with the full agent toolchain pre-installed:

octomind, octocode, octobrain
Web terminal
Git, Docker, Node, Bun
Headless browser, ffmpeg

Three sizes:

Size	vCPU	RAM	Active	Idle-Warm	Suspended
Small	1	2 GB	$0.05/hr	$0.01/hr	~$0
Medium	2	4 GB	$0.10/hr	$0.02/hr	~$0
Large	4	8 GB	$0.20/hr	$0.04/hr	~$0

Per-second metering. Automatic transitions between active, idle-warm, and suspended states.

10 GB working disk free with every machine.

Security details that matter:

Docker runs inside (real Docker Engine, sysbox runtime, no --privileged)
System filesystem is read-only
Agent runs as non-root user
Environment variables are encrypted and write-only — the panel never shows the value back
Shared directories between machines on the same host

Your machine is ready in about a minute. You get a web terminal, an API endpoint, and it's reachable from the panel, the developer API (api.octomind.run), or Telegram.

Sessions Survive You

Start a turn. Close the tab. The turn keeps running.

Open it from your phone 20 minutes later. Everything's there — full transcript recorded.

Sessions are reachable from:

The panel
Web terminal
Developer API (api.octomind.run)
Telegram

Telegram integration: One tap connects your bot. The machine answers in chat with real formatting — code blocks, lists, everything renders properly.

Switch models per session: /model gpt-5.6-luna works from the panel, Telegram, or the API. Same command, same result.

This is the same octomind you run locally. Same config, same sub-agent delegation, same tools. The only difference is it doesn't die when you close your laptop.

Embeddings: Index Once, Search Everywhere

Two lanes for embeddings:

Local (default, free):

Fastembed running on machine CPU
You pay compute, not tokens

Hub (paid):

Voyage 4, Jina v5, OpenAI text-embedding-3
Starts at $0.02 per million tokens

Index once. The index lands in shared storage. Every machine reuses it. No re-indexing, no duplication.

Pricing

Plan	Price	Usage Caps	Machines	Shared Storage
Free	$0	$0.15/day	1 Small	2 GB
Pro	$20/mo ($10 first month)	$20/4h · $60/week · $120/month	5	10 GB
Max	$100/mo	5x Pro caps	20	30 GB
Team	$500/mo	Pooled caps	Unlimited members	Shared machines

Credits are overflow. Hit your cap mid-task? Credits keep the session flowing. It doesn't die.

Every price is on the pricing page and in the panel. The meter bills from the same numbers. No reconciliation surprises.

What's Next

Octomind Cloud is in beta. It's live now.

The whole stack is open source, Apache 2.0. Self-host if you want, or use the cloud and move on with your day.

Try it: octomind.run/app — free tier, no card, start building.