Originally published at olbrain.com/blog.
Alchemist is Olbrain's agent builder. People use it from Claude Code, Codex, Cursor and a few other clients through an MCP server and an open plugin of skills and commands. On a build the user's coding agent drives, the split is simple: the coding agent reads the user's project and drafts the brief, the plan and the tests; Alchemist builds the agent, runs and grades its tests, checks every step in order and publishes it.
A coding agent with a dozen powerful tools and a goal picks its own order. In a repo, git and a test suite catch the mistakes. A build that ends with an agent answering a business's customers is different: the model can skip the brief, re-plan halfway, or call a draft done before anything tested it. We wanted the order enforced by the server, not requested in a prompt.
Two tools: read the state, take one step
The coding agent gets two step tools. alchemist_state returns where the build stands: its stage with that stage's instructions and checklist, the steps allowed now (each with its input schema), the steps blocked with the reason, open cards, the budget left, the last step's result and an inbox of news. alchemist_do takes one step: a tool, its args, and an op_id the caller makes up.
The stages run brief, plan, build, test, publish, go live. The skill the coding agent follows says it plainly: "Take only actions listed in allowed."
The stage is derived, never stored
The stage is not a field the model can set. The server works it out on every call from facts the platform wrote: card answers, what each card showed, the stored brief and plan, and each agent's record. Plan-step states are never read, because the driver can set those. The build-level part:
stage of a build:
the build is finished -> done
no card approved this exact brief -> brief
no card approved this exact plan -> plan
otherwise, for each agent in the plan -> build, test, publish or go live
"Approved" means approved for the brief as it is now. The brief card stores a hash of the brief it showed, so an update_brief after approval needs a new card. The plan works the same way with the job map. Per agent, the rule that matters most: a test run with no tests in it counts as untested, and nothing the user chose on a card skips the tests.
One design choice saved us real confusion. The list of allowed steps and the refusal for a disallowed one come from the same check, so what the client is shown and what it gets refused never disagree. We also run the same rulebook when Alchemist's own model drives a build instead of the user's coding agent.
Refusals that say what to do
A step out of order comes back with status: "refused", a reason such as out_of_order or not_offered, a fix in plain words, and the fresh state. The fix is written for a model to act on: "The brief isn't approved yet: write it (update_brief), then show it (ask_user kind=brief)."
The skill makes handling refusals a rule rather than leaving the model to improvise:
Refusals are not errors to retry.
out_of_order,not_offered,suite_incompleteor a rejected test come with ahintorfix: re-read the state, fix that, and retry once with a newop_id. Never resend the unchanged refused call under a newop_id, and never work around a refusal with another tool or another build.
The last clause matters: a refused agent's next idea is often another route to the same goal.
Idempotent step ids
Connections drop, and a step over about 20 seconds returns still_running and carries on. Without ids, a retry can build or test twice. The server keeps the last step's result and the 20 most recent op ids. The same op id with the same tool and args returns the first result with the note "This op_id already ran: this is its first result, nothing ran again." The same op id with a different tool or args is refused as op_id_reused. The order inside the dispatcher is fixed: op id replay, then the stage allowlist, then budgets, then test quality, then the tool.
Tests written by the agent, graded by the server
The coding agent writes a chat agent's tests, since it has read the user's docs. The server doesn't trust them blindly. Our rule of thumb: a weak suite passes a weak agent. Before tests are saved or run, deterministic rules (no model call) refuse an expected answer too short to judge, a question that already holds its answer or tells the agent what to reply, and a near-duplicate of a test already in the suite. Before a run, the suite needs one test per brief requirement (at least 5, at most 20 asked for) and, for a chat agent, a negative test. The server then runs the suite on the draft and grades it. A job is tested differently: by a live run on a real input, through a card.
Budgets, then ask the human
A model close to passing keeps trying. Each agent has limits, enforced in the dispatcher:
- 120 draft test runs
- 4 failed compiles
- 3 fix rounds per build, each capped at six passes
- a stop after 3 graded runs in a row that don't beat the best
After three graded runs in a row below the bar that don't beat the best pass rate, the server stops testing and returns a budget_stop with the best run, its failures and a note. A run that beats the best clears the count, so a rising pass rate is never stopped. The skill tells the model to show that result and offer "continue", which adds two more passes, only on the user's yes. It also tells the model never to start a new build to reset a budget, and never to switch models itself.
Human gates, and the ones Claude can't answer
Before Alchemist publishes an agent, makes it live, merges agents or changes how one behaves, it raises an approval card. Users decide how many cards their coding agent answers for them, in the chat or in CLAUDE.md, AGENTS.md or GEMINI.md. A line like "Approve plans and tests yourself; ask me before publishing" is enough.
Some answers are never delegated. Removing an agent, forgetting a customer, renaming or moving an agent, changing a job's email address, and publishing despite failing tests always need the user's own yes. In a client with MCP form elicitation, such as Claude Code, the card appears in the client's own dialog, which the person answers and the model can't; for anything that can't be undone they type the agent's name, and the server checks it. Other clients get a browser link.
What this doesn't solve
- The server checks order and shape, not judgement. It knows a brief was approved, not that it was a good brief. That is what the approval cards are for.
- "Continue" is finite. It lifts a stop at most twice per build; after that the user carries on in Noesis, Olbrain's console.
- Not every build runs this way. Builds started in Noesis, and some plugin builds while accounts move over, are still planned by Alchemist's own model. Hosts keep the tool list from session start, so an older session must reconnect to get the step tools.
- In-client confirmation depends on the client. Without form elicitation, the user gets a browser link.
If you are building MCP tools for coding agents
Derive state from facts the model can't write. Return the allowed next steps with their schemas, and refuse everything else with a fix in plain words. Make every write idempotent with a caller-chosen id. Check anything the model writes to grade itself. Put budgets on loops and hand the decision to a person when one runs out. Keep a few answers only a person can give, enforced on the server.
The plugin, its skills and commands are MIT licensed at github.com/Olbrain/alchemist-plugin.
In Claude Code:
/plugin marketplace add Olbrain/alchemist-plugin
/plugin install alchemist@alchemist
Top comments (0)