DEV Community

gracefullight
gracefullight

Posted on

oh-my-agent: failed runs now turn into skill regression tests

#ai #programming #agents #productivity

A failed agent run used to end with a log and a shrug. This week oh-my-agent closed that loop: a failure can be captured as an incident, promoted to a skill fixture, and fed to an optimizer that edits the skill under a dispatch budget. It took 131 commits and moved the CLI from 14.7.11 to 14.13.1.

What's new

  • Incident capture and promotion: oma harness incident scan lists failed, blocked, and partial runs that no incident references. oma harness incident promote derives a regression fixture for the skill the agent used, and admits it only when the recorded failing output fails it. oma harness feedback --scan-runs chains capture, promote, and optimize. Finished runs now keep the last 64 KiB of runner log so the fixture can be validated against what the agent actually said.
  • Budgeted skill evolution: oma skill optimize accepts an edit only when neither the validation nor the training split regresses and at least one improves. constitution.budget.max_dispatches_per_run is now enforced: the call that would exceed it is refused and promotion is blocked. Every applied edit records lineage, and oma skill promotions and oma skill rollback read it back.
  • Meta-optimization: oma skill meta-optimize treats the optimizer prompt itself as the candidate. Promotion needs a seeded paired bootstrap 95% interval above zero and at least three pairs. The first adopted procedure beat the current one by 0.27 on average, interval [0.04, 0.54], and added three grounding rules for proposed skill edits.
  • Routing eval: oma skill eval --routing shows the model every installed skill description and records whether it picks the target, a neighbor, or nothing. It separates "the body helps" from "the skill gets selected".
  • Code-intelligence guard: a PreToolUse hook denies native Grep, Glob, and recursive shell search (rg, grep -r, find -name, git grep) while a provider is configured. The deny message names the Serena tool to use. OMA_CI_ALLOW_NATIVE=1 is the escape hatch, and providers.code_intelligence_guard: off disables it.
  • Session-start notices: the state snapshot now announces skill and procedure promotions once, one line each with the edit and the lifts. oma doctor gained an Evolution note.
  • Orca reporting: spawned child agents report their lifecycle to Orca. Qwen gained native agent definitions.

What's fixed

  • The weekly cross-post job failed on every run after model_preset moved to auto. claude --output-format json wraps the answer in an envelope, and runAgent returned it raw. It now unwraps the configured response_jq path and falls through to raw stdout for vendors without one.
  • OS jobs registered before the command rename kept calling oma schedule:run <id> and failed on every fire, while schedule list reported them as synced. The legacy spelling is accepted when the OS invokes it, drift detection has a new stale state, and oma schedule sync and oma update rewrite stale registrations.
  • cd $HOME && oma link claude treated HOME as a project and rewrote the global Claude hook commands. link and update now refuse project mode from HOME (#788).
  • The skill judge parsed PASS/FAIL from the raw Claude envelope, where "failed":0 came before the verdict, so every judge verdict read FAIL. Outputs are unwrapped before scoring, and stale recordings are discarded.
  • --repeats 3 parsed to NaN because bare parseInt received the previous value as its radix. Variadic option values (--skill a b) no longer trip the stray-positional guard.
  • Projects synced before v11 still ran their own Serena per session. Migration 029 rewrites those launchers to the shared oma bridge entry in each vendor's native form.
  • oma update respects an explicit vendor list instead of detecting from project directories.

What's better

  • One optimization epoch with four candidates took forty minutes because every arm and judge ran serially. Live dispatches now run through a bounded pool (OMA_SKILL_EVAL_CONCURRENCY, default 4), and meta-optimization overlaps inner runs across skills (OMA_META_CONCURRENCY, up to 4). The per-dispatch timeout is 180s, with one retry on timeout.
  • Fixtures test what the body states. The oma-debug set was rewritten after the first version scored 89% without the skill. The new twelve score 25% baseline and 100% with the skill. oma-refactor went from 16.7% to 100% the same way. oma-docs, oma-scm, and oma-qa also have twelve body-aligned fixtures each.
  • The first loop-verified edit landed in oma-docs: held-in 55.6% to 77.8%, held-out validation unchanged at 100%.
  • Cost is visible. Runs report model calls per verified improvement, so a procedure that wins on gain by spending more shows up as such.
  • Gortex is out and code intelligence is back on the Serena bridge. It pinned the CPU re-patching a 4 GB index and exposed no cap.
  • Skill and shared prompt files were trimmed to cut unconditional preflight, scoring, and approval rules. Incident guides shipped in all 11 doc locales.

Installation 

# macOS / Linux
curl -fsSL https://raw.githubusercontent.com/first-fluke/oh-my-agent/main/cli/install.sh | bash
Enter fullscreen mode Exit fullscreen mode 
# Windows (PowerShell)
irm https://raw.githubusercontent.com/first-fluke/oh-my-agent/main/cli/install.ps1 | iex
Enter fullscreen mode Exit fullscreen mode

Links

oh-my-agent is built for teams who orchestrate more than they prompt. Next up: running the feedback loop on a schedule against real project runs, so skill edits arrive with their evidence attached.

https://github.com/first-fluke/oh-my-agent

Top comments (0)