DEV Community

Cover image for I Open-Sourced a Codex Skill That Researches, Tests, and Judges Competing Solutions
Felix
Felix

Posted on

I Open-Sourced a Codex Skill That Researches, Tests, and Judges Competing Solutions

#ai #llm #automation #openai

Why I built it

AI coding agents are very good at generating implementations quickly.

But project optimization is not only an implementation problem.

When I ask an agent to improve a real project, several problems often appear:

  • it implements the first plausible idea without exploring alternatives;
  • it searches for papers but never connects their mechanisms to the local project;
  • it compares experiments under different conditions;
  • it changes the success criteria after seeing the results;
  • it keeps tuning a failed idea without a stopping rule;
  • it reports that something is β€œbetter” without explaining the causal mechanism;
  • it leaves behind many disconnected notes that are difficult to resume.

I wanted a more disciplined workflow: one where research, brainstorming, implementation, measurement, review, and decision-making are connected.

That became Automation Exploration Pro, an open-source Codex Skill for evidence-guided project exploration and repeated optimization.

GitHub:

πŸ‘‰ github.com/fancheng5840074-bit/automation-exploration-pro

The core workflow

The Skill organizes exploration as a repeatable loop:

Inspect the project
        ↓
Confirm performance and complexity metrics
        ↓
Freeze the promotion gates
        ↓
Research papers and relevant domain journals
        ↓
Brainstorm and adapt mechanisms to the local project
        ↓
Execute the smallest testable candidate
        ↓
Audit the evidence and comparison conditions
        ↓
Explain the result from first principles
        ↓
PROMOTE / RETAIN / REVISE / REJECT / INCONCLUSIVE
        ↓
Continue with the next justified candidate
Enter fullscreen mode Exit fullscreen mode

Before comparing solutions, the user confirms:

  • performance metrics;
  • complexity metrics;
  • promotion or stopping gates;
  • whether subagent concurrency is allowed;
  • the maximum number of cycles.

This prevents the agent from inventing a convenient success criterion after the experiment has already finished.

Two exploration modes

1. Cyclic optimization

Cyclic optimization is designed for projects that already have a working baseline.

Examples include:

  • reducing API latency without breaking correctness;
  • improving model accuracy under a memory limit;
  • reducing hardware resource usage while preserving signal quality;
  • improving an analysis pipeline without increasing runtime beyond an accepted threshold;
  • reducing the cost or duration of an experimental workflow.

Each candidate is compared with the current incumbent under frozen conditions.

If a candidate passes the confirmed gate, it becomes the new incumbent. The next candidate must beat that promoted resultβ€”not an obsolete original baseline.

A request might look like this:

Use $automation-exploration-pro in cyclic optimization mode.

Performance metric:
All conformance tests must pass.

Complexity metrics:
p95 latency and peak memory.

Promotion gate:
Reduce p95 latency by at least 15%, with no correctness regression
and no more than 5% additional peak memory.

Subagent concurrency:
Allowed.

Maximum cycles:
4.
Enter fullscreen mode Exit fullscreen mode

The purpose is not to generate an unlimited parameter search. A materially different mechanism becomes a new candidate and consumes a new cycle.

2. Project exploration

Project exploration is intended for cases where the technical route is still uncertain.

Instead of optimizing one incumbent, the Skill researches and compares several mechanistically distinct approaches.

The literature workflow starts from the actual local problem. It can use broad scholarly indexes, preprint sources, publisher records, domain databases, and journals related to the project.

However, finding a paper is only the beginning.

Each research-backed candidate must be localized through the following mapping:

Paper mechanism
    β†’ local project constraint
    β†’ required adaptation
    β†’ predicted metric effect
    β†’ cheapest falsification test
Enter fullscreen mode Exit fullscreen mode

The workflow also separates three kinds of evidence:

Source-observed:
What the paper actually reported.

Local inference:
Why the mechanism might transfer to this project.

Local observation:
What the project's own implementation or experiment measured.
Enter fullscreen mode Exit fullscreen mode

This distinction matters because a result reported in a paper is not automatically evidence that the same method works in a different codebase, dataset, sample type, device, or laboratory environment.

Ideas without sufficient literature support can still be explored, but they must be labeled as first-principles hypotheses rather than presented as published conclusions.

Research and execution can overlap

The stages do not have to run in a strictly serial order.

While candidate N is executing, independent lanes can prepare future work:

Execution lane:
Test candidate N.

Literature lane:
Investigate mechanisms for candidate N+1.

Brainstorming lane:
Adapt those mechanisms to local constraints.

Review lane:
Check whether the benchmark or experimental comparison remains fair.
Enter fullscreen mode Exit fullscreen mode

If the user allows subagents, independent agents may own these lanes. If subagents are unavailable, the main agent can interleave the same work.

Prepared research cannot replace the current solution until its candidate has its own execution evidence, audit, explanation, and verdict.

Explicit evidence-based decisions

Every evaluated candidate ends with exactly one verdict:

  • PROMOTE β€” passes the gate and becomes the new incumbent;
  • RETAIN β€” useful as an alternative but does not replace the incumbent;
  • REVISE β€” the mechanism remains plausible and has a bounded next test;
  • REJECT β€” evidence contradicts the candidate or it fails a hard constraint;
  • INCONCLUSIVE β€” available evidence is insufficient for a valid decision.

A candidate cannot receive a positive verdict simply because the implementation looks reasonable.

The Skill first requires:

  1. execution evidence;
  2. an audit of metrics, baselines and confounders;
  3. a first-principles explanation;
  4. a comparison against the user-confirmed gate.

First-principles closeout

I also wanted the final explanation to go beyond a summary of edited files.

For a software project, the Skill traces the actual implementation path:

Input
β†’ entry point
β†’ changed function
β†’ downstream state transition
β†’ measured output
β†’ performance or complexity effect
Enter fullscreen mode Exit fullscreen mode

It explains where the cost comes from, which assumptions are necessary, what edge cases exist, and why the observed metric should change.

For a non-code project, such as an engineering or biological workflow, it follows the corresponding protocol and dataflow:

Sample or input
β†’ preparation
β†’ intervention
β†’ measurement
β†’ calibration
β†’ decision rule
β†’ reported metric
Enter fullscreen mode Exit fullscreen mode

This makes the Skill useful beyond software development. It can support engineering, data analysis, algorithms, and research workflows, as long as the project has measurable objectives and a safe validation path.

Minimal and resumable artifacts

The workflow keeps one living Markdown ledger:

docs/automation-exploration.md
Enter fullscreen mode Exit fullscreen mode

The ledger contains the frozen contract, candidate states, active lanes, evidence, audit results, explanations, verdicts, and resume state.

It reuses the project’s native tests, benchmarks, analysis outputs, or experimental records instead of creating a separate report directory for every candidate.

A dependency-free Python helper enforces the candidate lifecycle and prevents a verdict from being recorded before execution, review, and explanation are complete.

Self-contained installation

Automation Exploration Pro does not require another Skill for literature discovery or first-principles explanation.

Its repository includes:

  • the complete operating protocol;
  • a project-grounded literature workflow;
  • the first-principles closeout method;
  • a resumable ledger template;
  • a state-management helper;
  • tests and detailed cross-domain examples.

It uses whichever ordinary search, browser, scholarly-index, PDF-reading, coding, and execution tools are available in the host environment.

Clone the repository:

git clone https://github.com/fancheng5840074-bit/automation-exploration-pro.git
Enter fullscreen mode Exit fullscreen mode

Copy the Skill into your Codex Skill directory:

mkdir -p ~/.codex/skills

cp -R \
  automation-exploration-pro/skills/automation-exploration-pro \
  ~/.codex/skills/
Enter fullscreen mode Exit fullscreen mode

Restart or refresh Codex, then invoke it explicitly:

Use $automation-exploration-pro to explore several
research-backed solutions for this project.
Enter fullscreen mode Exit fullscreen mode

Or:

Use $automation-exploration-pro in cyclic optimization mode
to improve this baseline.
Enter fullscreen mode Exit fullscreen mode

What it does not promise

This Skill does not guarantee that every project has a better solution.

It does not turn a paper result into local evidence, and it does not hide failed experiments. If no candidate passes the confirmed gate, the correct outcome is β€œno promoted candidate.”

The goal is not to manufacture success. The goal is to make exploration more systematic, reproducible, explainable, and honest about uncertainty.

Feedback is welcome

Automation Exploration Pro is released under the MIT License.

I would especially appreciate feedback on:

  • the literature-to-local-adaptation workflow;
  • performance and complexity gate design;
  • cross-stage agent concurrency;
  • evidence review and verdict semantics;
  • applications outside conventional software projects.

If this workflow sounds useful, you can find the source, installation instructions, tests, and two detailed examples here:

πŸ‘‰ Automation Exploration Pro on GitHub

Issues and contributions are welcome.

Top comments (0)