Hugging Face measured it: the same model scores 33% or 62% depending on the harness

Hugging Face announced OpenEnv on October 5, 2026. It is an open-source capture proxy that turns real coding agent harnesses into reinforcement learning environments, without modifying the harnesses. Ten of them run through it today, including Claude Code, Codex, OpenCode, Pi and Hermes. The announcement post and full guide live in a Space: the multi-harness RL guide, with docs at huggingface.co/docs/openenv.

The reason I care is not the training pipeline. It is one measurement buried in the announcement, because it quantifies something config people keep suspecting and rarely get numbers for.

The number that stopped me: 33% vs 62% with the same weights

The team ran the same model, Liquid AI's LFM2.5-2.6B, through different harnesses on the same tasks. Under Mini-SWE-Agent, a lightweight scaffold, it solved 62% of them. Under Claude Code it solved 33%. Same weights, same tasks, same model. The harness alone nearly doubled the score.

That is a 29-point spread from configuration and tooling alone. When someone tells you "model X is better than model Y", and they tested X in one harness and Y in another, the comparison is close to meaningless. The harness is part of the measurement.

How the capture proxy works

The trick is that the proxy is not a rewrite. The harness believes it is talking to a normal model API. It is actually talking to a local proxy that speaks the four protocols coding agents use (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini), forwards requests to a vLLM server, and records the exact sampled token IDs and logprobs. TRL then trains on those recorded turns with async GRPO. Task execution and verification run in isolated Harbor and Docker sandboxes.

Each rollout is defined by a plain task shape, which is the part worth stealing even if you never train anything:

task = {
    "instruction": "Create binary_search.py exposing def binary_search(arr, target) -> int",
    "setup": [],   # bash commands run before the agent
    "verify": [    # bash commands run after, these produce the reward
        "python -c 'import binary_search; assert binary_search.binary_search([1,2,3], 2) == 1'",
    ],
}

Instruction, setup, verify. An outcome check the agent never sees. The proxy has two modes, transparent_proxy for training with logprob capture and black_box for smoke tests and data collection.

Training in one harness overfits

The second finding matters if you tune agents per tool. Training the model inside OpenCode alone pushed its OpenCode solve rate from 34% to 58%, but the gains barely transferred to other environments. Training across four harnesses at once lifted the average from 42% to 54%, with Claude Code specifically going from 33% to 49%.

The team's own warning was blunt: training against a single harness risks overfitting to its tool-call dialect. Replace "training" with "configuring" and the finding rhymes with something I have seen in repos. A rules file tuned to death inside one agent's quirks stops making sense the day you add a second tool or upgrade a version.

They also beat imitation learning with the same setup. Supervised fine-tuning on 3,189 rollouts distilled from Qwen3.8-27B plateaued at 47.5%, below the multi-harness RL result. Learning from outcomes in the real environment beat copying a stronger model's moves.

The reward you shape is a config decision

My favorite detail in the release is a one-line reward change. They added a small bonus for solving a task in fewer tool calls. On tasks the model already solved, tool calls dropped 31% across all harnesses, and about 50% under Codex. One term in a reward function, half the tool traffic gone.

That is the RL version of a knob config authors turn by writing prose. "Prefer one focused command over five exploratory ones" in a rules file is the same intent, executed with far less force. It is a useful reminder that behavioral budget instructions (tool calls, steps, context) are not filler. They are shaping, and they measurably move behavior.

Why this matters if you never train a model

You do not need a GPU rack to use the findings. Three takeaways cost nothing:

1. Your harness is part of your agent's measured performance. The rules file, the MCP server set, the permission gates, the context window the harness assembles. Swap any of them and you are evaluating a different system.
2. The instruction/setup/verify pattern is a free regression test for agent setups. Write a verify list per repo that asserts the invariants your agent must not break, and run it after config changes.
3. Model comparisons need a fixed harness, stated out loud. I stopped trusting "I switched to model X and it's worse" takes that do not name the tool and version they switched inside, and 7 reasons your agent ignores your rules covers several of the mechanisms that quietly cause it.

If you want this done for you, the kits in AgentConfig Studio version-pin the harness alongside the config, and the free Next.js sample shows the pattern in one repo.

What I would check on Monday

• Pin the agent version next to the config version, in the same commit. If a harness update can swing results by double digits, harness and config must move together, the way versioning your agent configs describes.
• Write one verify script per repository that checks agent-relevant invariants (build passes, tests pass, no secrets staged). Run it before and after any agent config change, the same way OpenEnv scores a rollout.
• Log tool calls per task for a week. You cannot shape what you do not count, and a sudden 2x spike is usually a config regression, not a model regression.
• When you publish a benchmark or a hot take, name the harness, its version, and the config files involved. Your readers cannot reproduce 33% vs 62% splits without them.

Limits, honestly

The numbers are Hugging Face's own, on a 2.6B model, mostly data-analysis tasks from SmolDataEnvs. The reference tutorial needs two GPUs, one serving the policy with vLLM and one training, and the docs describe the local sandbox path as validated end to end on Qwen3. A 2.6B model with a 29-point harness spread does not prove a frontier model spreads the same way, though the direction is well established by the scaffold-vs-production gap they set out to close.

Everything is open: the proxy, the TRL trainer integration, the task datasets, the SFT corpus, the training configs, and seven trained checkpoints. Even if you never run it, the codebase is a readable reference for how agent-environment contracts should look.