Your Scaffolding Is Training Wheels, Not Architecture

There's a belief I keep running into in architecture reviews: the workflow around the model is the real system, and the model is a swappable part inside it. Step-by-step plans, per-task hints, a critic that reviews every tool call, retry loops with backoff, a scratchpad that gets summarized every turn. The harness is the product. Change the model, keep the harness, ship.

I've built those harnesses. I still build them. But the belief is wrong in a specific way, and that way costs people real money.

The paper that pushed me on this is AutoScaffold, out of the RL-for-language-models crowd. The framing is simple enough to state in a sentence: instead of hand-writing one scaffold and hoping, you let a model generate and adapt the scaffolding during training — hints, decompositions, the intermediate crutches a student needs to get traction on hard tasks. Fine. Scaffolding helps during training; nobody who has trained anything is surprised.

Here's the part that made me sit up. At evaluation they strip every scaffold. All of it — no hints, no generated decompositions, no step-by-step plan, just the student and the task. And the students still beat their RL baselines. The scaffolding was curriculum, not crutch. It taught something that survived being taken away.

That's the whole ballgame, and it's the opposite of how most agent stacks are evaluated.

The demo measures the harness

Think about what a normal agent demo actually proves. You hand the model a task, it fails, your loop re-plans, it fails differently, your critic catches the bad tool call, the loop retries with the error message appended, and on attempt four it lands. Screenshot. Ship it.

What did attempt four demonstrate? That your retry loop is a variance-reduction machine. That's a real and useful thing — sampling is noisy, second attempts genuinely help — but it's not the model getting smarter. It's you buying more rolls of the dice with tokens. Price it as that and it's honest engineering. Describe it as "the agent solved it" and you're measuring your harness and crediting your model.

Per-task hints are worse, because they're the part people forget they injected. Every hardcoded hint in a prompt is domain knowledge a human wrote at authoring time and handed to the model at runtime. Great for a demo. Terrible as architecture, because the architecture only works on tasks you already know how to hint at. Your eval then measures your ability to write hints, not the model's ability to reason. Those are different skills and only one of them shows up in production.

The strip test

The diagnostic I've started running costs an afternoon and is embarrassingly simple. Take one task suite. Run it three ways, same model, same temperature:

1. Full harness — planner, hints, critic, retries.
2. Harness minus the thinking — retries and error-feedback stay, hints and planning go.
3. Bare — the raw task, one shot, no loop.

The delta between 1 and 2 is what your hints were worth. The delta between 2 and 3 is what your retry budget was worth. If 3 is a cliff and 1 is a plateau, you don't have an agent, you have a very expensive prompt.

I'll say the obvious caveat: not every collapse is a failure. Some harness pieces aren't thinking at all, they're plumbing — auth handling, tool schemas, idempotency keys, output parsing, the thing that turns a JSON blob into a valid database write. Strip that and of course the score dies, because you removed the interface to reality, not the intelligence. The distinction I care about is whether a harness component gives the model more shots at the problem or does the thinking for it. Retries are shots. A planner that decomposes the task is thinking. Keep the plumbing. Audit the thinking.

What the transfer result actually implies

If AutoScaffold's students internalize what the scaffold was doing, the scaffold was a teaching device — a temporary structure that gets removed when the building stands. That's a completely different mental model from "the scaffold is the building." It implies the scaffold should be designed to be deletable, and that you should be actively trying to delete it.

In practice that means: version the harness independently of the model. Keep a bare config in the repo that runs on every eval, not just the happy-path one. Report both numbers next to each other and watch the gap. That gap is your scaffolding debt, and like all debt it's fine until someone asks you to pay it — usually the moment you swap in a newer model and discover your careful hints are now actively fighting it.

Hedging, because I should: AutoScaffold is training-time scaffolding, and I'm extrapolating to inference-time orchestration, which is not the same thing. Maybe for some product shapes the harness really is the moat — a well-designed domain tool surface can be defensible even when the model does 100% of the reasoning. And I haven't run the paper's setup myself; I've run the strip test on my own stacks.

But the instinct is right and I'll keep repeating it. If your demo only works with the harness attached, you don't know what your model can do. You know what your harness can do. Those two numbers will diverge in production, and production is where the harness stops fitting.