The ~+9.4% You Can't Afford to Verify: Evaluating SDAR (and the FinOps of Trying)

#aws #machinelearning #reinforcementlearning #mlops

Recap.

Part 1 framed the supervision problem.

Part 2 architected the four-model system on AWS and counted the hardware.

Part 3 put the gate on the page in fifteen lines of PyTorch. Now the question that decides whether any of it mattered: does the gate actually earn its keep - and what would it cost to know?

This is the close. A verification design, the stability instrumentation that separates a real evaluation from a lucky one, and the FinOps reality that is the reason this whole series is a blueprint instead of a benchmark.

SDAR makes two claims, not one

It's tempting to reduce the paper to a single number. The reported gains over plain GRPO are real and worth stating - their numbers, not mine: roughly +9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop accuracy.

But the headline percentage is the less interesting claim. The one that matters is the second: SDAR avoids the training instability that naive GRPO+OPSD falls into. The whole point of the gate Part 3 was to keep the teacher's noisy rejections from destabilizing training. So a verification that only measures final task success has tested half the paper. You have to measure stability too - and most reproductions don't.

That two-pronged claim dictates the experiment.

The verification: three arms, not two

You cannot prove SDAR's thesis with a before/after. You need three training runs, identical except for the supervision:

Arm	What it is	What it proves
A. GRPO	Plain agentic RL, no teacher	The baseline the +9.4% is measured against
B. Naive GRPO+OPSD	Teacher distillation, no gate	The instability SDAR claims to fix
C. SDAR	Gated distillation Part 3	Beats A on score and beats B on stability

Drop arm B and you can show SDAR beats GRPO, but you've quietly deleted the paper's actual contribution - there's no evidence the gate did anything a plain teacher wouldn't. Arm B is the expensive arm nobody wants to run and the one that makes the result credible.

Metrics that actually test the claim

Metric	What it measures	Why it's here
Task success rate	% of episodes that hit the goal	The headline; compares all three arms
Turn efficiency	Steps taken vs optimal	Catches "succeeds but flails" agents
Gate-activation rate	Fraction of tokens amplified vs softened	SDAR-specific sanity check - a dead or saturated gate (Part 3, trap 3) shows up here
Per-turn loss variance	How much the loss jitters across turns	The stability signal. This is the metric that distinguishes arm C from arm B

That last row is the one to obsess over. SDAR's selling point lives or dies in the variance, not the mean.

Stability is the real story - so instrument it

Final accuracy is a lagging indicator. By the time arm B's success rate looks bad, you've already paid for a diverged run. The win is catching instability while it happens and killing the job - which is both good science and direct cost control.

Here's what to stream off every training run:

WATCH FOR DIVERGENCE  (kill early = save money)
  per-turn loss variance ── spikes ─────────► instability onset
  gate-value histogram   ── drifts to 0/1 ──► τ wrong, gate saturating
  gradient norm          ── blows up ───────► λ too high, teacher noise winning
  KL-to-reference        ── runs away ──────► actor drifting off-policy

On AWS, this is a standard observability wiring job:

Emit these as custom metrics to Datadog (or CloudWatch) from inside the training step - the same place the gate is computed Part 3.
Put a CloudWatch alarm on per-turn loss variance and gradient norm against the SageMaker training job; trip it and auto-stop.
A Grafana board with the four signals side by side, arm B vs arm C, is the single most persuasive artifact this experiment can produce - it shows the gate damping the wobble.

The honest version of the stability claim isn't "SDAR scored higher." It's "here are arm B and arm C on the same axes, and B's variance climbs while C's stays flat." That picture is worth more than any percentage.

The bill: why this is a blueprint

Time for the number that shaped the entire series. To run the three-arm comparison at the cheapest credible scale - one environment (ALFWorld), Qwen2.5-3B with LoRA, on a single p4d.24xlarge (8× A100 80GB):

Line item	On-demand	Spot (~50–70% off)
One converging run (~2–3 days × 8 GPU)	$1,570 – $2,360	$550 – $1,100
Three arms (A + B + C)	$4,720 – $7,080	$1,650 – $3,300
+ debugging & false starts (~+40%)	~$6,600 – $9,900	~$2,300 – $4,600
Idle node left running one weekend	$1,573	-

Public list pricing, US regions - verify yours; AWS cut P4/P5 by up to 45% in mid-2025.

So even the minimal honest verification is low-thousands on spot, near five figures on-demand - before you touch the second and third environments or scale past 3B. That is not a side-project line item, and pretending otherwise would be the dishonest move. Hence: I built and explained the mechanism; I am not buying the benchmark.

The cost levers, if you do run it:

Teacher on spot. It's frozen and inference-only Part 2 - fully interruptible, no training state to lose.
Aggressive S3 checkpointing, including the λ schedule position Part 3, so a spot reclaim resumes instead of restarting the warm-up you paid for.
Climb the model ladder. Debug the gate at 1.5B (minutes per loop), validate at 3B, only spend 7B money once arms B and C behave.
λ is a compute knob too - more distillation weight means more teacher forward passes in the hot path.
Auto-stop on every job. The $1,573 idle weekend is the most avoidable mistake on this list.

When SDAR isn't worth it

A technique is only as good as knowing when to skip it. SDAR's dense, gated supervision earns its complexity on long-horizon, multi-turn tasks where trajectory reward is hopelessly coarse. It buys you much less when:

The task is single-turn or short-horizon. Action and reward are already close together; plain GRPO assigns credit fine, and the teacher + gate are pure overhead.
Plain GRPO already converges cleanly on your task. If you're not seeing the instability, you don't need the cure.
Your budget is the constraint (hello). The four-model setup roughly adds a model's worth of memory and a teacher forward pass per step. On a tight budget, a well-tuned GRPO run is the rational call, and SDAR is a thing you design for later.

That last bullet is, honestly, my own situation - and naming it is more useful than pretending otherwise.

What this series was

Four parts, zero training runs, and a deliberate refusal to wave around a number I couldn't stand behind. What you have instead:

A clear model of the problem SDAR solves Part 1.
A buildable AWS architecture with the memory math that picks the hardware Part 2.
A working reference implementation of the gate, with the traps marked Part 3.
A verification design and an honest bill for proving it (this part).

That's a blueprint. Someone with GPU budget - a lab, a sponsor, a pile of credits - could pick this up and run the three-arm comparison tomorrow, and they'd know exactly what to measure and what it'll cost. The benchmarked reproduction is genuine future work, gated on compute, and I'd rather say that plainly than fake a convergence curve.

If you're that someone, the design is yours. And if you run it, I want to see the arm-B-vs-arm-C variance plot - that's the picture this whole series was pointing at.

That's the series. If it was useful, the implementation details in Part 3 are the part most worth stress-testing - tell me where the reconstruction is wrong.