LLMs carry a judgment signal you can read with a linear probe (and why steering it fails)

LLM hidden states encode reward-related information: roughly, whether the current trajectory is heading toward a correct answer.

We reproduced this on our own hardware, added the controls we felt were missing, and got a clear picture of what the signal can and cannot do.

Setup

• Generate N answers, label each by final correctness
• Take the last-token hidden state at layer L
• Fit a linear probe (ridge) predicting correctness
• Report AUC on a held-out set

No fine-tuning, no extra model.

Result 1: the signal is real across scales

Model	Probe AUC
1.5B	0.81
4B (Gemma-4 based)	0.96
9B	0.72
27B	0.95
26B MoE (Gemma-4)	0.987

Note it is not monotone in size. Architecture family matters more than parameter count.

Result 2: it is causal, not decorative

Zeroing the top 1% of dimensions by probe weight at one layer:

• accuracy 0.567 to 0.108 (minus 45.8 points)
• zeroing random dimensions, same count: minus 0.8 points

The control is the important half. Without it you cannot distinguish "this direction matters" from "perturbing anything hurts".

Result 3: reading works, steering does not

This is where most of our compute went, and where the result was negative:

Intervention	Held-out effect
Add alpha * w to activations	minus 4.7 points
Sigma-normalized dose	no gain
Clamp and gate (self-limiting)	no gain
Best-of-N selection by probe	loses to plain majority vote

And the killer control: injecting a random direction at matched magnitude produced an identical score (0.575 vs 0.575). The value direction was not special; only the magnitude was.

What it is genuinely good for

Selective prediction. Rank answers by probe score, answer only the top X percent:

Coverage	Accuracy
100%	73.0%
80%	85.0%
70%	92.9%
60%	100%

Held-out detection AUC: 0.940.

The probe is an excellent thermometer and a useless heater. You can read whether the model knows. You cannot push it into knowing.

The caveat that bit us

The probe is domain-dependent. Same model, same method: 0.94 AUC on math reasoning, 0.66 on factual/legal questions.

Models that are confused leave a trace. Models that are confidently ignorant do not.

If you deploy this, recalibrate per domain and report per-domain numbers. A single headline AUC is misleading.

---

From VIDRAFT, a Korean deep-tech company running an **AI Foundry* — we diagnose, breed and optimize AI models for specific industries. Open models: Hugging Face · vidraft.net*