The strongest LLMs are often the ones you have the least control over. You reach them through an API, you cannot touch their weights, and your only lever is the text you send in. Hand-writing one clever prompt helps, but it is a single fixed instruction applied to wildly different inputs, so it under-steers the hard cases. Directional Stimulus Prompting (Li et al., 2023) is a genuinely clever answer to this, and I built an interactive demo of the loop to make it click.
The idea: train a tiny policy, not the big model
DSP puts a small, tuneable policy model in front of the frozen LLM. For each input, the policy writes a short directional stimulus — a handful of instance-specific hint keywords — that you prepend to the prompt. The giant LLM never changes; only the little policy learns.
const policy = new SmallSeq2Seq("t5-base"); // ~220M params, trainable
async function dsp(input){
const stimulus = policy.generate(input); // the per-input nudge
return frozenLLM(build(input, stimulus)); // big model untouched
}
That split is the whole trick. The frozen LLM (maybe 100× bigger, maybe an API you can't reach for training) supplies fluency and world knowledge; the small policy carries the controllable part. Fine-tuning a frontier model per task is impossible or ruinously expensive; fine-tuning a T5-base to write good hints is a laptop-scale job.
What the stimulus actually is
It is a compact, task-shaped hint, not a paragraph and not a global instruction:
summarize -> keywords: "Aurora X1; foldable; $2,199; March; pre-orders"
chatbot reply -> act+slots: inform(restaurant); vegan=yes; party=4; price=cheap
Because it is generated per input, it can name what matters for this example — the difference between "include the key details" and "include Aurora X1, $2,199, ships March." Prepended to the prompt, the hints act as a soft constraint the model tends to satisfy, tilting its output toward the target without hard-coding the answer.
The demo: watch coverage climb
In the interactive version you pick a task (summarize an article, or write an assistant reply), then toggle the hint keywords the policy proposes. The steered output is assembled from whichever hints are active, so the target key-point coverage below is computed, not asserted. Turn the stimulus off and the frozen LLM falls back to its generic answer — "TechCorp showed off a new laptop at an event this week" — which technically answers but misses every fact you cared about. Toggle the on-target hints on and coverage visibly climbs from 20% to 100%, with the model unchanged the entire time. The point lands physically: you never touched the model, you changed what you fed it.
Training: warm up, then optimize on the LLM's own score
The policy is trained in two stages, and the second is the elegant one.
# 5a. SFT warm-up: imitate pseudo-hints pulled from the references
policy.fit(train_inputs, extract_keywords(references)) # a sane starting point
# 5b. RL (PPO): the reward IS the frozen LLM's downstream score
for x in train_inputs:
stim = policy.sample(x) # explore hint variants
out = frozen_llm(build(x, stim)) # black box = environment
R = reward(out, ref_for(x)) # ROUGE / coverage / task success
policy.ppo_update(x, stim, R) # only the policy moves
The supervised warm-up teaches the policy the shape of a reasonable stimulus so the RL step explores productively instead of flailing. Then reinforcement learning optimizes the policy against the LLM's measured performance. Crucially the LLM is treated as part of the (non-differentiable) environment — you never backprop through it, you just use its output as a scalar reward — which is exactly why this works on closed APIs.
Where it fits, and where it doesn't
DSP sits in a useful middle ground. Unlike fine-tuning the LLM, it never changes the big model, so it works on black-box APIs at a fraction of the cost. Unlike soft prompts or prefix-tuning, its stimulus is discrete, readable text — you can literally see the hints — and it needs no gradient access. Unlike RLHF, which uses RL to change the LLM's own weights, DSP trains a separate steering policy while the LLM stays frozen.
Its two requirements are honest: a way to score the output (so the policy has a reward) and enough data to train the policy. No measurable target or no training data, and you should fall back to a well-crafted hand-written or few-shot prompt. But where those exist, a model you can't touch becomes a model you can precisely aim.
Pick a task, toggle the hints, and watch the frozen model's answer snap toward the facts you care about:
https://dev48v.infy.uk/prompt/day34-directional-stimulus.html
Top comments (0)