The instruction was in capitals.
Never quote a price.
It held for four hundred conversations,
and then it quoted a price
to somebody who screenshotted it.
Nothing broke.
Nothing changed.
You had simply misunderstood
what that sentence was.
A line in a prompt is not a rule.
There is no part of the system
where it is enforced,
no branch, no check,
nothing that can refuse.
It is a strong suggestion
sitting in the same pile
as everything else in the window,
competing with the user's words,
which arrive through the same door
in the same format.
So it holds until something outweighs it.
A long conversation,
where your instruction is now
far away and small.
A document you pasted in
that contains its own instructions,
addressed to the model,
which the model cannot tell
from yours.
A person who asks the fifth way
after four refusals.
Most of the time you get compliance,
and you read that as a guarantee,
because a hundred passes in a row
looks exactly like a guarantee
until the day it does not.
Put the constraint somewhere real.
If the answer must come
only from your documents,
check the answer against them
before it is sent,
in code, deterministically,
and drop it if it does not match.
If it must never state a figure,
look for figures in the output.
If it must never promise a date,
that is a search,
not a request.
If the rule is legal,
a person signs it off,
and the model drafts.
Keep the prompt for steering.
Tone, shape, what to do
when the question is unclear.
That is what it is good at,
and the failures are cosmetic.
Keep boundaries in code
you can read, test,
and point at in a commit
when somebody asks
how you know.
The question is not
whether your instruction usually works.
It is what stands between you
and the output
on the day it does not,
and whether the answer
is another sentence
in the same paragraph.
– Serguey Asael Shinder
Top comments (0)