The answers got worse on Tuesday.
Nothing shipped on Monday.
The code is identical
to the code that worked last week.
The branch has not moved.
Somebody edited the prompt
in a browser console
on Friday afternoon.
There is no commit.
No author.
No diff.
Nothing to revert to.
The sentence that decides
what your product says to customers
is the only part of your system
with no history.
It got there honestly.
Prompts arrive as experiments.
You paste one into a playground,
it works, you paste it into the feature,
and by the time it is load bearing
nobody has ever treated it as code,
because it does not look like code.
It behaves exactly like code.
A clause removed to make it read better
was the clause keeping it from quoting prices.
An example deleted for length
was the one demonstrating the format
that three downstream things parse.
The word concise,
added by somebody being helpful,
shortened every answer
and took the disclaimer off the end.
And the model underneath moved too,
because you asked for the latest,
and the latest is a moving target
you agreed in advance to follow.
So treat it as what it is.
The prompt is a file in the repository.
It ships with the release.
It is reviewed with the release.
It rolls back with the release.
The model identifier is pinned,
written down as a version,
and changed deliberately,
on a day when somebody is watching.
Keep thirty real inputs
and the properties the answers must hold.
Not a score.
Properties.
It must refuse to quote a price.
It must return parseable output.
It must not promise anybody a refund.
Run them before anything goes out.
And record, with every response,
which prompt version and which model produced it,
so that when a customer arrives with a screenshot
you can answer the only question that matters.
What exactly was it told.
You would never let somebody change a pricing rule
by typing into a text box
that keeps no record.
At the moment, you do.
– Serguey Asael Shinder
Top comments (0)