The demo is a text box.
The product is the bad day.
A model is a service
you do not run,
on a network you do not control,
priced by the token,
and slower than everything else
in your stack.
Every rule you already hold
for a third party API
applies to it.
You keep forgetting
because the output sounds like a person.
Time it out.
Not the vendor default. Yours.
Decide how long a user will wait
before your product looks broken,
and make that the number.
Know what happens when it fails,
because it will fail,
in the middle of an afternoon,
in one region, for twenty minutes.
If your answer is
"the spinner keeps spinning",
you have not designed anything.
You have hoped.
Degrade into something.
The cached answer from yesterday.
The dumb deterministic path
you were too proud to keep.
An honest line saying
this part is unavailable right now.
Validate the output
the way you validate a form.
It is a form.
It is the least predictable form
you have ever integrated.
Parse it, check it, reject it,
and decide in advance
what the third bad parse does,
because "a human notices on Tuesday"
is not a plan.
Watch the money
the way you watch the latency.
A retry loop against a paid endpoint
is an invoice with a feedback loop.
Pin what you are allowed to pin.
The model behind that name
can change without a release of yours,
and the prompt you tuned so carefully
was tuned against something
that no longer exists.
Keep a small set of inputs
whose right answers you already know.
Run them on a schedule.
Then drift arrives as an alert
instead of arriving as a customer.
And keep it off the critical path
wherever you can.
Checkout does not need an opinion.
None of this is scepticism.
It is the ordinary discipline
you would apply to any dependency
that is slow, remote, metered
and occasionally wrong.
The most impressive thing
you can build around a model
is a failure nobody notices.
– Serguey Asael Shinder
Top comments (0)