The demo cost four pence.
You ran it thirty times
while building it,
glanced at the total,
and stopped thinking about it,
because four pence
is not a number anybody worries about.
Then it shipped,
and the cost stopped being yours.
It became a function
of what people do.
Somebody pastes in
a two hundred page document.
A nightly job decides
to re summarise everything,
including the things
that have not changed.
A page re asks the same question
on every keystroke,
because somebody wired it
to the wrong event.
None of that is abuse.
That is a feature working,
and the bill grows
in a place nobody is watching,
because the graph you watch
is latency.
Treat cost the way you treat latency.
Log it per request,
beside the duration,
with the user and the endpoint attached.
Then you can answer
the question that is actually coming,
which is not how much.
It is who.
Put a ceiling on one user.
Per hour, per day,
whatever suits the product.
Decide today what happens
when somebody reaches it,
because if you do not decide,
the answer is that everyone
gets a degraded service
funded by one person
who wrote a loop.
Look at what you are sending.
Most prompts are mostly boilerplate,
the same instructions
on every single call,
and half the context is there
because somebody was unsure
what mattered and included it all.
Trimming that is not tidiness.
It is the invoice.
Cache the repeats.
People ask the same question,
in almost the same words,
far more often than you expect.
And stop sending everything
to one model.
Most of the traffic is easy.
The expensive one is answering it
only because that is what
you happened to build with.
None of this is premature.
Cost is a property of the feature,
exactly like response time
and error rate,
and it is the only one
you learn about a month late,
in a summary,
from somebody who is not pleased.
The meter is running now.
Go and read it.
– Serguey Asael Shinder
Top comments (0)